7. Replication, High Availability and Scaling
A single Redis server is a single point of failure, and it is limited to one machine's memory and one core's command throughput. This chapter covers the three tools that remove those limits. Replication keeps live copies of the data on other servers, Sentinel promotes a copy automatically when the primary fails, and Redis Cluster splits the keyspace across several primaries. It ends with the scaling options outside the core server: client-side sharding, proxies, and multi-region Active-Active databases. Each tool trades something (consistency, latency, simplicity or multi-key operations) for availability or capacity, and the main skill is knowing which trade you are making.
<a id="71-replication"></a>
7.1 Replication
This subchapter covers how one Redis server keeps one or more copies of itself up to date. It explains the primary-replica model and the REPLICAOF command. It shows how a replica catches up after a disconnection, and how the replication backlog decides whether the catch-up is cheap or expensive. It also covers what replicas may serve, and what WAIT can and cannot promise about durability.
Primary-Replica Architecture
A server that holds the only copy of the data loses all of it, or at least its availability, when the machine dies. Persistence (chapter 6) helps only once the machine comes back. The usual answer in databases is to keep a second, live copy on another machine. Redis does this with replication: one server, the primary, accepts writes and sends a stream of them to one or more replicas, which apply the same changes and hold an identical dataset.
An analogy: a lecturer writes on the board, and several students copy everything into their notebooks as it happens. If the lecturer leaves, any student's notebook holds almost the whole lecture. The "almost" matters, and it is the main theme of this subchapter.
What replication gives you
| Goal | How replicas help |
|---|---|
| Redundancy | A replica on another host, rack or zone holds a complete copy |
| Failover | A replica can be promoted to primary (by hand, by Sentinel in 7.2, or by Cluster in 7.3) |
| Read scaling | Read-only traffic can be sent to replicas (see "Read-Only Replicas") |
| Offloading | Backups (RDB snapshots, chapter 6.3) and heavy analytical reads can run on a replica |
Replication alone does not give automatic failover. If the primary dies and nothing promotes a replica, writes stop.
How it works
- The replica connects to the primary and asks for a copy of the data (the handshake and full sync are covered in "Full and Partial Resynchronization").
- The primary sends a snapshot of the dataset.
- From then on, the primary forwards every command that changes data to all replicas, as a replication stream in the same protocol clients use (RESP).
- Every replica tracks how many bytes of that stream it has processed, its replication offset, and reports it to the primary about once per second (
REPLCONF ACK).
Two identifiers describe each dataset's history:
- Replication ID (
master_replid): a random 40-character string that names one history of the dataset. A new primary generates a new one. - Replication offset (
master_repl_offset): the number of bytes of the replication stream produced so far.
The pair (replication ID, offset) identifies an exact version of the dataset. Two servers with the same pair hold the same data.
Replication is asynchronous
The primary replies OK to the client before the replicas have received the write. This keeps write latency low, because no network round trip to the replicas is in the path, but it creates a window of loss:
client primary replica
│ SET k v │ │
│────────────────▶│ apply k=v │
│◀──────── OK ────│ │
│ │──── SET k v (stream) ──────▶│ (a few ms later)
│ ✖ crash before the stream is sent
│ │ replica is promoted: k is missing,
│ │ but the client was told "OK"
Use WAIT (see "WAIT and Write Consistency") to narrow this window for specific writes. Redis does not offer fully synchronous replication.
Topologies
Star (most common) Chained (cascading)
┌─────────┐ ┌─────────┐
│ primary │ │ primary │
└────┬────┘ └────┬────┘
┌──────┼──────┐ │
▼ ▼ ▼ ┌────▼────┐
replica replica replica │ replica │ replicates from primary
└────┬────┘
┌─────┴─────┐
▼ ▼
sub-replica sub-replica receive the primary's
stream, relayed unchanged
Chaining reduces the primary's outbound bandwidth when there are many replicas or a remote data center. Since Redis 4.0 a sub-replica receives exactly the same stream as its parent, so all servers in the chain share one replication ID and comparable offsets.
What the replication stream contains
- Write commands as executed. Commands that would give different results on each server are rewritten first. For example,
INCRBYFLOATis replicated as aSETof the result, and an expiration given in relative time (EXPIRE,SET ... EX) is replicated as an absolute timestamp (PEXPIREAT,SET ... PXAT). This keeps replicas correct if the stream arrives late. - Lua scripts and functions are replicated by their effects, the write commands they actually performed (always, since Redis 7.0).
- Expirations: replicas never delete expired keys on their own. When a key expires on the primary, through access or the active expiry cycle (chapter 2.1), the primary sends a
DEL(orUNLINK) to the replicas. Until it arrives, a replica answers reads of a logically expired key as if the key did not exist. This keeps the replicas consistent with the primary even if clocks differ slightly.
Example: a primary with two replicas (Docker)
docker network create redis-net
docker run -d --name primary --network redis-net redis:8
docker run -d --name replica1 --network redis-net redis:8 redis-server --replicaof primary 6379
docker run -d --name replica2 --network redis-net redis:8 redis-server --replicaof primary 6379
docker exec primary redis-cli SET greeting hello
docker exec replica1 redis-cli GET greeting # "hello"
docker exec primary redis-cli INFO replication
# Sample output (Redis 8.10):
# role:master
# connected_slaves:2
# slave0:ip=172.19.0.3,port=6379,state=online,offset=29,lag=0,io-thread=0
# slave1:ip=172.19.0.4,port=6379,state=online,offset=29,lag=0,io-thread=0
# master_replid:24bc81ad3600e6728cc736bb159cac6623cedf8a
# master_replid2:0000000000000000000000000000000000000000
# master_repl_offset:29
# second_repl_offset:-1
# repl_backlog_active:1
# repl_backlog_size:1048576
Field names such as role:slave and connected_slaves keep the old terminology for compatibility. Commands and configuration use "replica" (REPLICAOF, replica-read-only), with the old names kept as aliases.
Key Takeaways
- A primary forwards its write stream to replicas, which hold full copies of the dataset.
- Replication is asynchronous: an acknowledged write can be lost if the primary fails before replicas receive it.
- A replication ID plus an offset identifies one exact version of the dataset.
- Replicas do not expire keys themselves. They wait for the primary's
DEL, but already hide logically expired keys on reads. - Replication provides copies, not failover. Promotion needs a human, Sentinel or Cluster.
Practice
- (Beginner) Start the three containers above. Write 100 keys to the primary and check
DBSIZEon both replicas. Comparemaster_repl_offseton the primary withslave_repl_offseton each replica.- (Intermediate) On the primary, run
SET temp 1 EX 5, thenMONITORon a replica for 10 seconds. Which commands arrive from the primary, and in what form? Explain why theEX 5is not replicated literally.- (Interview) "We run one primary and two replicas in the same availability zone. Is our data safe?" Hint: separate the failure modes (process crash, host loss, zone loss, operator error such as
FLUSHALL) and check which ones replication covers.
REPLICAOF
Any Redis server can become a replica of another one at runtime, without a restart. A single command switches the role, which is convenient for migrations and failovers, and dangerous when it is pointed at the wrong server.
Syntax
| Command or directive | Effect |
|---|---|
REPLICAOF <host> <port> | Become a replica of the given primary. Replies OK immediately; the sync runs in the background |
REPLICAOF NO ONE | Stop replicating and become a primary, keeping the current data |
replicaof <host> <port> in redis.conf | Start as a replica after every restart |
SLAVEOF | Deprecated alias of REPLICAOF |
When a server becomes a replica of a primary with a different history, it discards its whole dataset and replaces it with the primary's. REPLICAOF on the wrong server, or pointing to the wrong primary, wipes data in seconds. Restrict the command with ACLs (chapter 11.1) or rename it on servers where roles never change.
REPLICAOF NO ONE is the manual promotion step. The server keeps its data, generates a new replication ID, and remembers the old one as master_replid2, together with the offset at which the history changed (second_repl_offset). Other replicas of the old primary can then switch to the new primary with a partial resync instead of a full copy (see the next topic).
Authentication and addressing
# redis.conf on a replica
replicaof 10.0.1.10 6379
# Credentials the replica uses to connect to the primary
masteruser replicator # ACL user (Redis 6+). Omit to use the default user
masterauth s3cr3t-repl-pass # password for that user
# Behind NAT or in containers, the primary sees the wrong address for the replica.
# The replica announces the address that others should use instead.
replica-announce-ip 203.0.113.25
replica-announce-port 6380
A minimal ACL user for replication only needs the commands used in the handshake (+psync +replconf +ping). Chapter 11.1 shows how to create it.
Watching a replica attach
The command returns at once, so check the result in INFO replication on the replica:
| Field | Meaning |
|---|---|
master_link_status | up once the initial sync is complete and the stream flows |
master_sync_in_progress | 1 while the snapshot is being transferred or loaded |
master_last_io_seconds_ago | Seconds since the last data from the primary (pings arrive every 10 s by default) |
slave_repl_offset | Bytes of the stream processed by this replica |
slave_read_only | 1 if writes from clients are refused |
ROLE gives a compact view and works on primaries, replicas and Sentinels:
redis-cli -h primary ROLE
# 1) "master"
# 2) (integer) 43642170 <- primary's replication offset
# 3) 1) 1) "172.19.0.3" <- one entry per replica: ip, port, acknowledged offset
# 2) "6379"
# 3) "43642170"
# ...
redis-cli -h replica1 ROLE
# 1) "slave"
# 2) "primary" <- primary host
# 3) (integer) 6379
# 4) "connected" <- connect | connecting | sync | connected
# 5) (integer) 43642170 <- offset processed by this replica
Example: attach a replica and wait until it is usable (Python)
import time
import redis
def attach_replica(replica: redis.Redis, host: str, port: int, timeout_s: float = 300) -> None:
"""Make `replica` replicate host:port and block until the initial sync is done."""
replica.execute_command("REPLICAOF", host, port) # returns immediately
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
info = replica.info("replication")
if info.get("master_link_status") == "up" and info.get("master_sync_in_progress") == 0:
return
time.sleep(0.5)
raise TimeoutError(f"replica did not finish syncing with {host}:{port}")
r = redis.Redis(host="replica1", decode_responses=True)
attach_replica(r, "primary", 6379)
print(r.execute_command("ROLE")[:4]) # ['slave', 'primary', 6379, 'connected']
Manual failover with REPLICAOF
Without Sentinel or Cluster, a planned switch of primaries looks like this:
- Stop writes on the old primary, for example with
CLIENT PAUSE 30000 WRITE. - Wait until the chosen replica's offset equals the primary's (
ROLEon both, orWAITwith all replicas). - On the chosen replica:
REPLICAOF NO ONE. - On the other replicas and the old primary:
REPLICAOF <new-primary> 6379. - Point the clients to the new primary, then
CLIENT UNPAUSEon the old one so that blocked clients get redirected or error out.
Redis also has a built-in coordinated version for one replica: FAILOVER [TO <host> <port>] [TIMEOUT <ms>], run on the primary. It pauses writes, waits for the target replica to catch up, and swaps the roles. Its progress appears as master_failover_state in INFO replication.
In cluster mode, REPLICAOF is refused. Use CLUSTER REPLICATE <node-id> instead (7.3).
Key Takeaways
REPLICAOF host portmakes a server a replica at runtime.REPLICAOF NO ONEpromotes it and keeps its data.- Switching to a primary with a different history discards the replica's dataset. Protect the command.
- The command replies before the sync is done. Wait for
master_link_status:upbefore relying on the replica. - Use
masteruser/masterauthfor authentication andreplica-announce-ip/-portbehind NAT. FAILOVERperforms a coordinated, lossless role swap between a primary and one replica.
Practice
- (Beginner) Turn a standalone server with a few keys into a replica of another server with different keys. Which keys remain afterwards? Then run
REPLICAOF NO ONEand write to it.- (Intermediate) With one primary and two replicas, perform the five-step manual failover above while a script writes an incrementing counter every 10 ms. Verify that no increment was lost, then repeat with
FAILOVER TO <replica-host> 6379and compare.- (Interview) "During an incident, someone ran
REPLICAOFon the wrong server and the primary became a replica of an empty instance. What happened to the data, and how could the setup have prevented it?" Hint: think about what a replica does with its dataset during a full sync, and about ACLs,rename-command, and backups.
Full and Partial Resynchronization
A replica can fall behind for many reasons: a restart, a network blip, a failover. Copying the whole dataset again every time would be slow and expensive for both servers. Redis therefore has two ways to catch up. A partial resynchronization sends only the missing part of the stream. A full resynchronization sends a complete snapshot.
The PSYNC handshake
Every time a replica connects, it sends PSYNC <replication-id> <offset>: "I have the history named X up to byte N, please continue from N+1."
replica primary
│── PING ──────────────────────────────────▶│
│── REPLCONF listening-port / capa ... ────▶│ capabilities, e.g. eof, psync2
│── PSYNC 24bc81ad...f8a 441852 ───────────▶│
│ │ Do I know this ID (as replid or replid2)?
│ │ Is byte 441852 still in my backlog?
│◀──── +CONTINUE ────── yes to both ─────────│ partial: send only the missing bytes
│◀──── +FULLRESYNC <id> <offset> ── no ──────│ full: send a snapshot, then the stream
A brand-new replica sends PSYNC ? -1 (no known history), which always leads to a full sync.
Partial resynchronization
Possible when both conditions hold:
- The replica's replication ID is the primary's current ID, or its previous ID (
master_replid2) with an offset up tosecond_repl_offset. This is what lets replicas of a failed primary partially sync with a promoted replica (PSYNC2, Redis 4.0+). - The requested offset is still inside the primary's replication backlog (next topic).
Then the primary sends only the bytes the replica missed. From a real run where the replica's link was dropped for a moment:
# replica log
* Trying a partial resynchronization (request 24bc81ad3600e6728cc736bb159cac6623cedf8a:441852).
* Successful partial resynchronization with master.
# primary log
* Partial resynchronization request from 172.19.0.3:6379 accepted. Sending 0 bytes of backlog starting from offset 441852.
Full resynchronization
Needed for a new replica, a replica with a different history, or a gap larger than the backlog. The primary must produce a snapshot of the whole dataset, send it, and keep the writes that happen in the meantime so the replica can apply them afterwards.
| Setting | Default | Effect |
|---|---|---|
repl-diskless-sync | yes (since 7.0) | The fork child streams the RDB directly to the replica sockets. With no, it writes dump.rdb to disk first and the parent sends the file |
repl-diskless-sync-delay | 5 s | Wait for more replicas to arrive so that one fork serves all of them |
repl-diskless-sync-max-replicas | 0 | Start before the delay ends once this many replicas are waiting (0 = always wait the full delay) |
repl-diskless-load | disabled | How the replica loads: disabled writes the received RDB to disk first; on-empty-db parses from the socket only if the replica is empty; swapdb keeps the old dataset in memory while parsing, so it can keep serving it (needs memory for both) |
Classic flow, used when the snapshot is written to disk (repl-diskless-sync no):
primary: fork ──▶ child writes RDB ──▶ parent sends RDB ──▶ parent sends buffered writes ──▶ live stream
meanwhile new writes accumulate in the replica's output buffer ON THE PRIMARY
replica: receive RDB ──▶ flush old data, load RDB ──▶ apply writes
RDB channel replication (Redis 8.0+)
With diskless sync, Redis 8.0 and later use a second connection, the RDB channel. The replica opens one connection for the RDB and keeps the main connection for the live stream. The primary sends both in parallel, and the replica buffers the live stream in its own memory until the RDB is loaded. This moves the buffering from the primary to the replica, so a large full sync no longer puts the primary at risk of hitting its output-buffer limit, and the primary's memory stays stable. A sample run on Redis 8.10:
# primary
* Replica 172.19.0.4:6379 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
* Full resync requested by replica 172.19.0.4:6379 (rdb-channel)
* Delay next BGSAVE for diskless SYNC <- repl-diskless-sync-delay
* Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
* BGSAVE done, 262204 keys saved, 0 keys skipped, 31196500 bytes written.
* Synchronization with replica 172.19.0.4:6379 succeeded
# replica
* PSYNC is not possible, initialize RDB channel.
* Starting to receive RDB and replication stream in parallel.
* MASTER <-> REPLICA sync: Flushing old data
* Done loading RDB, keys loaded: 262204, keys expired: 0.
* MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (...)
The replica's buffer is limited by replica-full-sync-buffer-limit (0 means "use the primary's client-output-buffer-limit replica hard limit"). INFO replication on the replica shows replica_full_sync_buffer_size and replica_full_sync_buffer_peak. With repl-diskless-sync no, Redis falls back to the classic flow. Valkey has a similar mechanism, called dual-channel replication (Valkey 8.0+, dual-channel-replication-enabled, off by default).
What a full sync costs
| Resource | Cost |
|---|---|
| Primary CPU and latency | A fork() (chapter 6.1) and RDB serialization |
| Primary memory | Copy-on-write pages, plus buffered writes (classic flow) |
| Network | The whole dataset, once per replica (or once per batch with diskless) |
| Replica | Flushes its data, then is busy loading. With replica-serve-stale-data yes it answers from the old data until the flush; while loading, it replies LOADING |
A full sync of a 25 GB dataset can take minutes and cause several seconds of extra latency on the primary. Repeated full syncs (a "sync loop") are a common production incident. They usually come from a backlog that is too small or an output-buffer limit that is too low (next topic).
Monitoring
redis-cli INFO stats | grep -E 'sync_full|sync_partial'
# sync_full:3 <- full syncs served by this primary since startup
# sync_partial_ok:41 <- successful partial resyncs
# sync_partial_err:1 <- partial requests that had to become full syncs
A growing sync_partial_err or sync_full without new replicas means replicas are losing their place in the stream.
Key Takeaways
- A replica always asks for a partial resync (
PSYNC id offset). The primary decides. - Partial resync needs a known replication ID (current or previous) and an offset still in the backlog.
- Full resync means fork, snapshot and transfer of the whole dataset, then the writes made in between.
- Diskless sync is the default. On Redis 8.0+ it uses an RDB channel, so the replica, not the primary, buffers the stream during the transfer.
- Watch
sync_fullandsync_partial_err. Unexplained growth means a sync loop or an undersized backlog.
Practice
- (Beginner) Run
CLIENT KILL TYPE masteron a replica and read both logs. Then restart the replica container and compare: which kind of resync happens in each case, and why? (Remember that the replica's dataset and replication ID live only in memory unless it has persistence.)- (Intermediate) Load 1 GB of data on a primary, attach a new replica with
repl-diskless-sync yes, then withno. For each run, recordlatest_fork_usec, the time untilmaster_link_status:up, and the primary's peakused_memory.- (Interview) "After a 2-second network glitch, all replicas did a full resync of a 40 GB dataset and the primary's latency spiked. Why, and what would you change?" Hint: estimate how many bytes of replication stream 2 seconds of traffic produce and compare that with the default backlog size.
Replication Backlog
A partial resync can only send what the primary still has. The replication backlog is the place where it keeps the recent stream: a fixed-size circular buffer of the most recent bytes sent to replicas. Its size decides how long a replica can be disconnected and still catch up cheaply.
How it works
replication stream offsets ─────────────────────────────────────────────────▶
backlog (repl-backlog-size, e.g. 1 MB)
┌──────────────────────────────────────────┐
overwritten│ first_byte_offset master_repl_offset
◀────────────┤ 204401 ......................... 220944 │
└──────────────────────────────────────────┘
replica asks for offset 210000 ──▶ inside the window ──▶ +CONTINUE (partial)
replica asks for offset 150000 ──▶ already overwritten ──▶ +FULLRESYNC (full)
The backlog is allocated when the first replica connects. If no replica is connected for repl-backlog-ttl seconds (default 3600), a primary frees it. Replicas never free their own backlog, because they may be promoted and then have to serve partial resyncs to the other replicas.
redis-cli INFO replication | grep backlog
# repl_backlog_active:1
# repl_backlog_size:1048576 <- configured size (1 MB default)
# repl_backlog_first_byte_offset:204401 <- oldest offset still available
# repl_backlog_histlen:16544 <- bytes currently stored
Backlog versus output buffer
These two buffers are easy to confuse:
| Replication backlog | Replica output buffer | |
|---|---|---|
| Where | One per primary, shared by all replicas | One per connected replica, on the primary |
| Used when | The replica is disconnected and reconnects | The replica is connected but reads slowly |
| Limit | repl-backlog-size | client-output-buffer-limit replica 256mb 64mb 60 (default: disconnect at 256 MB, or above 64 MB for 60 s) |
| When the limit is reached | Old data is overwritten, so later reconnections need a full sync | The primary drops the replica's connection |
A replica that is slow but still connected is served from its output buffer, not the backlog. In a test, a replica was frozen with docker pause while 220 KB of writes arrived. It stayed online and caught up from its buffer after being resumed, even though the backlog was only 16 KB. Only when its connection was dropped while it was behind did it need the backlog. That backlog no longer held its offset, so the result was:
* Unable to partial resync with replica 172.19.0.4:6379 for lack of backlog (Replica request was: 1040186).
* Replica 172.19.0.4:6379 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
The same reasoning explains the sync loop. During a classic full sync, the primary buffers new writes in the replica's output buffer. If the transfer takes long and the writes exceed the limit, the primary drops the replica, which then starts a new full sync, and the cycle repeats. The fix is a larger client-output-buffer-limit replica, or the RDB channel flow (Redis 8.0+, previous topic), which moves this buffering to the replica.
Sizing the backlog
The backlog must cover the longest disconnection you want to survive without a full sync:
backlog size >= write stream rate (bytes/s) × longest expected disconnection (s) × safety factor
Measure the stream rate from the offset:
import time
import redis
r = redis.Redis(host="primary", decode_responses=True)
def stream_rate(sample_s: float = 60.0) -> float:
"""Average replication stream throughput in bytes per second."""
start = r.info("replication")["master_repl_offset"]
time.sleep(sample_s)
end = r.info("replication")["master_repl_offset"]
return (end - start) / sample_s
rate = stream_rate()
outage_s = 60 # survive a 1-minute network partition or replica restart
factor = 2 # headroom for peak traffic
size = int(rate * outage_s * factor)
print(f"{rate / 1024:.0f} KB/s -> repl-backlog-size {size // (1024 * 1024) + 1}mb")
# e.g. 900 KB/s -> repl-backlog-size 106mb
At peak, 1 MB, the default, covers only about one second of a busy primary. Values of 100 MB to a few GB are common in production. Measure at peak traffic, not on average.
redis-cli CONFIG SET repl-backlog-size 128mb # takes effect immediately, keeps existing content
redis-cli CONFIG REWRITE # persist it to redis.conf
A replica that restarts loses its replication ID and offset unless it can reload them. Since Redis 4.0, a replica that shuts down cleanly stores them in its RDB file, so it can attempt a partial resync after a restart if it has persistence enabled and the backlog still covers the gap.
Key Takeaways
- The backlog is a circular buffer of recent stream bytes. It enables partial resyncs after disconnections.
- The default 1 MB is too small for most production primaries. Size it from the measured stream rate at peak times the longest outage to survive.
- A connected but slow replica uses its output buffer, not the backlog. Exceeding
client-output-buffer-limit replicadrops the replica. - Backlog overflow leads to full syncs, and output-buffer overflow during a full sync can lead to a sync loop.
Practice
- (Beginner) Set
repl-backlog-sizeto 16 KB on a test primary. Pause a replica (docker pause), write about 50 MB withredis-benchmark -t set -n 300000 -d 100, drop its link withCLIENT KILL TYPE replicaon the primary, and unpause it. (With only a few hundred KB, the data still in the TCP buffers is enough for a partial resync.) Confirmsync_partial_errincreased. Then repeat with 64 MB.- (Intermediate) Write a script that reads
master_repl_offsetevery second for 10 minutes during a load test and prints the backlog size needed to survive a 30-second outage at the 99th-percentile write rate.- (Interview) "Explain the difference between
repl-backlog-sizeandclient-output-buffer-limit replica. Which one would you raise if replicas keep restarting full syncs that never finish?" Hint: ask whether the replica is disconnected before or during the transfer, and check the primary's log for the reason it closed the connection.
Read-Only Replicas
Replicas hold the same data as the primary, so they can also answer reads. That is an easy way to multiply read capacity, but every replica read may be slightly out of date. This topic covers what replicas accept, what they return during trouble, and when reading from them is safe.
Read-only by default
redis-cli -h replica1 SET x 1
# (error) READONLY You can't write against a read only replica.
replica-read-only yes (the default) refuses writes from clients. With no, writes are accepted but stay local: they are not sent to the primary or other replicas, and the next full sync erases them. Writable replicas are a historical feature for storing temporary computed results. In practice they cause confusion, and they are best avoided.
Read-only does not mean harmless. Administrative commands such as CONFIG, DEBUG, FLUSHALL (refused as a write) or SHUTDOWN are governed by ACLs, not by the read-only flag. Expose replicas with the same ACLs as the primary.
Replication lag and stale reads
A replica is always slightly behind. In a healthy setup the lag is usually under a millisecond to a few milliseconds, but it grows during full syncs, network trouble or heavy write bursts. Typical consequences:
| Scenario | What goes wrong |
|---|---|
| Read-your-writes | A user updates their profile (to the primary), then reloads the page (read from a replica) and sees the old value |
| Monotonic reads | Two requests go to two replicas with different lag, so data appears to "go back in time" |
| Counters and locks | Reading a rate limiter or lock state from a replica gives decisions based on old values |
Strategies:
- Read from the primary for data the same user has just written (session-sticky routing, or "read from primary for N seconds after a write").
- Send only lag-tolerant reads to replicas: leaderboards, catalog data, analytics.
- Measure lag as the difference between the primary's
master_repl_offsetand each replica's acknowledged offset (offset=in the primary'sINFO replication), not withlag=, which counts seconds since the last acknowledgment.
When the link is down
| Setting | Behavior while the link to the primary is down or the first sync is running |
|---|---|
replica-serve-stale-data yes (default) | Keeps answering with whatever data it has, which may be old |
replica-serve-stale-data no | Answers most data commands with MASTERDOWN Link with MASTER is down and replica-serve-stale-data is set to 'no'. (INFO, PING, REPLICAOF and a few others still work) |
Choose no when stale data is worse than an error, for example for configuration flags or permissions.
Sending reads to replicas from a client
import random
import redis
primary = redis.Redis(host="primary", decode_responses=True)
replicas = [redis.Redis(host=h, decode_responses=True) for h in ("replica1", "replica2")]
def write_profile(user_id: int, name: str) -> None:
primary.hset(f"user:{user_id}", "name", name)
def read_profile(user_id: int, just_written: bool = False) -> dict:
# Read-your-writes: go to the primary right after this user wrote
node = primary if just_written else random.choice(replicas)
return node.hgetall(f"user:{user_id}")
write_profile(42, "Alice")
print(read_profile(42, just_written=True)) # always current
print(read_profile(42)) # may briefly be stale
Sentinel-aware clients return a replica connection directly (slave_for in 7.2), and cluster clients can route reads to replicas with the READONLY command (7.3). Managed services often expose a separate "reader endpoint" that load-balances across replicas.
Replicas and memory
replica-ignore-maxmemory yes (the default) means a replica never evicts keys on its own. It relies on the primary's DEL commands, so its dataset stays identical. Give replicas at least as much memory as the primary, plus room for replication buffers. A replica with a smaller maxmemory setting does not evict, so it can run out of memory.
Key Takeaways
- Replicas refuse client writes by default (
READONLYerror). Writable replicas hold local, temporary changes and should be avoided. - Replica reads can be stale. Route read-your-writes and decision-making reads (locks, rate limits) to the primary.
replica-serve-stale-data noturns stale reads during link loss intoMASTERDOWNerrors.- Measure lag in bytes (offset difference), and size replicas at least as large as the primary.
Practice
- (Beginner) Try
SET,DEL,CONFIG SET maxmemory 1gbandFLUSHALLon a replica. Which succeed? Explain why theCONFIGchange does not reach the primary.- (Intermediate) Pause a primary for 30 seconds (
docker pause) withrepl-timeout 5on the replica. Read a key from the replica withreplica-serve-stale-dataset toyes, thenno, and record the replies andmaster_link_status.- (Interview) "We added three read replicas to cut the load on the primary, and users now report that their changes sometimes disappear after saving. What is happening, and how do you fix it without removing the replicas?" Hint: consider which requests follow a write, and what routing rule could guarantee read-your-writes.
WAIT and Write Consistency
Asynchronous replication means an OK from the primary says nothing about the replicas. Some writes, such as a payment status or an issued license, are important enough that you want to know they exist on more than one machine before you answer the user. WAIT gives that confirmation for each write. It does not make Redis strongly consistent, and it is important to understand why.
WAIT
WAIT <numreplicas> <timeout-ms> blocks the current client until all its previous writes have been acknowledged by at least numreplicas replicas, or until the timeout. It returns the number of replicas that acknowledged. A timeout of 0 waits forever.
# Sample run (Redis 8.10): primary with two replicas
redis-cli SET order:1 paid
redis-cli WAIT 2 1000
# (integer) 2 <- both replicas have order:1
# Same, with one replica frozen
redis-cli SET order:2 paid
redis-cli WAIT 2 1000
# (integer) 1 <- returned after 1000 ms: only one replica confirmed
WAIT does not fail or roll back when the count is not reached. The write has already happened on the primary. The caller must decide what a lower number means.
WAITAOF (Redis 7.2+)
WAIT confirms that replicas received the write in memory, not that anything reached a disk. WAITAOF <numlocal> <numreplicas> <timeout> waits until the write has been fsynced to the AOF (chapter 6.2) locally (numlocal = 0 or 1) and on numreplicas replicas. It returns two counts:
redis-cli SET order:4 paid
redis-cli WAITAOF 1 1 1000
# 1) (integer) 1 <- fsynced to the primary's AOF
# 2) (integer) 0 <- no replica has AOF enabled, so none can confirm
# With appendonly no on the primary:
# (error) ERR WAITAOF cannot be used when numlocal is set but appendonly is disabled.
Why WAIT is not strong consistency
- If the primary fails after the write but before the replicas acknowledge, the client sees a timeout or an error. The write may or may not survive, depending on which replica is promoted.
- Sentinel and Cluster promote the "best" replica by offset among those reachable, but there is no guarantee that this is a replica that acknowledged the write.
- Writes from other clients are not affected. Only the calling connection waits.
The Redis documentation states it directly: WAIT does not make Redis a strongly consistent store. It shrinks the window of loss for the writes you choose, and Redis remains an availability-first (AP-leaning) system. Use a consensus-based database when you need linearizable writes.
min-replicas-to-write: refusing writes when replicas are missing
A primary can refuse writes when it does not have enough healthy replicas:
min-replicas-to-write 1 # need at least 1 replica...
min-replicas-max-lag 10 # ...that acknowledged within the last 10 seconds
# Sample run (Redis 8.10): min-replicas-to-write 2, one of two replicas frozen
redis-cli INFO replication | grep min_slaves
# min_slaves_good_slaves:1
redis-cli SET order:5 paid
# (error) NOREPLICAS Not enough good replicas to write.
This is a global guard, not a per-write confirmation. It does not check that a specific write reached a replica. It limits how long an isolated primary keeps accepting writes that will be lost: no more than about min-replicas-max-lag seconds after it loses its replicas. Its main use is limiting split-brain damage (7.2, "Split-Brain Scenarios"). The cost is availability: losing the replicas also stops all writes.
| Tool | Scope | Guarantees | Cost |
|---|---|---|---|
| Nothing (default) | — | Write is in the primary's memory | Fastest |
WAIT n t | Per client, after chosen writes | n replicas received it, or you know they did not | Adds one replication round trip to those writes |
WAITAOF l n t | Per client | Write is fsynced locally and/or on n replicas | Adds fsync latency (up to ~1 s with everysec) |
min-replicas-to-write | Whole server | Primary stops accepting writes when replicas are missing or lagging | Writes fail when replicas are down |
Example: confirm critical writes in Python
import redis
r = redis.Redis(host="primary", decode_responses=True)
def mark_paid(order_id: str) -> bool:
"""Return True only when the write is on the primary and at least one replica."""
r.hset(f"order:{order_id}", mapping={"status": "paid"})
acked = r.wait(1, 500) # num_replicas=1, timeout=500 ms
if acked < 1:
# The write is on the primary but unconfirmed elsewhere.
# Decide: report "pending", retry the WAIT, or alert. Do not report success yet.
return False
return True
Key Takeaways
WAIT n timeoutblocks the caller until n replicas acknowledge its previous writes, and returns how many did. It never rolls back.WAITAOF(7.2+) waits for fsync to the AOF locally and on replicas.- Neither command turns Redis into a strongly consistent store. A failover can still lose an acknowledged write.
min-replicas-to-writeplusmin-replicas-max-lagstops writes on a primary that has lost its replicas, which bounds split-brain loss at the cost of write availability.
Practice
- (Beginner) With one primary and two replicas, measure the latency of 1,000
SETcommands with and without a followingWAIT 2 100. Then freeze one replica and repeat.- (Intermediate) Configure
min-replicas-to-write 1andmin-replicas-max-lag 5. Freeze both replicas, write once per second, and record how many seconds pass beforeNOREPLICASappears. Explain the number.- (Interview) "If I call
WAIT 2 0after every write, is my Redis setup now strongly consistent?" Hint: consider who chooses the replica to promote after a failure, and what happens to a client whoseWAITis interrupted by the primary crashing.
<a id="72-redis-sentinel"></a>
7.2 Redis Sentinel
This subchapter covers Redis Sentinel, the high-availability system for non-clustered Redis. It explains how a group of Sentinel processes monitors a primary, how they agree that it is down, how they choose and promote a replica, how clients find the current primary, and how network partitions can still lose acknowledged writes.
Sentinel Architecture
Replication gives you copies, but someone still has to notice that the primary is dead, choose a replica, promote it, re-point the other replicas, and tell the clients. Doing this by hand at 3 a.m. takes minutes. Sentinel automates it. It is a separate mode of the Redis binary, so there is no extra software to install, and it watches one or more primary-replica groups.
A single watcher is not enough: if it fails, nobody fails over. If it is the one cut off by the network, it would "fail over" a healthy primary. Sentinel is therefore designed to run as a group of at least three processes on independent machines that must agree before acting. The analogy is a jury rather than a single judge.
Responsibilities
| Role | What Sentinel does |
|---|---|
| Monitoring | Checks that the primary and replicas respond |
| Notification | Publishes events (and can run scripts) when something changes |
| Automatic failover | Promotes a replica and reconfigures the others when the primary fails |
| Configuration provider | Tells clients the current primary's address |
Typical deployment
host A (zone 1) host B (zone 2) host C (zone 3)
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ Redis primary │ │ Redis replica │ │ Redis replica │
│ Sentinel 1 │ │ Sentinel 2 │ │ Sentinel 3 │
└───────┬────────┘ └───────┬────────┘ └───────┬────────┘
└───────── each Sentinel monitors all Redis servers ──┘
and talks to the other Sentinels
Clients ──▶ ask any Sentinel "where is mymaster?" ──▶ connect to that address
Sentinels can share hosts with the Redis servers (as above) or run on application servers. What matters is that they fail independently: three Sentinels in one virtual machine are one Sentinel as far as availability is concerned.
Auto-discovery
You configure each Sentinel with only the primary's address. It finds everything else by itself:
- Replicas: from the primary's
INFO replication, refreshed every 10 seconds (every second during a failover). - Other Sentinels: every Sentinel publishes a "hello" message every 2 seconds on the
__sentinel__:helloPub/Sub channel of every monitored server. Sentinels watching the same server see each other's messages. - Liveness: every Sentinel sends
PINGto every known server and Sentinel once per second.
Sentinel rewrites its own config file to store what it learns (replicas, other Sentinels, epochs), so the file must be writable. Otherwise it refuses to start:
# Sentinel config file /data/sentinel.conf is not writable: Permission denied. Exiting...
Minimal configuration
# sentinel.conf (one per Sentinel, identical at the start)
port 26379
sentinel monitor mymaster 10.0.1.10 6379 2 # name, primary ip, port, quorum
sentinel down-after-milliseconds mymaster 5000 # default 30000
sentinel failover-timeout mymaster 60000 # default 180000
sentinel parallel-syncs mymaster 1 # default 1
sentinel auth-user mymaster sentinel-user # if the Redis servers require auth
sentinel auth-pass mymaster s3cr3t
# In containers/NAT, announce a reachable address, and use hostnames if IPs change
# sentinel announce-ip 203.0.113.10
# sentinel resolve-hostnames yes
# sentinel announce-hostnames yes
Start it with redis-sentinel /etc/redis/sentinel.conf or redis-server /etc/redis/sentinel.conf --sentinel. The log marks Sentinel lines with X (chapter 1.2). A Sentinel can monitor many primaries, each under its own name.
After startup, the file contains what the Sentinel learned (from the lab used in this subchapter):
# Generated by CONFIG REWRITE
sentinel myid 410d1b3312b616a9df46858347eda545b2b8e033
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 172.19.0.3 6379
sentinel known-replica mymaster 172.19.0.4 6379
sentinel known-sentinel mymaster 172.19.0.6 26379 7596481da0a35dc4b06cd61576b79a5e666257a7
sentinel known-sentinel mymaster 172.19.0.7 26379 41a69db242ebda3c7e09f5d3cff46e768ac33458
Inspecting Sentinel
| Command (on port 26379) | Returns |
|---|---|
SENTINEL MASTERS / SENTINEL MASTER mymaster | State of monitored primaries: address, flags, quorum, timeouts |
SENTINEL REPLICAS mymaster | Known replicas and their state |
SENTINEL SENTINELS mymaster | The other Sentinels |
SENTINEL GET-MASTER-ADDR-BY-NAME mymaster | Current primary's address (what clients use) |
SENTINEL CKQUORUM mymaster | Whether enough Sentinels are reachable to fail over |
SENTINEL FAILOVER mymaster | Forces a failover now, without asking the other Sentinels |
SENTINEL RESET <pattern> | Forgets learned replicas/Sentinels (after removing nodes) |
Sentinel and Cluster are separate solutions
| Sentinel | Redis Cluster (7.3) | |
|---|---|---|
| Sharding | No: one primary holds all data | Yes: data split across primaries |
| Failover | Done by the Sentinel processes | Done by the cluster nodes themselves |
| Client | Any client plus Sentinel discovery | Cluster-aware client |
| Multi-key operations | Unrestricted | Only within one hash slot |
| Fits when | The data fits in one server's memory and write throughput | You need more memory or write throughput than one server |
Key Takeaways
- Sentinel automates monitoring, failover and primary discovery for non-clustered Redis.
- Run at least three Sentinels on hosts that fail independently.
- Sentinels discover replicas through
INFOand each other through the__sentinel__:hellochannel. Only the primary is configured. - Sentinel rewrites its config file, so the file must be writable.
- Use Sentinel when one primary is enough for capacity. Use Cluster when you need sharding.
Practice
- (Beginner) Build the lab: one primary, two replicas and three Sentinels in Docker (write each
sentinel.confinside the container so it is writable). RunSENTINEL MASTER mymasterand identifynum-slaves,num-other-sentinelsandquorum.- (Intermediate) Subscribe to
__sentinel__:helloon the primary withredis-cli SUBSCRIBEand decode one message. Which fields does a Sentinel announce about itself and about the primary?- (Interview) "Why not run one Sentinel next to each of our two Redis servers?" Hint: consider what each Sentinel can conclude when the link between the two hosts breaks, and how many votes a failover needs.
Monitoring and Quorum
A failover is expensive and disruptive, so Sentinel must avoid reacting to a single slow reply or to a Sentinel that is itself cut off. It uses two levels of "down" and two different thresholds, and confusing the thresholds is a classic mistake.
Subjectively down (SDOWN)
A Sentinel marks a server SDOWN when it gets no valid reply to PING for down-after-milliseconds. Valid replies are +PONG, -LOADING and -MASTERDOWN. Any other reply, or none, counts as failure. SDOWN is one Sentinel's opinion. For a replica or another Sentinel, SDOWN only changes its state. For a primary, it is the first step towards a failover.
Objectively down (ODOWN)
When a Sentinel sees the primary SDOWN, it asks the other Sentinels (SENTINEL IS-MASTER-DOWN-BY-ADDR). If at least quorum Sentinels, itself included, report it down, it marks the primary ODOWN. Only primaries can be ODOWN.
Quorum versus majority
| Threshold | Set by | Needed for |
|---|---|---|
| Quorum | sentinel monitor ... <quorum> | Agreeing that the primary is down (ODOWN) |
| Majority | Always floor(N/2) + 1 of all known Sentinels | Electing the leader that is allowed to perform the failover |
Both are needed. With 5 Sentinels and quorum 2, two Sentinels can declare the primary down, but the failover still needs 3 votes. With quorum 5, all five must agree that the primary is down, so a single Sentinel being unreachable prevents failover.
| Sentinels (N) | Majority | Sensible quorum | Sentinels that can fail with failover still possible |
|---|---|---|---|
| 1 | 1 | 1 | 0 (and a lone Sentinel cannot tell its own isolation from a dead primary) |
| 2 | 2 | 1 | 0: losing one leaves no majority |
| 3 | 2 | 2 | 1 |
| 5 | 3 | 3 | 2 |
A lower quorum detects failures faster under partial visibility, but makes false positives more likely. A quorum equal to the majority is the usual choice. SENTINEL CKQUORUM checks both:
redis-cli -p 26379 SENTINEL CKQUORUM mymaster
# OK 3 usable Sentinels. Quorum and failover authorization can be reached
Choosing down-after-milliseconds
| Value | Effect |
|---|---|
| Low (1-5 s) | Fast failover, but GC-like stalls, a long fork(), a slow KEYS command or a network hiccup can cause unnecessary failovers |
| High (30 s, the default) | Fewer false failovers, but up to 30 s of write unavailability before the failover even starts |
The value must be larger than the longest legitimate pause of the primary. Check latest_fork_usec (chapter 6.1) and the slow log (chapter 12.1) before lowering it.
TILT mode
Sentinel relies on the system clock to measure timeouts. If it notices that time jumped, or that it was frozen (for example, the VM was paused or the process was blocked), it enters TILT mode for 30 seconds: it keeps monitoring but does not act. The log shows +tilt and -tilt. Frequent TILTs point to an overloaded host or clock problems.
Watching state changes
Every state change is published on the Sentinel's own Pub/Sub channels, named after the event:
redis-cli -p 26379 PSUBSCRIBE '*'
# "+sdown" "master mymaster 172.19.0.2 6379"
# "+odown" "master mymaster 172.19.0.2 6379 #quorum 3/2"
import redis
sentinel = redis.Redis(host="sentinel1", port=26379, decode_responses=True)
pubsub = sentinel.pubsub()
pubsub.psubscribe("+sdown", "-sdown", "+odown", "-odown", "+switch-master", "+tilt")
for msg in pubsub.listen():
if msg["type"] == "pmessage":
# e.g. +odown -> master mymaster 172.19.0.2 6379 #quorum 3/2
print(f'{msg["channel"]:>14} -> {msg["data"]}') # forward to your alerting system
Key Takeaways
- SDOWN is one Sentinel's opinion after
down-after-milliseconds. ODOWN needsquorumSentinels to agree. - Quorum decides detection. A majority of all Sentinels is always needed to authorize the failover.
- Use an odd number of Sentinels (3 or 5) with quorum equal to the majority.
- Set
down-after-millisecondsabove the longest normal pause of the primary, including fork time. - Sentinel publishes every event on Pub/Sub channels named after it. Subscribe for alerting.
Practice
- (Beginner) In the lab, stop one Sentinel and run
SENTINEL CKQUORUM. Stop a second one and run it again. Explain both replies.- (Intermediate) Set
down-after-millisecondsto 1000 and runDEBUG SLEEP 2(on a test server with debug commands enabled) on the primary. Does a failover happen? Repeat with 5000.- (Interview) "We have 5 Sentinels and set quorum to 1 to fail over as fast as possible. What could go wrong?" Hint: separate who can declare the primary down from who can authorize a failover, and think about one Sentinel with a bad network link.
Automatic Failover
Once the primary is ODOWN, the Sentinels must make sure that exactly one of them performs the failover, choose the best replica, promote it, and make every other server and Sentinel agree on the new layout. This topic follows those steps using the log of a real failover.
Step 1: elect a leader
A Sentinel that sees ODOWN increments the current epoch (a logical clock shared by all Sentinels) and asks the others for their vote. Each Sentinel votes once per epoch, for the first candidate that asks. A candidate that gets a majority (and at least the quorum) becomes the leader for that epoch. If no one wins, for example because of split votes, they retry later with a higher epoch.
Step 2: choose the replica
The leader filters out replicas that are unsuitable, then ranks the rest:
- Exclude: replicas that are SDOWN or disconnected, that have not answered
INFOrecently, or whose link to the old primary has been down for more than about 10 timesdown-after-milliseconds(their data is considered too old). Also exclude replicas withreplica-priority 0. - Lowest
replica-prioritywins (default 100). - On a tie, the highest replication offset: the replica that received the most data.
- On a tie, the lexicographically smallest run ID, an arbitrary but deterministic rule.
Use replica-priority to prefer replicas in the same zone as the applications, or to set it to 0 for replicas that should never become primary, such as a backup or analytics replica.
Step 3: promote and reconfigure
The leader sends REPLICAOF NO ONE to the chosen replica and waits until it reports role:master. Then it reconfigures the other replicas with REPLICAOF <new-primary>, parallel-syncs at a time. Each of them may need a resync, during which it does not serve fresh data, so the default of 1 keeps most replicas available for reads.
Step 4: propagate the new configuration
The new layout is tagged with the epoch in which it was decided, the config epoch. The leader announces it in its hello messages, and every Sentinel accepts a configuration with a higher epoch than the one it knows. This is how all Sentinels, including ones that were partitioned during the failover, converge on the same primary. When the old primary comes back, Sentinels see that it claims to be primary with an outdated configuration and turn it into a replica of the new primary (+convert-to-slave).
A real failover, step by step
Lab: one primary (172.19.0.2), two replicas, three Sentinels, down-after-milliseconds 5000, quorum 2. The primary was stopped with docker stop. Log of the Sentinel that won the election (Redis 8.10):
09:01:30.944 # +sdown master mymaster 172.19.0.2 6379 <- 5 s without valid reply
09:01:30.999 # +odown master mymaster 172.19.0.2 6379 #quorum 3/2 <- 3 Sentinels agree
09:01:30.999 # +new-epoch 1
09:01:30.999 # +try-failover master mymaster 172.19.0.2 6379
09:01:31.030 * 41a69db2... voted for 41a69db2... 1 <- another Sentinel voted for itself
09:01:31.049 * 410d1b33... voted for 7596481d... 1 <- this Sentinel got 2 of 3 votes
09:01:31.102 # +elected-leader master mymaster 172.19.0.2 6379
09:01:31.192 # +selected-slave slave 172.19.0.3:6379 ... <- best replica chosen
09:01:31.192 * +failover-state-send-slaveof-noone slave 172.19.0.3:6379 ...
09:01:32.068 # +promoted-slave slave 172.19.0.3:6379 ... <- it now reports role:master
09:01:32.121 * +slave-reconf-sent slave 172.19.0.4:6379 ... <- REPLICAOF sent to the other replica
09:01:33.094 * +slave-reconf-done slave 172.19.0.4:6379 ...
09:01:33.171 # +failover-end master mymaster 172.19.0.2 6379
09:01:33.171 # +switch-master mymaster 172.19.0.2 6379 172.19.0.3 6379 <- clients listen for this
Two Sentinels started a candidacy in the same epoch, and the votes split 2 to 1. Because only a majority wins, there was still exactly one leader. Total time: 5 s of detection plus about 2.2 s of failover. In production, detection time (down-after-milliseconds) usually dominates.
Timeouts and retries
failover-timeout (default 3 minutes) has several uses:
- A Sentinel does not try to fail over the same primary again within 2 × failover-timeout of a previous attempt. In the lab log:
Next failover delay: I will not start a failover before Tue Nov 10 09:03:31 2026, exactly 120 s after the attempt with a 60 s timeout. - A failover in progress is aborted if the promotion does not happen within this time.
- It caps the time allowed to reconfigure the replicas.
Hooks
# Called with event type and description for every warning-level event (alerting)
sentinel notification-script mymaster /var/redis/notify.sh
# Called after a failover with: <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# Use it to update a DNS name, a virtual IP or a load balancer
sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
By default (sentinel deny-scripts-reconfig yes), these paths cannot be changed at runtime with SENTINEL SET. This prevents a client with access to Sentinel from turning a script setting into remote code execution.
Planned failover
For maintenance, SENTINEL FAILOVER mymaster promotes a replica immediately, without waiting for ODOWN or asking the other Sentinels for agreement. It selects the replica with the same rules as above, but it does not stop the primary from accepting writes first. A write that arrives just before the switch may not have reached the new primary. To make a planned switch lossless, pause writes on the primary (CLIENT PAUSE <ms> WRITE), check that the replicas have reached its offset, and then trigger the failover.
Key Takeaways
- Failover is done by a single leader Sentinel, elected by a majority in a numbered epoch.
- Replica choice: exclude unhealthy and stale replicas and priority 0, then lowest
replica-priority, highest offset, smallest run ID. - The new configuration carries a higher config epoch, which makes all Sentinels converge and demotes the old primary when it returns.
failover-timeoutalso sets the retry delay (2×).parallel-syncslimits how many replicas resync at once.- Detection time (
down-after-milliseconds) usually dominates total failover time.
Practice
- (Beginner) Stop the primary in the lab and match your Sentinel log lines to the four steps above. How long did each phase take?
- (Intermediate) Give one replica
replica-priority 10and anotherreplica-priority 0. Fail over twice (SENTINEL FAILOVER), restarting the old primary in between, and record which replica gets promoted each time. Then restart the old primary and look for+convert-to-slave.- (Interview) "During a failover, how does Sentinel make sure two Sentinels do not promote two different replicas?" Hint: describe epochs, one vote per epoch, and the majority requirement, and what happens when votes split.
Client Discovery via Sentinel
After a failover, the primary has a new address. Hard-coding the primary's IP in the application would defeat the purpose, so clients connect to the Sentinels first and ask them where the primary is. A Sentinel-aware client hides this behind a normal connection object.
The discovery protocol
- Connect to the first reachable Sentinel from a list (try them in turn).
- Ask
SENTINEL GET-MASTER-ADDR-BY-NAME mymaster, which returns the primary's IP and port. - Connect to that address and verify with
ROLEthat it really is a primary. A Sentinel that was partitioned may answer with an outdated address. - On connection errors, or on
READONLYerrors (the server was demoted), discard the connections and repeat from step 1. - Optionally subscribe to
+switch-masteron a Sentinel to react to failovers immediately instead of waiting for errors.
After a failover, Sentinel also disconnects the clients of the servers it reconfigured, so that they reconnect and rediscover the primary instead of keeping a connection to a server whose role changed.
redis-cli -p 26379 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster
# 1) "172.19.0.3"
# 2) "6379"
redis-cli -p 26379 SENTINEL REPLICAS mymaster # replica addresses for read traffic
Example: redis-py
from redis.sentinel import Sentinel
import redis
# Several Sentinels: any one of them can answer
sentinel = Sentinel(
[("sentinel1", 26379), ("sentinel2", 26379), ("sentinel3", 26379)],
socket_timeout=0.5, # timeout for talking to Sentinels
decode_responses=True,
# sentinel_kwargs={"password": "..."}, # if the Sentinels themselves require auth
)
print(sentinel.discover_master("mymaster")) # ('172.19.0.3', 6379)
print(sentinel.discover_slaves("mymaster")) # [('172.19.0.4', 6379)]
# Connection objects that re-discover the address on every new connection
primary = sentinel.master_for("mymaster", socket_timeout=0.5)
replica = sentinel.slave_for("mymaster", socket_timeout=0.5)
primary.set("k", "v")
print(replica.get("before")) # reads may be slightly stale
try:
replica.set("x", 1)
except redis.exceptions.ReadOnlyError as e:
print("ReadOnlyError:", e) # You can't write against a read only replica.
This was run with redis-py 8.1 against the lab after a failover. The output is shown in the comments.
Handling the failover window
During a failover there is a period, typically down-after-milliseconds plus a few seconds, when writes fail. The client library rediscovers the primary, but the application decides what to do with the failed commands:
import time
import redis
from redis.backoff import ExponentialBackoff
from redis.retry import Retry
from redis.sentinel import Sentinel
sentinel = Sentinel([("sentinel1", 26379), ("sentinel2", 26379), ("sentinel3", 26379)])
primary = sentinel.master_for(
"mymaster",
socket_timeout=1.0,
# Retry connection errors and timeouts with backoff, rediscovering each time
retry=Retry(ExponentialBackoff(cap=2.0, base=0.1), retries=8),
retry_on_error=[redis.exceptions.ConnectionError, redis.exceptions.ReadOnlyError],
)
# Safe to retry: SET with an explicit value is idempotent
primary.set("session:abc", "data", ex=3600)
# Careful: retrying INCR after a timeout may count twice (chapter 8.2, idempotency keys)
Alternatives to client-side discovery
| Approach | How | Trade-off |
|---|---|---|
| Sentinel-aware client (above) | Library asks Sentinels | Most robust; needs library support |
client-reconfig-script + DNS or virtual IP | Script moves a DNS name or VIP to the new primary | Works with any client; DNS caching and TTLs delay the switch |
Proxy (HAProxy with a ROLE check, or a Redis proxy) | Proxy routes to whichever server reports role:master | One more hop and component; clients stay simple |
| Managed service endpoint | Provider updates a stable primary endpoint | Least work; failover behavior is the provider's |
Key Takeaways
- Clients never hard-code the primary. They ask any Sentinel (
GET-MASTER-ADDR-BY-NAME) and verify withROLE. - On connection or
READONLYerrors, discard connections and rediscover. Subscribing to+switch-masterspeeds this up. - Sentinel disconnects clients of servers it reconfigures, which forces them to rediscover.
- Libraries hide discovery, but the application must handle commands that fail during the failover window, and retry only idempotent ones.
Practice
- (Beginner) Run the redis-py example against the lab, stop the primary, and keep calling
primary.set()once per second in a loop with atry/except. Record how many calls failed and how long the gap was.- (Intermediate) Write a minimal discovery function using only a plain Redis client: query a list of Sentinels in turn, connect to the returned address, verify with
ROLE, and cache the result until an error occurs.- (Interview) "Our application uses a DNS name that a script updates after Sentinel fails over. After a failover, some instances kept writing to the old primary for several minutes. Why?" Hint: consider DNS caching in the runtime and OS, connection pools that never reconnect, and what the old primary does when it comes back.
Split-Brain Scenarios
A split brain happens when two servers both act as primary at the same time and accept writes. Sentinel's majority rule prevents two failovers. It cannot stop an old primary that is cut off from the Sentinels from still accepting writes from clients that can reach it. When the partition heals, those writes are discarded. This is the most important limitation of Redis high availability to understand.
How it happens
network partition
┊
┌──────────────────┐ ┊ ┌───────────────────────────────────────────┐
│ old primary │ ┊ │ Sentinel 1, 2, 3 replica A replica B │
│ still role:master│ ┊ │ majority: primary unreachable -> ODOWN │
│ │ ┊ │ promote replica A -> new primary │
│ client X writes │ ┊ │ clients Y, Z write to replica A │
│ here: OK, OK, OK │ ┊ │ │
└──────────────────┘ ┊ └───────────────────────────────────────────┘
┊
partition heals: Sentinels demote the old primary -> REPLICAOF replica A
-> full sync -> client X's writes are flushed and lost
A real reproduction
In the lab, the primary was given a second network shared only with one client, and was then disconnected from the network of the Sentinels and replicas. The client kept writing to it once per second:
# Client side, during the partition: every write succeeded
SET cart:42 write-1 ... SET cart:42 write-15 # 15 x OK
GET cart:42 # "write-15"
# Meanwhile, the Sentinels promoted another replica (172.19.0.4).
# cart:42 does not exist there.
# Partition heals: the Sentinels turn the old primary into a replica
ROLE # "slave"
GET cart:42 # (nil): 15 acknowledged writes are gone
The old primary's log shows why:
1:S * MASTER <-> REPLICA sync: Flushing old data
1:S * Done loading RDB, keys loaded: 2, keys expired: 0.
Bounding the loss with min-replicas-to-write
The old primary cannot know that a failover happened, but it can notice that it has lost its replicas. With:
min-replicas-to-write 1
min-replicas-max-lag 10
the isolated primary starts refusing writes with NOREPLICAS once no replica has acknowledged for 10 seconds (7.1, "WAIT and Write Consistency"). The writes lost in a split brain are limited to roughly those 10 seconds instead of the whole partition. The price is that writes also stop when all replicas fail, even without a partition.
| Setting | Data lost in a split brain | Writes when all replicas are down |
|---|---|---|
min-replicas-to-write 0 (default) | Everything written to the old primary during the partition | Continue |
min-replicas-to-write 1, max-lag 10 | Up to about 10 s of writes | Stop with NOREPLICAS |
Other split-brain sources
- A primary without persistence that restarts automatically. It comes back empty, faster than the Sentinels detect the failure, and the replicas sync the empty dataset (chapter 6.3, "Running Without Persistence").
- Sentinels placed with the primary. If two of three Sentinels share the primary's side of a partition, the majority sees the primary as healthy and no failover happens. That is safe, but the clients on the other side have no primary at all.
- Clients with stale discovery. A client that keeps an old connection or a cached address writes to the old primary until it gets an error.
Design guidelines
- Spread Sentinels across failure domains so that a majority is on the side where most clients are.
- Set
min-replicas-to-writeon primaries that hold data you cannot afford to lose silently. - Treat Redis as a store where the last few seconds of writes may be lost in rare failures. Data that must never be lost belongs in a system with consensus-based replication, or must be written to one as well.
Key Takeaways
- A partitioned old primary keeps accepting writes. When it rejoins as a replica, they are discarded by the full sync.
- Sentinel's majority prevents two simultaneous failovers, not writes to an isolated old primary.
min-replicas-to-writewithmin-replicas-max-laglimits the loss to aboutmax-lagseconds, at the cost of write availability.- Persistence-less primaries with automatic restarts can wipe replicas. Disable one or the other.
- Place Sentinels so that the majority sits with most of the clients.
Practice
- (Beginner) Reproduce the split brain from this topic with Docker networks (
docker network disconnect, plus a second network for one client). Count the lost writes.- (Intermediate) Repeat with
min-replicas-to-write 1andmin-replicas-max-lag 5on all Redis servers. How many writes were accepted by the old primary, and how many were lost?- (Interview) "Design a Sentinel deployment across two data centers so that a network split between them never results in two primaries accepting writes for more than a few seconds." Hint: consider where a tie-breaking third Sentinel should live, what each side can do without a majority, and
min-replicas-to-write.
<a id="73-redis-cluster"></a>
7.3 Redis Cluster
This subchapter covers Redis Cluster, the built-in way to spread one dataset over several primaries. It explains how keys are mapped to hash slots, how hash tags keep related keys together, how clients are redirected to the right node, how nodes talk to each other, how slots move between nodes, how the cluster fails over on its own, and which features a cluster gives up compared with a single server.
Hash Slots and Sharding
Replication copies the same data to every server, so adding replicas adds read capacity but not memory or write capacity: every replica still holds everything, and all writes still go to one primary. To grow beyond one machine, the data itself must be split (sharded) so that each primary holds only part of it. Redis Cluster does this automatically.
Why not hash(key) % number_of_nodes?
That is the obvious approach, but adding or removing a node changes the divisor, so almost every key changes owner. In a test with 100,000 keys, going from 3 to 4 nodes moved 75% of them (see 7.4, "Client-Side Sharding"). Redis Cluster adds a fixed indirection layer instead:
key ──CRC16──▶ 0..65535 ──mod 16384──▶ hash slot ──slot map──▶ node
(fixed) (changes when you reshard)
- There are always exactly 16,384 hash slots. A key's slot never changes:
slot = CRC16(key) mod 16384. - Each primary owns a subset of the slots. Moving capacity means moving whole slots from one node to another. Only the keys in those slots move.
- Every node knows the full slot-to-node map, so any node can tell a client where a key lives.
An analogy: a library with 16,384 numbered shelves. Each book always goes on the same shelf number, computed from its title. Shelves are assigned to rooms, and when a new room opens, some whole shelves are carried to it. No book ever needs a new shelf number.
16,384 is a compromise. Nodes exchange their slot ownership as a bitmap in every heartbeat, and 16,384 bits are 2 KB. More slots would make heartbeats bigger, and fewer would limit how evenly data can spread. The design targets clusters of up to about 1,000 nodes.
A cluster's layout
slots 0-5460 slots 5461-10922 slots 10923-16383
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ primary A │ │ primary B │ │ primary C │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
▼ ▼ ▼
replica A1 replica B1 replica C1
every node ◀── cluster bus (port + 10000): heartbeats, gossip, failover votes ──▶ every node
A minimal production cluster has 3 primaries, the smallest number that can form a majority for failover, each with at least one replica: 6 nodes.
Creating a cluster
# redis.conf for each node
port 6379
cluster-enabled yes
cluster-config-file nodes.conf # written and maintained by Redis itself: never edit by hand
cluster-node-timeout 15000 # ms without reply before a node is suspected (default 15000)
appendonly yes
# Six servers started with the config above, then (from any host):
redis-cli --cluster create \
10.0.0.1:6379 10.0.0.2:6379 10.0.0.3:6379 \
10.0.0.4:6379 10.0.0.5:6379 10.0.0.6:6379 \
--cluster-replicas 1 # one replica per primary
# ...
# M: 981ae2d3... 172.19.0.3:6379
# slots:[0-5460] (5461 slots) master
# 1 additional replica(s)
# M: 4a912354... 172.19.0.4:6379
# slots:[5461-10922] (5462 slots) master
# ...
# [OK] All nodes agree about slots configuration.
# [OK] All 16384 slots covered.
Every node also listens on a cluster bus port, by default the client port + 10000 (16379). Both ports must be open between all nodes.
Inspecting the cluster
redis-cli CLUSTER KEYSLOT user:1001 # (integer) 5712
redis-cli CLUSTER KEYSLOT user:1002 # (integer) 9779: a different slot, maybe a different node
redis-cli CLUSTER INFO | head -4
# cluster_state:ok
# cluster_slots_assigned:16384
# cluster_slots_ok:16384
# cluster_known_nodes:6
redis-cli CLUSTER NODES
# <id> <ip:port@bus-port> <flags> <primary-id> <ping-sent> <pong-recv> <config-epoch> <link> <slots>
# 4a91...f72 172.19.0.4:6379@16379 master - 0 1794301409381 2 connected 5461-10922
# 9560...461 172.19.0.8:6379@16379 slave 4a91...f72 0 1794301411395 2 connected
# 981a...335 172.19.0.3:6379@16379 myself,master - 0 0 1 connected 0-5460
redis-cli CLUSTER SHARDS # (7.0+) same information grouped by shard, with health and offsets
Computing the slot yourself
Clients compute the slot locally, so they can send each command straight to the right node. The algorithm is CRC-16/XMODEM, and it matches the server exactly:
def crc16(data: bytes) -> int:
"""CRC-16/XMODEM (polynomial 0x1021, initial value 0), as used by Redis Cluster."""
crc = 0
for byte in data:
crc ^= byte << 8
for _ in range(8):
crc = ((crc << 1) ^ 0x1021) if crc & 0x8000 else (crc << 1)
crc &= 0xFFFF # keep 16 bits
return crc
def hash_slot(key: str) -> int:
k = key.encode()
start = k.find(b"{")
if start != -1:
end = k.find(b"}", start + 1)
if end != -1 and end != start + 1: # non-empty hash tag: hash only the tag (next topic)
k = k[start + 1:end]
return crc16(k) % 16384
print(hash_slot("user:1001")) # 5712, same as CLUSTER KEYSLOT
print(hash_slot("foo")) # 12182
Key Takeaways
- Redis Cluster splits the keyspace into 16,384 fixed hash slots:
CRC16(key) mod 16384. - Each primary owns a set of slots. Resharding moves whole slots, never re-hashes keys.
- A minimal production cluster has 3 primaries with 1 replica each. Nodes also need the cluster bus port (+10000).
CLUSTER KEYSLOT,CLUSTER NODES,CLUSTER SHARDSandCLUSTER INFOshow the mapping and health.- Clients compute slots locally and talk to the owning node directly. There is no central router.
Practice
- (Beginner) Create a 6-node cluster with Docker and
redis-cli --cluster create. Insert 10,000 keys withredis-cli -cand compareDBSIZEon each primary.- (Intermediate) Using the Python
hash_slotfunction, compute the slots ofuser:1touser:100000and count how many fall on each of the three default slot ranges. How even is the distribution?- (Interview) "Why does Redis Cluster use 16,384 hash slots instead of consistent hashing on a ring, or instead of 65,536 slots?" Hint: think about what must fit in every heartbeat, how slot ownership is moved and agreed on, and the intended maximum cluster size.
Hash Tags for Multi-Key Operations
A Redis Cluster only runs a multi-key command (MGET, SUNIONSTORE, RENAME, a transaction or a script touching several keys) when all its keys are in the same hash slot. Otherwise, a single node could not execute it atomically. Since user:1001:cart and user:1001:profile hash to unrelated slots, you need a way to say "these keys belong together". That is a hash tag.
redis-cli -c MSET user:1001 a user:1002 b
# (error) CROSSSLOT Keys in request don't hash to the same slot
The rule
If a key contains {, followed later by }, with at least one character between them, only the part between the first { and the first } after it is hashed.
| Key | Hashed part | Slot (verified with CLUSTER KEYSLOT) |
|---|---|---|
user:1001 | user:1001 | 5712 |
{user:1001}:cart | user:1001 | 5712 |
{user:1001}:profile | user:1001 | 5712 |
foo | foo | 12182 |
foo{}bar | whole key (empty tag) | 14292 |
{}foo | whole key (empty tag) | 9500 |
foo{{bar}} | {bar (from the first { to the first }) | 4015 |
{bar | whole key (no closing brace) | 4015 |
redis-cli -c MSET '{user:1001}:cart' a '{user:1001}:profile' b # OK: same slot
redis-cli -c MGET '{user:1001}:cart' '{user:1001}:profile'
Designing with hash tags
Put the tag on the entity that your atomic operations are scoped to:
{user:1001}:profile hash
{user:1001}:cart hash
{user:1001}:sessions set -> all user 1001 data on one node,
{user:1001}:history stream multi-key ops on them are allowed
{order:77}:items / {order:77}:status -> transaction over one order
lock:{invoice:9} / fence:{invoice:9} -> lock plus fencing token in one script
The tag also works without a cluster, so using tags from day one costs nothing and makes a later move to a cluster easier.
The cost: hot slots
All keys with the same tag land on one slot, so on one node. A tag that is too coarse concentrates data and traffic:
| Tag choice | Effect |
|---|---|
{user:1001} per user | Thousands of small groups, evenly spread |
{tenant:acme} for a large tenant | One tenant's whole dataset on one node, which may not fit or may overload it |
{global} on everything | The cluster degenerates into a single node |
A slot cannot be split. If one tag's data outgrows a node, the only fix is to change the key design.
Alternatives when keys must stay spread out
- Use a client that splits multi-key commands by slot and sends the parts in parallel. This is not atomic, but it is often enough for reads. redis-py's
RedisCluster.mget()refuses keys in different slots, whilemget_nonatomic()groups them by slot:
from redis.cluster import RedisCluster, ClusterNode
rc = RedisCluster(startup_nodes=[ClusterNode("node1", 6379)], decode_responses=True)
rc.set("{user:1001}:cart", "3 items")
rc.set("{user:1001}:profile", "alice")
print(rc.mget("{user:1001}:cart", "{user:1001}:profile")) # ['3 items', 'alice'], one slot
# rc.mget("user:1", "user:2", "user:3")
# -> RedisClusterException: MGET - all keys must map to the same key slot
print(rc.mget_nonatomic("user:1", "user:2", "user:3")) # ['v1', 'v2', 'v3'], split per slot
- Use pipelining per node (chapter 4.3). Cluster clients group a pipeline's commands by node.
- Restructure: a single hash with many fields instead of many keys, when the data fits on one node anyway.
Key Takeaways
- Multi-key commands, transactions and scripts need all keys in one slot, or they fail with
CROSSSLOT. - A hash tag
{...}makes only its content count for hashing: the first{, up to the first}, non-empty. - Tag by the unit of atomicity (a user, an order), not by something large like a tenant or a global name.
- A slot can never be split. Coarse tags create hot or oversized nodes.
- When atomicity is not needed, let the client split commands per slot instead of forcing keys together.
Practice
- (Beginner) Predict, then verify with
CLUSTER KEYSLOT, whether these pairs share a slot:{a}band{a}c;a{b}andc{b};{a}{b}and{a}{c};{}aand{}b.- (Intermediate) Rewrite the key schema of a shopping cart (cart hash, item-count string, last-updated sorted-set entry in a global index) so that the checkout script runs in a cluster. Which key cannot be co-located with the others, and how do you handle it?
- (Interview) "We tag every key with the customer ID, and one enterprise customer now generates 40% of our traffic. What happens in the cluster, and what are your options?" Hint: consider slot granularity, whether the tag is needed for every key type, and splitting one logical entity into several tags.
MOVED and ASK Redirections
In a cluster there is no router in front of the nodes. A client may send a command to the wrong node, either because it does not know the slot map yet or because the map changed. The node does not forward the request. It tells the client where to go. There are two kinds of redirection, and they mean different things.
MOVED: the slot lives elsewhere permanently
redis-cli -h node1 SET foo bar
# (error) MOVED 12182 172.19.0.5:6379
MOVED <slot> <host:port> means "slot 12182 is owned by that node; update your slot map". A smart client then:
- Resends the command to the given node.
- Updates its map for that slot, or more commonly refreshes the whole map with
CLUSTER SLOTSorCLUSTER SHARDS, because oneMOVEDoften means that many slots moved.
redis-cli -c follows redirections automatically, which is handy for manual work:
redis-cli -c -h node1 SET foo bar # OK (redirected to 172.19.0.5 behind the scenes)
redis-cli -c -h node1 GET foo # "bar"
ASK: the slot is being moved right now
During resharding (see "Resharding and Slot Migration") a slot is half on the source node and half on the target. The source answers for keys it still has. For keys it no longer has, it replies:
ASK 12182 172.19.0.3:6379
ASK means "try that node, for this one request only". The client must:
- Send
ASKINGto the target node, then the command, on the same connection. - Not update its slot map. The source still owns the slot until the migration finishes.
Without ASKING, the target refuses, because it does not own the slot yet. A real sequence during a manual migration of slot 12182, after the key foo had already been moved:
redis-cli -h source GET foo
# (error) ASK 12182 172.19.0.3:6379
redis-cli -h target GET foo
# (error) MOVED 12182 172.19.0.5:6379 <- target: "not mine yet, ask the owner"
printf 'ASKING\nGET foo\n' | redis-cli -h target
# OK
# "bar" <- accepted for this one command
redis-cli -h source SET '{foo}x' 1
# (error) ASK 12182 172.19.0.3:6379 <- new keys in a migrating slot go to the target
| MOVED | ASK | |
|---|---|---|
| Meaning | Slot permanently owned by another node | Slot in migration; this key is (or will be) on the target |
| Client action | Retry at new node and update the slot map | Send ASKING + command to target, keep the map unchanged |
| When | After resharding or failover, or with a stale map | Only while a slot is migrating |
TRYAGAIN
A multi-key command whose keys are split between source and target during a migration cannot run on either node. It gets TRYAGAIN, and the client should retry after a short delay. Atomic slot migration (next topics) avoids this.
Reading from replicas in a cluster
Cluster replicas redirect everything to their primary by default, even reads. A client that accepts stale reads sends READONLY on its connection to a replica. After that, the replica serves read commands for the slots of its primary (READWRITE cancels it). Cluster clients expose this as an option. In redis-py (5.3+) it is RedisCluster(..., load_balancing_strategy=LoadBalancingStrategy.ROUND_ROBIN_REPLICAS), which replaces the deprecated read_from_replicas=True.
How a client keeps up
startup: CLUSTER SHARDS (or CLUSTER SLOTS) from any seed node -> slot map cached locally
request: slot = CRC16(key) mod 16384 -> node from map -> send
MOVED: resend to the node in the reply, refresh the map
ASK: ASKING + resend to the target, map unchanged
errors: connection refused/timeouts -> refresh the map from another node, retry with backoff
A good client therefore almost never sees MOVED in steady state. Most commands go directly to the right node in one round trip. Many MOVED errors in monitoring mean the client is not caching the map.
Key Takeaways
- Nodes never proxy requests. They redirect.
MOVED= permanent change: retry elsewhere and refresh the slot map.ASK= slot migrating: sendASKINGand the command to the target once, without updating the map.TRYAGAIN= multi-key command spanning a slot that is migrating. Retry later.- Replica reads in a cluster need
READONLYon the connection.
Practice
- (Beginner) Without
-c, sendSETfor 10 different keys to one node of a cluster and note which replies areMOVED. Predict the fraction before you start.- (Intermediate) Start a manual migration of one slot (
CLUSTER SETSLOT ... IMPORTING/MIGRATING) and migrate half of its keys withMIGRATE. Then tryGETon a moved key,GETon a key not yet moved,SETof a new key andMGETof one key of each kind, and record each reply.- (Interview) "Why must a client not update its slot map after an
ASKredirection?" Hint: think about where the keys that have not yet moved are, and what would happen to the next request for one of them.
Cluster Bus and Gossip Protocol
A Redis Cluster has no central coordinator. No ZooKeeper or etcd process holds the "truth". Every node keeps its own view of the cluster and updates it by talking to the others over the cluster bus: a separate TCP port with a compact binary protocol. Knowledge spreads like rumors in an office (gossip). Each node regularly tells a few colleagues what it knows about itself and about some others, and within a short time everyone knows everything.
What travels on the bus
| Message | Purpose |
|---|---|
PING / PONG | Heartbeats. Each carries the sender's state and gossip about a few other nodes |
MEET | Adds a node to the cluster (CLUSTER MEET ip port) |
FAIL | Broadcast: "node X is failed", once the failure is agreed |
PUBLISH | Classic Pub/Sub messages, broadcast to every node (chapter 4.1) |
UPDATE | Tells a node with an outdated slot view about a newer configuration |
FAILOVER_AUTH_REQUEST / _ACK | A replica asking primaries for votes, and the votes |
MFSTART | Starts a manual failover |
Every heartbeat has a header (sender ID, its slot bitmap of 16,384 bits, its config epoch, its role and flags, its replication offset) and a gossip section with the state of a few random nodes the sender knows. Through the gossip sections, nodes learn about nodes they have not yet talked to directly.
Heartbeat timing
- Every second, each node picks a few random nodes and pings the one it has heard from least recently.
- It also pings every node from which it has not received a
PONGfor more than half ofcluster-node-timeout.
Bus traffic therefore grows with the number of nodes and with smaller timeouts. This is one reason the practical limit is about 1,000 nodes.
redis-cli CLUSTER INFO | grep messages
# cluster_stats_messages_ping_sent:12
# cluster_stats_messages_pong_sent:15
# cluster_stats_messages_sent:27
# (plus _received counters, and meet/fail/publish/update/auth-req counters when used)
Failure detection: PFAIL and FAIL
Similar to Sentinel's SDOWN and ODOWN, there are two levels:
- PFAIL (possible failure): a node has not answered a
PINGforcluster-node-timeout. This is local knowledge, and it is spread in gossip sections. - FAIL: a primary collects reports of PFAIL for a node from a majority of the primaries, within a validity window of twice the node timeout. It then marks the node FAIL and broadcasts a
FAILmessage, which every node accepts at once.
FAIL on a primary triggers the failover of its replicas (see "Cluster Failover"). The FAIL flag is cleared when the node is reachable again and is a replica or a primary without slots. A primary that still owns slots has its flag cleared only if it comes back and no replica has taken over its slots after a long time (several node timeouts).
Configuration epochs: who owns a slot?
Ownership conflicts are resolved with numbers, not with a coordinator. Every primary has a config epoch. When two nodes claim the same slot, the claim with the higher config epoch wins. A replica that wins a failover takes a new, unique, higher epoch, so its claim beats the claim of the old primary when the old primary comes back. UPDATE messages tell nodes with outdated claims to step back.
Node identity and nodes.conf
Each node has a permanent 40-character ID, generated the first time it starts and stored in nodes.conf with its view of the cluster. The ID, not the IP address, identifies the node, so a node that restarts with a new IP rejoins as itself. Never copy nodes.conf between nodes, and never edit it by hand.
Networking settings
| Setting | Use |
|---|---|
cluster-port | Bus port. 0 (the default) means client port + 10000 |
cluster-announce-ip, cluster-announce-port, cluster-announce-bus-port | Address to advertise when the node is behind NAT or in containers |
cluster-announce-hostname + cluster-preferred-endpoint-type hostname | Advertise DNS names to clients instead of IPs (needed for TLS hostname checks) |
Key Takeaways
- Every node gossips over the cluster bus (port + 10000). There is no central coordinator.
- Heartbeats carry the sender's slot bitmap and config epoch, plus gossip about other nodes.
- PFAIL is a local suspicion after
cluster-node-timeout. FAIL requires reports from a majority of primaries and is broadcast. - Slot ownership conflicts are resolved by the higher config epoch.
- The node ID in
nodes.confis the node's identity. Behind NAT, set thecluster-announce-*options.
Practice
- (Beginner) In a 6-node cluster, read
CLUSTER INFOtwice, 60 seconds apart, and compute how many bus messages per second one node sends at rest.- (Intermediate) Freeze one primary with
docker pauseand runCLUSTER NODESon another node every second. Record when the frozen node's flags change tofail?(PFAIL) and when tofail, and relate the times tocluster-node-timeout.- (Interview) "Two nodes both claim to own slot 500 after a network partition heals. How does the cluster decide, without a coordinator?" Hint: describe config epochs, how a failover obtains a new epoch, and the
UPDATEmessage.
Resharding and Slot Migration
Clusters change: you add a primary for more memory, remove one after traffic drops, or rebalance because one node fills up faster. All of these mean moving hash slots, with their keys, from one primary to another while clients keep reading and writing. Redis has two mechanisms for this: the classic key-by-key migration that all versions support, and atomic slot migration in recent versions.
Adding and removing nodes
# Add an empty primary (new-node first, then any existing node)
redis-cli --cluster add-node 10.0.0.7:6379 10.0.0.1:6379
# Add a replica for a specific primary
redis-cli --cluster add-node 10.0.0.8:6379 10.0.0.1:6379 --cluster-slave --cluster-master-id <primary-id>
# Move 1000 slots from all primaries to the new one (interactive by default; these flags script it)
redis-cli --cluster reshard 10.0.0.1:6379 \
--cluster-from all --cluster-to <new-primary-id> --cluster-slots 1000 --cluster-yes
# Or let the tool even out slots across primaries (by weight if given)
redis-cli --cluster rebalance 10.0.0.1:6379 --cluster-use-empty-masters
# Remove a node: first move all its slots away (reshard), then
redis-cli --cluster del-node 10.0.0.1:6379 <node-id>
# Check that every slot is covered and no slot is stuck half-migrated
redis-cli --cluster check 10.0.0.1:6379
redis-cli --cluster fix 10.0.0.1:6379 # repairs open slots after an interrupted migration
A new primary receives no slots until you reshard. It holds no data and receives no traffic.
Classic migration: what the tool does for each slot
1. target: CLUSTER SETSLOT <slot> IMPORTING <source-id> accept ASKING requests for this slot
2. source: CLUSTER SETSLOT <slot> MIGRATING <target-id> answer ASK for missing keys
3. loop: CLUSTER GETKEYSINSLOT <slot> <batch> list some keys on the source
MIGRATE <target> <port> "" 0 <timeout> KEYS k1 k2 ... move them atomically, per batch
4. both: CLUSTER SETSLOT <slot> NODE <target-id> ownership changes; MOVED from now on
MIGRATE serializes each key, restores it on the target and deletes it from the source. For each key this is atomic, but it blocks both nodes while it runs, so a single huge key (a hash with millions of fields, chapter 8.4) can stall both nodes for seconds. During the migration, clients see ASK redirections, and multi-key commands spanning moved and unmoved keys get TRYAGAIN.
If the process is interrupted, slots stay in the IMPORTING/MIGRATING state. redis-cli --cluster check reports them as "open slots", and --cluster fix completes or rolls back the move.
Atomic slot migration (Redis 8.4+)
Redis 8.4 added a different approach: the target replicates the slots instead of receiving keys one at a time. It takes a snapshot of the slots' data from the source, streams the changes made meanwhile (like a replica, 7.1), and when it has caught up, ownership switches atomically, after a very short pause of writes. Clients never see ASK or TRYAGAIN, big keys do not block, and a failed migration can be cancelled without leaving the slot half-moved.
# Run on the DESTINATION primary: pull slots 5461-5600 from whoever owns them
redis-cli -h target CLUSTER MIGRATION IMPORT 5461 5600
# "20564b1347b055e993757c7d638dc6b3ad4d38dc" <- task ID
redis-cli -h target CLUSTER MIGRATION STATUS ALL
# id 20564b13...
# slots 5461-5600
# source 4a912354... dest 981ae2d3...
# operation import
# state completed
# retries 0
# write_pause_ms 0 <- writes were paused for less than 1 ms (Redis 8.10, small dataset)
redis-cli -h target CLUSTER MIGRATION CANCEL ID 20564b1347b055e993757c7d638dc6b3ad4d38dc
Valkey 9.0 has its own atomic slot migration with different commands: CLUSTER MIGRATESLOTS SLOTSRANGE <start> <end> NODE <node-id>, CLUSTER GETSLOTMIGRATIONS and CLUSTER CANCELSLOTMIGRATIONS. Check which mechanism your version and tooling use before planning a large reshard.
Classic (MIGRATE) | Atomic slot migration | |
|---|---|---|
| Availability | All versions | Redis 8.4+, Valkey 9.0+ (different commands) |
| Unit moved | Key batches | Whole slot ranges, by replication |
| Client impact | ASK, TRYAGAIN for multi-key commands | No redirections until a single ownership switch |
| Big keys | Block both nodes during MIGRATE | Streamed like replication |
| Interruption | Open slots, need --cluster fix | Cancel or retry as one task |
Planning a reshard
- Reshard during low traffic, and watch latency on source and target.
- Check memory on the target before moving: it needs room for the incoming data plus replication buffers.
- Find big keys first (
redis-cli --bigkeysor--memkeys, chapter 12.1) if you use classic migration. - Add replicas for new primaries before or right after giving them slots, so that no shard runs without a replica.
Key Takeaways
- Capacity changes in a cluster are slot moves. New primaries get no slots until you reshard.
redis-cli --cluster add-node / reshard / rebalance / del-node / check / fixcovers the life cycle.- Classic migration moves keys in batches with
MIGRATE. Clients seeASKandTRYAGAIN, and big keys block both nodes. - Atomic slot migration (Redis 8.4+:
CLUSTER MIGRATION IMPORT, run on the target; Valkey 9.0+:CLUSTER MIGRATESLOTS) replicates whole slots and switches ownership in one step. - An interrupted classic migration leaves open slots. Find them with
--cluster check.
Practice
- (Beginner) Add a 4th primary to a 3-primary cluster with
--cluster add-node, then--cluster rebalance --cluster-use-empty-masters. Record how many slots and keys it receives.- (Intermediate) Run
redis-benchmarkagainst the cluster (--clusteroption) while moving 1,000 slots, once withredis-cli --cluster reshardand once withCLUSTER MIGRATION IMPORT(Redis 8.4+). Compare the latency percentiles and the number of redirections reported by the client.- (Interview) "A reshard was interrupted halfway and now some requests return
ASKforever. What state is the cluster in, and how do you repair it?" Hint: look at theIMPORTING/MIGRATINGflags inCLUSTER NODESoutput and at what--cluster fixdecides for each open slot.
Cluster Failover
Redis Cluster does not need Sentinel. Its nodes detect failures themselves and promote a replica of the failed primary, using the same gossip bus. The design is similar to Sentinel (agreement, then a vote, then a new epoch), but the voters are the primaries of the cluster.
Automatic failover
- The primary is marked FAIL (PFAIL reports from a majority of primaries, previous topic).
- Each of its replicas checks that its own data is not too old: it does not run if its link to the primary has been down for longer than
cluster-node-timeout × cluster-replica-validity-factor(default factor 10) plus the ping period. A factor of 0 means "always try". - Replicas wait before asking for votes:
500 ms + random(0-500 ms) + rank × 1000 ms, where rank 0 is the replica with the highest replication offset. The most up-to-date replica usually asks first and wins. - The replica increments the cluster's current epoch and asks all primaries for a vote. Each primary votes once per epoch and refuses to vote for another replica of the same primary for a while.
- With votes from a majority of all primaries, the replica promotes itself, takes the new config epoch and announces its slots. Its higher epoch overrides the old primary's claim everywhere.
- The other replicas of the old primary switch to the new one. When the old primary returns, it becomes a replica too.
A real run (Redis 8.10, cluster-node-timeout 15000, primary stopped with docker stop):
replica log:
09:04:34.769 * FAIL message received from 8808d7cb... about 4a912354...
09:04:34.785 * Start of election delayed for 515 milliseconds (rank #0, offset 26234).
09:04:35.390 * Starting a failover election for epoch 8.
09:04:35.439 * Failover election won: I'm the new master.
09:04:35.440 * configEpoch set to 8 after successful failover
The primary was stopped at about 09:04:16.5, so its slots were unavailable for about 19 seconds: 15 s of node timeout until PFAIL, about 3 s until a majority of primaries had exchanged PFAIL reports and FAIL was broadcast, then 0.6 s of election delay and 0.05 s for the vote itself. The node timeout dominates.
Manual failover
For maintenance or upgrades, run CLUSTER FAILOVER on the replica you want to promote:
| Variant | Behavior |
|---|---|
CLUSTER FAILOVER | Safe: the primary stops accepting writes, the replica catches up to its exact offset, then takes over. No acknowledged write is lost |
CLUSTER FAILOVER FORCE | Does not wait for the old primary (use when it is unreachable), but still needs the primaries' votes |
CLUSTER FAILOVER TAKEOVER | Takes over without any vote, bumping its own epoch. Last resort, e.g. when most primaries are in a lost data center |
replica log, CLUSTER FAILOVER (Redis 8.10):
09:05:39.775 * Manual failover user request accepted.
09:05:39.775 * Received replication offset for paused master manual failover: 27834
09:05:39.775 * All master replication stream processed, manual failover can start.
09:05:39.775 * Starting a failover election for epoch 9.
09:05:39.818 * Failover election won: I'm the new master.
Here the switch took about 40 ms, even though another shard of the cluster was down at the time: two of the three primaries were enough for a majority. A rolling upgrade therefore usually goes: upgrade a replica, CLUSTER FAILOVER to it, upgrade the old primary (now a replica), and repeat for each shard.
When a whole shard is lost
If a primary and all its replicas fail, its slots have no owner, and the behavior depends on two settings:
| Setting | Default | Effect |
|---|---|---|
cluster-require-full-coverage | yes | The whole cluster refuses commands (CLUSTERDOWN The cluster is down) while any slot is uncovered |
cluster-allow-reads-when-down | no | With yes, nodes keep serving reads for their own slots while the cluster state is fail |
In a test, both nodes of one shard were stopped. With the default, even keys on healthy shards failed:
redis-cli CLUSTER INFO | grep -E 'state|slots_fail'
# cluster_state:fail
# cluster_slots_fail:5460
redis-cli -c GET user:1 # slot 10778, on a healthy shard
# (error) CLUSTERDOWN The cluster is down
After CONFIG SET cluster-require-full-coverage no on the surviving nodes, the state went back to ok. Keys on healthy shards worked again, and keys in the lost shard's slots were still redirected to the dead node, so they kept failing. Choose no when partial availability is better than none, for example for a cache.
Replica migration
Replicas can move automatically to a primary that has none. If one primary has two replicas and another has lost its only one, a spare replica moves over (cluster-allow-replica-migration yes), as long as its own primary keeps at least cluster-migration-barrier (default 1) replicas. Giving a few shards one extra replica therefore protects all shards.
Minority partitions
A primary on the minority side of a partition keeps accepting writes for up to cluster-node-timeout. After that, it notices that it cannot reach the majority and stops accepting writes, while the majority side promotes one of its replicas. Writes made in that window are lost when the partition heals. As with Sentinel (7.2, "Split-Brain Scenarios"), the cluster limits this loss but does not prevent it.
Key Takeaways
- Cluster nodes fail over without Sentinel. Replicas are elected by a majority of primaries, one vote per epoch.
- The best replica (highest offset) waits least and usually wins. Stale replicas do not run (validity factor).
- Unavailability ≈
cluster-node-timeoutplus about 1-2 s. Lowering the timeout speeds up failover but risks false failovers. CLUSTER FAILOVERon a replica is a lossless planned switch.FORCEandTAKEOVERare for emergencies.- Losing a whole shard stops the whole cluster unless
cluster-require-full-coverage no.
Practice
- (Beginner) Stop a primary in your test cluster and measure, with a loop of
redis-cli ROLEagainst its replica, how long the promotion takes. Repeat withcluster-node-timeout 5000on all nodes.- (Intermediate) Perform a rolling restart of all three shards using
CLUSTER FAILOVERwhile a client writes an incrementing counter. Verify that no increment was lost and count the errors seen by the client.- (Interview) "Our 3-primary cluster spans two data centers: two primaries in DC1, one in DC2. DC1 goes offline. What happens, and what can an operator do?" Hint: count the primaries that can vote in DC2, and look at what
CLUSTER FAILOVER TAKEOVERdoes and risks.
Cluster Limitations
Redis Cluster scales memory and throughput almost linearly, but it is not a transparent replacement for a single server. Some commands and patterns work differently or not at all, and an application must be designed for this. This topic collects the differences, so you can check a design against them before migrating.
Functional limits
| Area | Single server | Redis Cluster |
|---|---|---|
Multi-key commands (MGET, SINTER, RENAME, BLMOVE...) | Any keys | Same slot only (CROSSSLOT), use hash tags |
Transactions (MULTI/EXEC, WATCH) | Any keys | Same slot, sent to one node |
| Lua scripts and functions | Any keys | Declared keys in one slot. Undeclared key access is unsafe |
| Databases | 16 by default (SELECT) | Only database 0 (Valkey 9.0 adds multiple databases in cluster mode) |
KEYS, SCAN, DBSIZE, FLUSHALL, RANDOMKEY | Whole dataset | One node. The client must run them on every primary and merge |
| Classic Pub/Sub | Local | Broadcast to all nodes. Use sharded Pub/Sub (SPUBLISH, 7.0+, chapter 4.1) |
| Clients | Any | Must be cluster-aware (slot map, redirections) |
Example: scanning all keys in a cluster
from redis.cluster import RedisCluster, ClusterNode
rc = RedisCluster(startup_nodes=[ClusterNode("node1", 6379)], decode_responses=True)
def scan_all(pattern: str):
"""SCAN runs per node, so iterate every primary and yield its keys."""
for node in rc.get_primaries():
conn = rc.get_redis_connection(node)
for key in conn.scan_iter(match=pattern, count=1000):
yield key
print(sum(1 for _ in scan_all("user:*")))
redis-py also offers rc.scan_iter() which does this internally. Check what your client does with KEYS, DBSIZE or FLUSHALL: some send them to all nodes, some to one random node.
Operational limits
- Big keys cannot be split. A key lives in one slot on one node. A 10 GB sorted set needs a node with 10 GB of free memory, and all its traffic hits one node.
- Hot keys and hot slots. Sharding spreads keys, not load. One very popular key still runs on one core (chapter 8.4).
- Consistency. Asynchronous replication and failover rules mean acknowledged writes can be lost (see "Cluster Failover").
WAITworks per shard, with the same limits as in 7.1. - Minimum size and overhead. At least 3 primaries, plus replicas. Each node has a gossip and memory overhead (the slot-to-keys mapping), and the practical upper limit is about 1,000 nodes.
- Network setup. Every node must reach every other node on both ports, and clients must reach all nodes directly. NAT, Docker port mapping and Kubernetes need
cluster-announce-*settings or hostnames. - Cross-shard work is the client's job. Aggregations over all data (
SUNIONof keys on different shards, global leaderboards) need application logic or a different key design.
Cluster or not?
| Situation | Better choice |
|---|---|
| Data fits comfortably in one server (with headroom), one core handles the writes | Primary + replicas + Sentinel (or a managed equivalent) |
| Data or write throughput beyond one server | Cluster |
| Workload depends on many multi-key operations across unrelated keys | Single server, or redesign keys before clustering |
| Many independent tenants or datasets | Separate instances, or a cluster with per-tenant hash tags (if no tenant is huge) |
A single modern server handles well over 100,000 operations per second and tens of GB of memory. Many applications never need a cluster. Moving to one is easiest when the key design (hash tags, no unbounded keys, no cross-entity transactions) was cluster-ready from the start.
Key Takeaways
- Multi-key operations, transactions and scripts must stay within one slot. Only database 0 exists (in Redis).
- Keyspace-wide commands (
KEYS,SCAN,DBSIZE,FLUSHALL) act on one node. Iterate the primaries. - Big keys and hot keys are not fixed by sharding.
- Clients must be cluster-aware and able to reach every node. Networking needs care behind NAT.
- Use a cluster when one server is not enough. Otherwise, replication plus Sentinel is simpler.
Practice
- (Beginner) Run
DBSIZE,KEYS user:*andFLUSHALLwithredis-cli -cagainst one node of a cluster. Which nodes did each command affect?- (Intermediate) Take an existing application's list of Redis commands (for example from
MONITORorCOMMANDSTATS) and classify each multi-key command as cluster-safe, needing hash tags, or needing redesign.- (Interview) "Our 60 GB Redis is at 85% memory and the team wants to move to Cluster next week. What do you check first?" Hint: look for multi-key commands across entities,
SELECTusage, big keys, scripts with undeclared keys,KEYS/SCANusage, client library support, and whether a bigger server or eviction would be enough.
<a id="74-scaling-strategies"></a>
7.4 Scaling Strategies
This subchapter steps back from individual mechanisms and looks at the ways a Redis deployment can grow: a bigger server, more servers, sharding done by the client or by a proxy, and databases that accept writes in several regions at once. It ends with the conflict-free data types that make multi-region writes possible.
Vertical vs Horizontal Scaling
When a Redis deployment runs out of something (memory, CPU, network or availability), there are two directions to grow. Vertical scaling (scaling up) means a bigger server. Horizontal scaling (scaling out) means more servers, each doing part of the work. The right choice depends on which resource is exhausted, so the first step is always to measure.
Which resource is the bottleneck?
| Symptom | Metric (chapter 12.1) | Usually the limit |
|---|---|---|
| Evictions or OOM errors | used_memory near maxmemory, evicted_keys rising | Memory |
| Latency grows with load, one core at 100% | used_cpu_user, per-thread CPU | Command execution (main thread) |
| High CPU in network/system time, many clients | used_cpu_sys, instantaneous_input_kbps | Network I/O |
| Slow snapshots, latency spikes every few minutes | latest_fork_usec, rdb_last_cow_size | Fork and persistence cost of a large dataset |
| Outage when one host fails | — | Availability, not capacity |
Vertical scaling: a bigger server
Simple: no application change, all commands keep working. But there are limits specific to Redis:
- Commands run on one core (chapter 5.1). More cores help I/O threads, persistence and background work, but not the execution of commands. Once the main thread is saturated, a machine with more cores does not make the instance faster.
- Large datasets cost more to operate. Fork time grows with memory (roughly 10-20 ms per GB, chapter 6.1), a full resync of a replica transfers the whole dataset, and a restart loads it all. A 200 GB instance can take many minutes to resync or restart.
- Price and availability. Very large memory machines are expensive, and a single bigger box is still a single point of failure.
Horizontal scaling: more servers
| Technique | Scales | Cost |
|---|---|---|
| Read replicas (7.1) | Reads | Stale reads; writes still on one primary |
| Sharding (Cluster 7.3, client 7.4, proxy 7.4) | Memory, writes and reads | Multi-key limits, more nodes to operate |
| Several independent instances | Everything, by splitting unrelated workloads (cache, sessions, queues) | No data shared between them, more instances to manage |
| Active-Active (7.4) | Writes in several regions | Commercial products only, eventual consistency |
Splitting unrelated workloads onto separate instances is often the cheapest first step. A cache with allkeys-lru eviction and a job queue that must never lose data should not share a server anyway (different maxmemory-policy and persistence settings).
A typical growth path
1 instance ──▶ + replica(s) + Sentinel ──▶ split workloads ──▶ bigger instances ──▶ Redis Cluster
(dev) (availability, reads) (cache vs data) (up to ~25-50 GB) (memory/writes
beyond one node)
The 25-50 GB range is not a hard limit but a common rule of thumb: above it, fork, resync and restart times start to dominate operations. Many teams prefer several medium shards to one huge instance for this reason, even when one machine could hold all the data.
Estimating capacity
def shards_needed(dataset_gb: float, write_ops: float,
max_gb_per_shard: float = 25.0, max_ops_per_shard: float = 100_000.0,
headroom: float = 0.6) -> int:
"""Smallest shard count that keeps memory and write load under `headroom` of the per-shard limit."""
by_memory = dataset_gb / (max_gb_per_shard * headroom)
by_writes = write_ops / (max_ops_per_shard * headroom)
return max(1, int(-(-max(by_memory, by_writes) // 1))) # ceiling
print(shards_needed(dataset_gb=120, write_ops=150_000)) # 8 (memory-bound: 120 / 15 = 8)
print(shards_needed(dataset_gb=10, write_ops=400_000)) # 7 (write-bound: 400k / 60k = 6.7)
Replace the per-shard limits with numbers from redis-benchmark and memtier_benchmark runs of your workload on your hardware. Generic numbers are only a starting point.
Key Takeaways
- Identify the exhausted resource first: memory, the command thread, network, operational cost or availability.
- Vertical scaling is simple but limited by single-threaded command execution and by fork, resync and restart times of big datasets.
- Replicas scale reads only. Sharding scales memory and writes.
- Splitting unrelated workloads onto separate instances is often the cheapest scaling step.
- Several medium shards are easier to operate than one huge instance.
Practice
- (Beginner) Run
redis-benchmark -t set,get -n 1000000 -P 16 -qagainst one instance, with--threads 4on the benchmark, and watch the server's CPU usage per thread (top -H). Which thread saturates?- (Intermediate) Measure
latest_fork_usecand the full-resync time of a replica at 1 GB, 5 GB and 10 GB. Extrapolate both to 100 GB and decide at which size you would rather shard.- (Interview) "Our single Redis instance is at 90% CPU. The team proposes moving to a machine with twice as many cores. Will it help?" Hint: check which thread is busy, what I/O threads can offload, and whether the load is reads (replicas) or writes (sharding).
Client-Side Sharding
Before Redis Cluster existed (it arrived in Redis 3.0), the common way to spread data across several Redis servers was to let the client decide which server holds each key. The servers are plain, independent instances that know nothing about each other. The technique is still used, mostly for caches, and it explains the problems that Cluster solves.
The idea
application
┌──────────────────┐
│ node = f(key) │──────▶ redis-a (independent instances,
│ │──────▶ redis-b no replication or
│ │──────▶ redis-c communication between them)
└──────────────────┘
Every client must use the same function and the same server list, or different clients will look for the same key in different places.
Modulo hashing and why it breaks on resize
node = hash(key) % N spreads keys evenly, but when N changes, almost every key maps to a different node. For a cache that means a sudden, near-total miss rate, a stampede of requests to the database. Consistent hashing fixes this. Each node is placed at many points ("virtual nodes") on a ring of hash values, and a key belongs to the first node clockwise from its own hash. Adding a node only takes over the arcs just before its points, about 1/N of the keys.
import bisect
import hashlib
def h(s: str) -> int:
return int.from_bytes(hashlib.md5(s.encode()).digest()[:8], "big")
class HashRing:
def __init__(self, nodes: list[str], vnodes: int = 160):
# Each node appears vnodes times on the ring, which evens out the distribution
self.points = sorted((h(f"{n}#{i}"), n) for n in nodes for i in range(vnodes))
self.hashes = [p for p, _ in self.points]
def node_for(self, key: str) -> str:
i = bisect.bisect(self.hashes, h(key)) % len(self.points) # first point clockwise, wrap around
return self.points[i][1]
keys = [f"user:{i}" for i in range(100_000)]
old, new = ["r1", "r2", "r3"], ["r1", "r2", "r3", "r4"]
moved_modulo = sum(h(k) % 3 != h(k) % 4 for k in keys)
a, b = HashRing(old), HashRing(new)
moved_ring = sum(a.node_for(k) != b.node_for(k) for k in keys)
print(f"modulo: {moved_modulo / len(keys):.1%} ring: {moved_ring / len(keys):.1%}")
# modulo: 75.0% ring: 24.5% <- adding a 4th node; the ideal is 1/4 = 25%
This is the ketama scheme that memcached clients made popular. With 160 virtual nodes per server, the four nodes in this run got between 24,157 and 26,235 of the 100,000 keys each.
Using it with real connections
import redis
class ShardedRedis:
"""Minimal client-side sharding for single-key commands (a cache, typically)."""
def __init__(self, addresses: dict[str, tuple[str, int]]):
self.ring = HashRing(list(addresses))
self.clients = {name: redis.Redis(host=h, port=p) for name, (h, p) in addresses.items()}
def _client(self, key: str) -> redis.Redis:
return self.clients[self.ring.node_for(key)]
def get(self, key):
return self._client(key).get(key)
def set(self, key, value, ex=None):
return self._client(key).set(key, value, ex=ex)
cache = ShardedRedis({"r1": ("10.0.0.1", 6379), "r2": ("10.0.0.2", 6379), "r3": ("10.0.0.3", 6379)})
cache.set("page:/home", "<html>...</html>", ex=300)
Hash tags work here too, if the hash function applies them: hash only the {...} part, and keys that must be used together land on the same node.
Limitations
| Problem | Why |
|---|---|
| Moving data on resize | The servers do not move keys. After a resize, keys that changed owner are simply missing from their new node. Acceptable for a cache, not for primary data |
| No automatic failover | Each shard needs its own replicas and Sentinel. A dead shard either loses its keys or must be removed from the ring, which moves them |
| Multi-key operations | Only for keys known to be on the same node |
| Configuration drift | Every client must have exactly the same node list and algorithm, in every language and version |
When it still makes sense
- A pure cache over several independent instances, where a miss only costs a database read.
- Client libraries that implement it already (for example Predis for PHP), when Cluster is not available.
- In most other cases, Redis Cluster or a proxy is the better choice because they handle rebalancing and failover.
Key Takeaways
- Client-side sharding: the client maps keys to independent servers. All clients must agree on the function and the node list.
hash % Nmoves most keys when N changes (75% from 3 to 4 nodes). Consistent hashing moves about 1/N (24.5% measured).- The servers never move data, so resizing loses the moved keys. Fine for caches, not for data you must keep.
- Failover, rebalancing and multi-key operations are left to you. Cluster solves them on the server side.
Practice
- (Beginner) Run the ring example with 3 → 4 nodes, then with 10 → 11 nodes. Compare the fraction of moved keys with 1/N.
- (Intermediate) Change
vnodesto 1, 10, 100 and 1,000 and measure the smallest and largest share of keys per node. What does this tell you about choosing the number of virtual nodes?- (Interview) "We cache product pages over 8 Redis instances using client-side consistent hashing. One instance dies. What happens to hit rate and to the database, and what are the options?" Hint: compare removing the node from the ring with keeping it until it is replaced, and think about how many keys each option moves.
Proxy-Based Sharding
Client-side sharding puts the routing logic in every application, in every language. A proxy moves that logic into one service: clients connect to the proxy as if it were a single Redis server, and the proxy forwards each command to the right backend. Applications do not need cluster-aware clients, and the sharding scheme can change without redeploying them.
app (any Redis client) app app
│ │ │
└────────────────┼──────────────┘
▼
┌───────────────────┐ speaks RESP, looks like one server
│ proxy │ key -> shard, connection pooling,
│ (one or several) │ pipelining to backends
└───┬──────┬──────┬─┘
▼ ▼ ▼
shard 1 shard 2 shard 3 (plain instances or a Redis Cluster)
What a proxy adds
- Simple clients: any Redis client works, including ones without cluster support.
- Fewer connections: thousands of application connections are multiplexed onto a few connections per backend.
- Central control: routing changes, backends are added, and metrics are collected in one place.
What it costs
- One more network hop on every command, typically a fraction of a millisecond, plus proxy CPU.
- The proxy must be highly available itself: run several instances behind a load balancer or DNS.
- Command restrictions: a proxy can only route commands whose keys it can hash. Commands without keys, with keys on several shards, or bound to one connection (transactions, blocking pops, subscriptions) are often refused or limited.
Examples
| Proxy | Backends | Notes |
|---|---|---|
| Twemproxy (nutcracker) | Plain Redis or memcached instances | Distribution ketama (default), modula or random; several hash functions (default fnv1a_64). auto_eject_hosts can drop a failing server from the ring. Does not support Redis Cluster backends. Its documentation lists commands it does not support, including KEYS, SCAN, MULTI/EXEC, SUBSCRIBE/PUBLISH and blocking pops such as BLPOP |
| Envoy (Redis proxy filter) | Plain instances (ring hash load balancing) or Redis Cluster | Supports only commands that can be hashed to one server; MULTI is supported (in cluster mode, all keys of the transaction must share a slot); blocking commands and SUBSCRIBE are not in its supported list |
| Redis Software / Redis Cloud proxy | The product's own shards | Clients see one endpoint; the proxy handles sharding and failover transparently |
Always check the supported-command list of a proxy against the commands your application uses, before choosing it.
Example: Twemproxy configuration
# nutcracker.yml
cache:
listen: 0.0.0.0:22121 # clients connect here as if it were Redis
redis: true # speak the Redis protocol (default is memcached)
hash: fnv1a_64 # hash function applied to the key (or its hash tag)
hash_tag: "{}" # only hash the part inside {...}
distribution: ketama # consistent hashing across the servers below
auto_eject_hosts: true # temporarily remove a server after repeated failures
server_retry_timeout: 30000 # ms before retrying an ejected server
server_failure_limit: 2 # consecutive failures before ejection
servers:
- 10.0.0.1:6379:1 redis-a # host:port:weight name (the name is what gets hashed)
- 10.0.0.2:6379:1 redis-b
- 10.0.0.3:6379:1 redis-c
redis-cli -p 22121 SET page:/home "<html>...</html>" # routed by twemproxy
redis-cli -p 22121 KEYS '*' # not supported by the proxy
Ejecting hosts moves their keys to other servers. This keeps a cache available, but it can make different proxies disagree about the ring for a moment. For persistent data, keep auto_eject_hosts: false and give each backend its own replicas and failover.
Proxy or cluster-aware client?
| Cluster-aware client | Proxy | |
|---|---|---|
| Latency | Direct to the right node | One extra hop |
| Client requirements | Cluster support in every language used | Any Redis client |
| Connections to the servers | Every client to every node | Pooled by the proxy |
| Moving parts | None besides Redis | Proxy fleet to deploy, monitor and scale |
| Typical fit | Modern services with good client libraries | Legacy or many-language clients, huge numbers of connections, managed platforms |
Key Takeaways
- A proxy presents many shards as one Redis endpoint and does the routing centrally.
- Benefits: any client works, connections are pooled, routing changes in one place.
- Costs: one extra hop, a component that must itself be highly available, and command restrictions.
- Twemproxy shards plain instances with consistent hashing and does not support cluster backends. Envoy supports both plain and Cluster backends. Both limit commands to those that can be routed by key.
- Compare the proxy's supported commands with your application's before adopting one.
Practice
- (Beginner) Run Twemproxy with the configuration above against three Docker Redis instances. Insert 1,000 keys through the proxy and check
DBSIZEon each backend.- (Intermediate) Stop one backend while a loop reads and writes through the proxy with
auto_eject_hosts: true. Record the errors and when they stop. Then restart the backend and explain what happened to the keys written during the outage.- (Interview) "We have 3,000 application processes, each opening connections to every node of a 30-node Redis Cluster. The nodes spend a lot of CPU on connection handling. What are the options?" Hint: count the connections per node, then compare connection pooling in clients, a proxy layer, and fewer processes with larger pools.
Active-Active Geo-Replication
Everything so far has a single place where each key can be written: one primary per shard. For an application served from several continents, that means writes from far away pay a cross-ocean round trip (often 100-200 ms), and a regional outage stops writes for the keys whose primary was there. Active-Active databases remove this limit: every region holds a full copy and accepts writes locally, and the regions exchange changes in the background.
Availability
Active-Active is a feature of the commercial products, Redis Software (self-managed) and Redis Cloud. It is not part of Redis Open Source or Valkey, where replication always has one writable primary per shard. The terms used by Redis:
- An Active-Active database spans two or more participating clusters, typically one per region.
- The copy of the database in each cluster is an instance. (The older name for the whole database is CRDB, conflict-free replicated database.)
- Databases are created in the admin console, through the REST API or with the
crdb-clicommand-line tool.
Europe US East Asia
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ instance (R/W) │◀──────▶│ instance (R/W) │◀───────▶│ instance (R/W) │
└───────▲─────────┘ ▲ └───────▲─────────┘ ▲ └───────▲─────────┘
│ │ │ │ │
local apps: multi-directional, asynchronous local apps:
sub-ms reads sync of changes between instances sub-ms reads
and writes and writes
How it works
- Each instance applies local writes immediately and acknowledges them, with the latency of a local Redis.
- A syncer process ships changes to the other instances, using a replication backlog and partial or full syncs, in a way similar to replication (7.1). The flow is multi-directional.
- Concurrent writes to the same key in different regions are merged with CRDTs (next topic) instead of being overwritten blindly. Vector clocks track which operations each instance has seen, so causally ordered operations are applied in order.
- The consistency model is strong eventual consistency: instances can briefly differ, but once they have received the same set of updates, they hold the same value, without coordination or manual conflict resolution.
- An optional causal consistency mode preserves the order of operations as they are synchronized among instances.
What changes for the application
| Aspect | Behavior |
|---|---|
| Latency | Reads and writes are served by the nearest instance, with local latency (Redis documents sub-millisecond) |
| Region failure | Clients of the failed region switch to another region's instance. The other regions continue normally |
| Reading another region's write | Not immediately visible: it arrives after the cross-region sync |
| Data types | Merged with type-specific rules (next topic). Counters add up, sets merge, strings use last-write-wins |
| Lua scripts | Supported, always replicated by their effects |
| Eviction | The default policy is noeviction. If an instance runs out of memory, it is marked inconsistent, stops serving traffic and is fully reconciled by the syncer |
DBSIZE and key counts | Can include internal versions of keys with unresolved conflicts, so they differ from the number of logical keys |
| Deleted keys | Kept as tombstones until all instances have seen the deletion, then garbage-collected |
Design guidance
- Route each user to one home region when possible ("sticky region"). Conflicts then happen only during failovers.
- Prefer operations with merge-friendly semantics:
INCRBYinstead of read-modify-SET,SADD/SREMinstead of rewriting a serialized list. - Do not rely on Active-Active for global uniqueness or mutual exclusion. Two regions can both "acquire" the same lock at the same moment (chapter 8.2).
- Size each participating cluster for the whole dataset. Every region holds a complete copy.
Key Takeaways
- Active-Active lets every region write locally to its own instance of the same database. It is available in Redis Software and Redis Cloud, not in Redis Open Source.
- Writes are acknowledged locally and synced to the other instances asynchronously in all directions.
- Conflicts are resolved automatically with CRDTs and vector clocks, giving strong eventual consistency.
- It gives local latency and survives the loss of a region, but cross-region reads can be stale and global locks are unsafe.
- Design for it: sticky regions, merge-friendly commands, a full copy per region.
Practice
- (Beginner) For a shopping cart, a global view counter and a username registry, decide whether each would work well in an Active-Active database, and why.
- (Intermediate) Estimate the write latency seen by a user in Singapore for a single-primary deployment in Frankfurt versus an Active-Active deployment with instances in Frankfurt and Singapore. Then describe what the Frankfurt user sees immediately after the Singapore user updates a shared document.
- (Interview) "Our game needs low-latency writes on three continents and a global leaderboard. Would you use Active-Active, and how would you model the leaderboard?" Hint: think about how sorted-set scores merge (
ZINCRBYversusZADD) when two regions update the same member.
CRDTs in Redis
When two regions change the same key at the same moment, someone has to decide the result. Asking the regions to agree before each write would bring back the cross-region latency that Active-Active was meant to remove. CRDTs (conflict-free replicated data types) solve this differently: each data type comes with merge rules designed so that applying the same set of operations in any order gives the same result. No coordination is needed, and every instance converges to the same value.
An analogy: two people keep tally marks for the same event in two notebooks. Combining the notebooks means adding up the tallies, and the order in which you read them does not matter. Overwriting one notebook with the other ("last write wins") would lose half the count.
Two basic strategies
| Strategy | Rule | Good for |
|---|---|---|
| Operation-based merge | Replay the intent of each operation: add increments, add set members, keep both inserts | Counters, sets, lists, sorted-set increments |
| Last-write-wins (LWW) | When two writes truly conflict, keep one consistently | Values that are replaced as a whole (strings, sorted-set scores set with ZADD) |
For LWW, Redis first uses the vector clocks to see whether one write happened after the other. Only for truly concurrent writes does it compare the wall-clock timestamps stored with each operation, and the later one wins.
Merge rules per data type (Active-Active in Redis Software / Redis Cloud)
| Type / operation | Conflict rule |
|---|---|
Strings: SET vs SET | Last write wins |
Strings as counters: INCR, INCRBY, DECRBY | Increments from all instances add up (59-bit counters) |
| Hashes | Fields added on different instances are merged. HDEL only removes fields it has seen, so a concurrent HSET of that field wins. Field values behave like strings, and can be counters via HINCRBY |
Sets: SADD vs SREM | Add wins (observed-remove): a remove only deletes what that instance had seen |
| Sorted sets: membership | Add/update wins over a concurrent ZREM |
Sorted sets: score with ZADD | Last write wins |
Sorted sets: score with ZINCRBY | Increments from all instances add up |
| Lists | Concurrent inserts are all kept, ordered consistently. Concurrent pops may return the same element on two instances (at-least-once, not exactly-once) |
| Streams | Each region appends to its own part. XREADGROUP never skips entries but may redeliver; plain XREAD may skip entries if several regions write |
| HyperLogLog | Merged, but DEL wins over concurrent adds |
DEL vs concurrent update (sets, hashes, sorted sets, lists, streams) | Observed-remove: the delete removes only what was seen, so a concurrent add survives |
EXPIRE vs EXPIRE | The longer TTL wins. PERSIST (no expiry) beats any TTL |
Worked example: counter versus last-write-wins
time Europe instance US instance after sync (both)
t0 page:views = 100 page:views = 100
t1 INCRBY page:views 7 INCRBY page:views 3
t2 (local value 107) (local value 103) 110: increments add up
t3 SET status "open" SET status "closed" the write with the later
timestamp, on both instances
If the counter had been updated with GET, then SET of the new value, the result would be 107 or 103, and one region's increments would be lost. Use the operation that expresses your intent: increments for counters, set additions for collections.
Worked example: add-wins set
time Europe US after sync
t0 tags = {a, b} tags = {a, b}
t1 SREM tags a SADD tags c
t2 {b} {a, b, c} {b, c}
Europe's SREM a had observed a, so a is removed everywhere. The US's SADD c was concurrent with it, so c survives. If the US had also run SREM c before syncing, while Europe concurrently ran SADD c, the result would again contain c: a remove only cancels the additions it has observed, so an element added concurrently elsewhere survives. That is usually what users expect from collaborative data such as tags or carts: nothing someone added disappears silently.
Modeling for CRDTs
import redis
r = redis.Redis(host="local-region-endpoint", decode_responses=True)
# Good: increments merge across regions
r.incrby("article:42:views", 1)
r.hincrby("cart:{u7}", "sku:123", 2) # quantity as a per-field counter
r.sadd("article:42:tags", "redis") # concurrent adds and removes merge
r.zincrby("leaderboard", 50, "player:9") # score changes add up
# Risky: read-modify-write of a whole value; one region's change is lost under LWW
views = int(r.get("article:42:views") or 0)
r.set("article:42:views", views + 1) # avoid in Active-Active
# Unsafe: uniqueness and locks assume a single writer
r.set("username:alice", "u7", nx=True) # both regions can succeed concurrently
For data that needs a single global decision (unique usernames, inventory that must never oversell, financial balances), route those writes to one home region, or use a system with consensus.
Key Takeaways
- CRDTs are data types with merge rules that converge to the same value regardless of the order of operations, without coordination.
- Redis Active-Active uses operation-based merging for counters, sets, sorted-set increments and lists, and last-write-wins for whole-value replacements.
- Removes are observed-remove: concurrent adds survive (add wins), except for HyperLogLog, where
DELwins. - The longer TTL wins, and lists and streams have at-least-once semantics for consumers.
- Express intent with merge-friendly commands (
INCRBY,SADD,ZINCRBY) and keep uniqueness and locks out of multi-region writes.
Practice
- (Beginner) Using the rules table, predict the merged result: Europe runs
SADD s xwhile the US runsDEL son a set that contained{y}. Then the same with a HyperLogLog key andPFADD.- (Intermediate) Implement a tiny in-memory simulation in Python of a grow-only counter (one count per region, value = sum) and of a last-write-wins register (value, timestamp). Apply the same operations in two different orders and check that both replicas converge.
- (Interview) "Why is a distributed lock built with
SET NXunsafe on an Active-Active database, even though each region's Redis executes commands atomically?" Hint: trace two regions runningSET lock:x me NXwithin the same 50 ms, before either change has been synchronized, and apply the string merge rule.
<a id="8-application-design-patterns"></a>