4. Messaging and Atomic Operations
Individual Redis commands are atomic, but real applications rarely need just one command: they broadcast events to many listeners, update several keys as one unit, push thousands of writes as fast as the network allows, or run read-check-write logic that must not be interrupted. This chapter covers the four tools Redis provides for this: Pub/Sub messaging, MULTI/EXEC transactions with optimistic locking, pipelining, and server-side scripting with Lua and Redis Functions. Choosing the right one decides whether your code is correct under concurrency and whether it is fast across a network.
<a id="41-publishsubscribe"></a>
4.1 Publish/Subscribe
This subchapter covers Redis's fire-and-forget messaging system: publishing to channels, subscribing by name or pattern, scaling Pub/Sub in a cluster with sharded channels, what Pub/Sub does and does not guarantee, and keyspace notifications, which turn changes to keys into Pub/Sub events.
PUBLISH and SUBSCRIBE
Many features need to tell several running processes "something just happened": a chat message was posted, a price changed, a configuration flag was flipped, a cache entry must be dropped on every app server. Polling a key every second wastes work and adds up to a second of delay. What you want is a push model: interested processes register once, and the server forwards each event to them the moment it occurs.
Publish/Subscribe (Pub/Sub) is that model. Publishers send messages to a named channel. Subscribers listen on channels. Publishers and subscribers do not know about each other: the channel name is the only thing they share. Think of a radio station: the station broadcasts on a frequency, anyone tuned in hears it, and anyone not tuned in at that moment misses it for good.
channel "news:sports"
publisher ──PUBLISH──▶ ┌────────────────────┐ ──▶ subscriber A (web server 1)
│ Redis server │ ──▶ subscriber B (web server 2)
└────────────────────┘ ──▶ subscriber C (analytics)
keeps no copy of the message
Commands
| Command | Purpose | Reply |
|---|---|---|
PUBLISH channel message | Send a message to a channel | Number of clients that received it |
SUBSCRIBE channel [channel ...] | Start listening on one or more channels | One confirmation per channel, then a stream of messages |
UNSUBSCRIBE [channel ...] | Stop listening (all channels if none given) | One confirmation per channel |
PUBSUB CHANNELS [pattern] | List channels with at least one subscriber | Array of names |
PUBSUB NUMSUB [channel ...] | Subscriber count per channel | Name/count pairs |
Key properties:
- Channels are not keys. They do not appear in
KEYS/SCAN, take no memory when nobody listens, and are not tied to a logical database: a message published whileSELECT 3is active reaches subscribers that selected database 0. PUBLISHis O(N + M), where N is the number of subscribers of the channel and M the number of active pattern subscriptions (see "Pattern Subscriptions").- A return value of 0 means nobody was listening, and the message is gone. In Redis Cluster the count only includes subscribers connected to the node that received the
PUBLISH.
The subscribed state
A connection that runs SUBSCRIBE enters subscribed mode. Under RESP2 it can then only run SUBSCRIBE, SSUBSCRIBE, PSUBSCRIBE, the matching UNSUBSCRIBE variants, PING, RESET and QUIT. Everything the server sends is a three-element array whose first element names the kind of message:
1) "subscribe" 1) "message" 1) "unsubscribe"
2) "news:sports" 2) "news:sports" 2) "news:sports"
3) (integer) 1 3) "Goal! 1-0" 3) (integer) 0
^ channels this ^ payload ^ channels left
connection
listens to
With RESP3 (HELLO 3, chapter 10.2), messages arrive as out-of-band push frames, so the same connection can keep running ordinary commands. In practice most clients still use a dedicated connection for subscriptions.
Example in two terminals
# Terminal 1: subscriber
127.0.0.1:6379> SUBSCRIBE news:sports news:weather
1) "subscribe"
2) "news:sports"
3) (integer) 1
1) "subscribe"
2) "news:weather"
3) (integer) 2
Reading messages... (press Ctrl-C to quit or any key to type command)
# Terminal 2: publisher
127.0.0.1:6379> PUBLISH news:sports "Goal! 1-0"
(integer) 1 # one subscriber received it
127.0.0.1:6379> PUBLISH news:politics "Election results"
(integer) 0 # nobody listening: dropped
# Terminal 1 immediately prints
1) "message"
2) "news:sports"
3) "Goal! 1-0"
In Python
A cache-invalidation bus: every app server keeps a local in-process cache and drops an entry when any server announces a change.
import json
import redis
r = redis.Redis(decode_responses=True)
local_cache: dict[str, dict] = {}
def update_product(product_id: int, fields: dict) -> None:
r.hset(f"product:{product_id}", mapping=fields) # source of truth
receivers = r.publish("invalidate:product", json.dumps({"id": product_id}))
print(f"invalidation delivered to {receivers} servers")
def run_listener() -> None:
p = r.pubsub(ignore_subscribe_messages=True) # hide the "subscribe" confirmations
p.subscribe("invalidate:product") # uses its own dedicated connection
for msg in p.listen(): # blocks, yields one dict per message
data = json.loads(msg["data"])
local_cache.pop(f"product:{data['id']}", None)
p.listen() blocks forever. In an application, run it in a background thread (p.run_in_thread()), or poll with p.get_message(timeout=1.0) inside your own loop.
Key Takeaways
- Pub/Sub pushes messages from publishers to all current subscribers of a channel. The server stores nothing.
- Channels are a separate namespace from keys and ignore the selected database.
PUBLISHreturns how many subscribers received the message; 0 means it was dropped.- Under RESP2 a subscribed connection cannot run normal commands, so use a dedicated connection.
Practice
- (Beginner) Open three
redis-clisessions. Subscribe two of them tochat:lobby, publish from the third, and check the return value ofPUBLISH. Then runPUBSUB NUMSUB chat:lobbyandPUBSUB CHANNELS.- (Intermediate) Write a small chat program: each process subscribes to
chat:<room>in a background thread and publishes lines typed by the user. Make sure a process does not print its own messages twice.- (Interview) "Why does a Redis client need a separate connection for subscriptions, and what changes with RESP3?" Hint: think about how a client matches replies to requests on a RESP2 connection, and what out-of-band push messages allow.
Pattern Subscriptions
Subscribing by exact name works when the set of channels is known in advance. Often it is not: a monitoring tool wants every orders:* event regardless of region, or a dashboard wants all sensor channels even when new sensors appear later. Subscribing to each name individually means tracking which channels exist, which Pub/Sub has no reliable way to tell you.
Pattern subscriptions solve this. PSUBSCRIBE takes glob-style patterns, and the server delivers every message whose channel name matches.
| Pattern syntax | Matches | Example |
|---|---|---|
* | Any sequence of characters | orders:* matches orders:eu, orders:us:east |
? | Exactly one character | room:? matches room:1 but not room:12 |
[abc] / [a-z] | One character from the set or range | log:[ew]* matches log:error, log:warn |
[^a] | One character not in the set | node:[^0] excludes node:0 |
\ | Escape the next character | price\* matches the literal name price* |
Commands and message format
PSUBSCRIBE pattern [pattern ...]
PUNSUBSCRIBE [pattern ...]
PUBSUB NUMPAT # number of distinct patterns subscribed by all clients
Pattern matches arrive as four-element pmessage arrays, so the subscriber knows both which pattern matched and the real channel:
1) "pmessage"
2) "orders:*" # the pattern that matched
3) "orders:eu" # the actual channel
4) "{\"id\": 1001}" # payload
Example
# Subscriber
127.0.0.1:6379> PSUBSCRIBE orders:* alerts:?
1) "psubscribe"
2) "orders:*"
3) (integer) 1
1) "psubscribe"
2) "alerts:?"
3) (integer) 2
# Publisher
127.0.0.1:6379> PUBLISH orders:eu "order 1001"
(integer) 1
127.0.0.1:6379> PUBLISH alerts:high "disk full"
(integer) 0 # alerts:? needs exactly one character after the colon
Duplicate delivery
A connection subscribed to orders:eu and to orders:* receives every message on orders:eu twice: once as message, once as pmessage. Likewise, two overlapping patterns both fire. Design subscribers so that either their subscriptions do not overlap or the handler tolerates duplicates.
Cost
Every PUBLISH checks the channel against every active pattern on the server, which is why PUBLISH is O(N + M) with M = number of patterns. A few dozen patterns are irrelevant. Tens of thousands of distinct patterns (for example one per user session) make every publish slow, for every publisher. Prefer exact channels for per-entity subscriptions, and patterns for a small number of broad listeners such as loggers and dashboards.
import redis
r = redis.Redis(decode_responses=True)
p = r.pubsub(ignore_subscribe_messages=True)
def on_order(msg: dict) -> None:
region = msg["channel"].split(":", 1)[1] # "orders:eu" -> "eu"
print(f"[{msg['pattern']}] region={region} data={msg['data']}")
p.psubscribe(**{"orders:*": on_order}) # handler called for each pmessage
thread = p.run_in_thread(sleep_time=0.01, daemon=True)
Key Takeaways
PSUBSCRIBEuses glob patterns (*,?,[...]) and deliverspmessageframes with pattern, channel and payload.- Overlapping channel and pattern subscriptions on one connection cause duplicate deliveries.
- Every publish is checked against all patterns, so keep the number of distinct patterns small.
Practice
- (Beginner) Subscribe to
sensor:*:tempand publish tosensor:kitchen:temp,sensor:kitchen:humidityandsensor:attic:temp. Predict which messages arrive before you run the commands.- (Intermediate) Subscribe one connection to both
news:sportsandnews:*, publish one message, and observe the result. Then write a Python handler that deduplicates by a message ID included in the payload.- (Interview) "A team gives every logged-in user a pattern subscription like
user:<id>:*. With 50,000 users online, publishing slows down for everyone. Why, and what would you change?" Hint: look at the complexity ofPUBLISHand at how many channels each user actually needs.
Sharded Pub/Sub
In a single Redis server, Pub/Sub is cheap. In Redis Cluster (chapter 7.3) classic Pub/Sub has a scaling problem: a subscriber may be connected to any node, so every PUBLISH is broadcast over the cluster bus to every node. Adding nodes adds capacity for keys, but Pub/Sub traffic is copied to all of them, so total Pub/Sub throughput does not grow with the cluster, and each new node adds more internal traffic.
Sharded Pub/Sub (Redis 7.0+) applies the cluster's key sharding to channels. A shard channel name is hashed to a slot, exactly like a key, and messages stay inside the shard (primary plus its replicas) that owns that slot. Each shard only handles its own channels, so throughput scales with the number of shards.
Classic Pub/Sub in a 3-shard cluster Sharded Pub/Sub
PUBLISH chat:42 ──▶ node A SPUBLISH chat:42 ──▶ shard owning slot(chat:42)
├──bus──▶ node B (primary + its replicas only)
└──bus──▶ node C
every message reaches every node message reaches one shard
Commands
| Classic | Sharded | Notes |
|---|---|---|
PUBLISH | SPUBLISH shardchannel message | Routed by slot like a key command |
SUBSCRIBE | SSUBSCRIBE shardchannel [shardchannel ...] | All channels in one call must hash to the same slot |
UNSUBSCRIBE | SUNSUBSCRIBE [shardchannel ...] | |
PUBSUB CHANNELS / NUMSUB | PUBSUB SHARDCHANNELS / SHARDNUMSUB | Report only the node you ask |
PSUBSCRIBE | (none) | No pattern form: a pattern could match channels on every shard |
Messages arrive as smessage frames: ["smessage", channel, payload].
Points to know:
- Shard channels are a separate namespace: a
SUBSCRIBE chat:42client does not receiveSPUBLISH chat:42. - Hash tags work as for keys:
chat:{room42}:messagesandchat:{room42}:typingland in the same slot and can be subscribed together. - If a slot migrates to another node (resharding or failover), the server pushes an
sunsubscribeto affected clients. Cluster-aware clients resubscribe on the new owner. Messages published in between are lost. - On a standalone server the sharded commands also work, which keeps code portable between development and a cluster.
Example
# Cluster started with redis-cli -c (follows redirections)
127.0.0.1:7000> SSUBSCRIBE chat:{room42}:messages chat:{room42}:typing
-> Redirected to slot [10342] located at 127.0.0.1:7001
1) "ssubscribe"
2) "chat:{room42}:messages"
3) (integer) 1
...
127.0.0.1:7000> SSUBSCRIBE chat:a chat:b
(error) CROSSSLOT Keys in request don't hash to the same slot
from redis.cluster import RedisCluster
rc = RedisCluster(host="127.0.0.1", port=7000, decode_responses=True)
# Publisher: routed to the shard that owns the channel's slot
rc.spublish("chat:{room42}:messages", "hello")
# Subscriber: the client connects to the right node for each channel
p = rc.pubsub()
p.ssubscribe("chat:{room42}:messages")
while True:
msg = p.get_sharded_message(timeout=1.0) # returns None when nothing arrived
if msg and msg["type"] == "smessage":
print(msg["channel"], msg["data"])
Choosing between them
| Situation | Use |
|---|---|
| Standalone server or Sentinel setup | Classic or sharded, no difference in cost |
| Cluster with many channels (per room, per user, per entity) | Sharded |
| Cluster with a few global broadcast channels that every node's clients need | Classic is acceptable |
| Need pattern subscriptions | Classic only |
Key Takeaways
- Classic Pub/Sub in Redis Cluster broadcasts every message to every node, so it does not scale horizontally.
- Sharded Pub/Sub (7.0+) hashes channel names to slots and keeps messages within one shard.
- Shard channels are a separate namespace, have no pattern subscriptions, and follow the same-slot rule for multi-channel calls.
- Slot migrations unsubscribe clients, which must resubscribe on the new owner.
Practice
- (Beginner) On a standalone server, run
SSUBSCRIBE newsin one terminal andPUBLISH news hiin another. Explain the return value. Then useSPUBLISH.- (Intermediate) Start a 3-node local cluster, compute
CLUSTER KEYSLOT chat:{room1}andCLUSTER KEYSLOT chat:{room2}, and verify withPUBSUB SHARDCHANNELSon each node where the subscriptions live.- (Interview) "A chat service on a 10-shard Redis Cluster uses
PUBLISH room:<id>. CPU and network usage grow on every node as rooms are added, even nodes with no subscribers. Explain and propose a fix." Hint: follow a single message through the cluster bus.
Delivery Guarantees and Limitations
Pub/Sub looks like a message queue, and teams often use it as one until messages start disappearing. Understanding exactly what it promises prevents that mistake.
At-most-once delivery
Redis Pub/Sub delivers each message at most once to each subscriber that is connected at the moment of publishing. Nothing is written to memory or disk for later, and nothing is acknowledged.
| Situation | What happens to the message |
|---|---|
| No subscriber connected | Dropped (PUBLISH returns 0) |
| Subscriber disconnects, then reconnects | Messages published while it was away are lost |
| Subscriber crashes while processing a message | That message is lost; Redis never knew it was not handled |
| Subscriber reads too slowly | Messages pile up in its output buffer until the limit is hit, then the connection is closed and the backlog is lost |
| Primary fails over to a replica | In-flight messages are lost; subscribers must reconnect |
Ordering
Redis is single-threaded for command execution, so a given subscriber receives messages of one channel in the order the server processed the PUBLISH commands. There is no ordering guarantee across different channels or across shards.
Slow subscribers and output buffers
Redis pushes messages into each subscriber's output buffer and sends them as fast as the socket allows. If a subscriber cannot keep up, its buffer grows in Redis's memory. The client-output-buffer-limit setting protects the server:
# redis.conf: class hard-limit soft-limit soft-seconds (these are the defaults)
client-output-buffer-limit pubsub 32mb 8mb 60
A subscriber whose buffer exceeds 32 MB, or stays above 8 MB for 60 seconds, is disconnected. CLIENT LIST shows each client's buffer usage (omem), and the Redis log records every client closed for exceeding its output buffer limit.
Replication
PUBLISH is propagated to replicas, so subscribers connected to a replica also receive messages. This spreads subscriber load but does not add durability.
When to use something else
| Need | Pub/Sub | Streams (3.4) | Lists (2.3) |
|---|---|---|---|
| Fan-out to all current listeners | Yes | Yes (each reader with its own ID) | No |
| Message survives with no listener online | No | Yes | Yes |
| Replay history | No | Yes | No |
| Acknowledgment and redelivery | No | Yes (consumer groups) | Partial (LMOVE pattern) |
| Work sharing between consumers | No | Yes (consumer groups) | Yes |
| Latency | Lowest | Low | Low |
| Memory when idle | None | Grows until trimmed | Grows until consumed |
The "signal plus state" pattern
A common way to get Pub/Sub's low latency without losing correctness: keep the real data in keys or a stream, and use Pub/Sub only as a wake-up signal. A subscriber that missed signals simply re-reads the state when it reconnects.
import redis
r = redis.Redis(decode_responses=True)
def set_config(name: str, value: str) -> None:
r.hset("config", name, value) # durable state
r.incr("config:version") # lets readers detect missed changes
r.publish("config:changed", name) # low-latency hint only
def config_watcher(apply) -> None:
p = r.pubsub(ignore_subscribe_messages=True)
p.subscribe("config:changed") # subscribe first...
apply(r.hgetall("config")) # ...then load full state: no gap
for _ in p.listen():
apply(r.hgetall("config")) # re-read state on every signal
Subscribing before the initial read matters: if you read first, a change that lands between the read and the subscription would be missed.
Key Takeaways
- Pub/Sub is at-most-once: no persistence, no acknowledgments, no replay.
- Offline, crashed or slow subscribers lose messages. Slow ones are disconnected by
client-output-buffer-limit pubsub. - Order is preserved per channel for each subscriber, nothing more.
- Use streams or lists when every message must be processed, and use Pub/Sub as a wake-up signal next to durable state.
Practice
- (Beginner) Subscribe to a channel, stop the subscriber, publish three messages, and restart it. Which messages does it see?
- (Intermediate) Lower the limit with
CONFIG SET client-output-buffer-limit "pubsub 1mb 256kb 10", publish a burst of large messages to a subscriber that sleeps in its handler, and observe the disconnection inCLIENT LISTand the Redis log.- (Interview) "An order service publishes
order.createdover Redis Pub/Sub and the billing service subscribes. A few invoices go missing every week. What are the likely causes and how would you redesign it?" Hint: list every moment where the billing service is not connected and reading, then look at consumer groups in chapter 3.4.
Keyspace Notifications
Sometimes the event you care about is a change to the data itself: a session key expired, a cache entry was evicted, someone pushed to a list. Instead of making every writer also publish an event, Redis can publish these events itself. Keyspace notifications turn key operations into Pub/Sub messages on special channels.
Two channel families
For every event Redis can publish to two channels, which describe the same event from different angles:
Command: DEL session:abc (in database 0)
__keyspace@0__:session:abc message: "del" "what happened to this key?"
__keyevent@0__:del message: "session:abc" "which keys had this event?"
Enabling them
Notifications are off by default because they cost CPU on every write. The notify-keyspace-events setting takes a string of flags:
| Flag | Meaning |
|---|---|
K | Publish keyspace events (__keyspace@<db>__:<key>) |
E | Publish keyevent events (__keyevent@<db>__:<event>) |
g | Generic commands: DEL, EXPIRE, RENAME, COPY, ... |
$ | String commands |
l / s / h / z / t | List / set / hash / sorted set / stream commands |
x | Expired events (key deleted because its TTL elapsed) |
e | Evicted events (key removed by maxmemory eviction) |
d | Module key type events |
m | Key miss: a read found no key (not included in A) |
n | New key created (Redis 7.0+, not included in A) |
A | Alias for g$lshzxetd |
At least one of K or E must be present, otherwise nothing is published.
127.0.0.1:6379> CONFIG SET notify-keyspace-events Ex # keyevent channel, expirations only
OK
127.0.0.1:6379> CONFIG SET notify-keyspace-events KEA # everything except m and n (noisy)
OK
Example: reacting to expirations
# Terminal 1
127.0.0.1:6379> CONFIG SET notify-keyspace-events Ex
OK
127.0.0.1:6379> PSUBSCRIBE __keyevent@*__:expired
...
# Terminal 2
127.0.0.1:6379> SET session:abc data EX 2
OK
# Terminal 1, about 2 seconds later
1) "pmessage"
2) "__keyevent@*__:expired"
3) "__keyevent@0__:expired"
4) "session:abc"
Event names follow the commands: set, del, expire, expired, evicted, rename_from/rename_to, lpush, hset, incrby, zadd, xadd, new and others. Hash field expiration (Redis 7.4+) emits hexpired under the h class.
Caveats
- Expired events fire on deletion, not at the TTL. Redis removes expired keys lazily (on access) and with a sampling background cycle. On a busy instance with many TTLs the event can come noticeably later than the expiry time. Do not build precise schedulers on it; use a sorted set of due times instead (chapter 8.3).
- The value is gone. An
expiredordelevent carries only the key name. If the handler needs the data, store it in a second key without a TTL (or with a longer one) that the handler reads and deletes. - Same delivery rules as Pub/Sub. A disconnected listener misses events forever.
- Node-local in a cluster. Each node publishes events only for its own keys, and these events are not broadcast. A listener must connect to every primary (and track topology changes).
- Cost. Every matching write publishes up to two messages. Enable only the classes you consume.
import redis
r = redis.Redis(decode_responses=True)
r.config_set("notify-keyspace-events", "Ex") # usually set in redis.conf instead
def on_expired(msg: dict) -> None:
key = msg["data"] # keyevent channel: payload is the key name
if key.startswith("session:"):
sid = key.split(":", 1)[1]
shadow = r.getdel(f"session-shadow:{sid}") # copy kept without TTL for this purpose
print(f"session {sid} expired, last data: {shadow}")
p = r.pubsub(ignore_subscribe_messages=True)
p.psubscribe(**{"__keyevent@0__:expired": on_expired})
p.run_in_thread(sleep_time=0.01, daemon=True)
Key Takeaways
- Keyspace notifications publish key changes on
__keyspace@<db>__:<key>(event as payload) and__keyevent@<db>__:<event>(key as payload). - They are disabled by default. Enable only the flags you need, always including
KorE. - Expired events fire when Redis actually deletes the key, which can be late, and the value is no longer readable.
- They inherit Pub/Sub's at-most-once delivery and are node-local in a cluster.
Practice
- (Beginner) Enable
KEA, subscribe to__keyspace@0__:cart:*, then runHSET cart:1 item 2,EXPIRE cart:1 5andDEL cart:1. List the events you received.- (Intermediate) Create 100,000 keys with TTLs of 1 to 10 seconds and log the delay between each key's scheduled expiry and its
expiredevent. How does the delay change when you never read the keys?- (Interview) "We send a reminder email when a
reminder:<id>key expires, using keyspace notifications. Some reminders arrive late and some never arrive. Why?" Hint: consider how expiry is detected, what happens when the listener restarts, and how a cluster distributes keys.
<a id="42-transactions"></a>
4.2 Transactions
This subchapter covers Redis transactions: grouping commands with MULTI and EXEC, abandoning them with DISCARD, making them conditional with WATCH, how errors behave inside a transaction, and exactly which of the classic ACID properties Redis transactions provide.
MULTI and EXEC
Every Redis command is atomic on its own, but many operations span several commands. Transferring 10 points from Alice to Bob is two writes. Saving a product updates its hash and two indexes. If another client's commands run between yours, it can observe a half-finished state (Alice debited, Bob not yet credited), or its own writes can interleave with yours.
A transaction groups commands so that Redis executes them one after another, with no other client's command in between. Think of a supermarket checkout: you put all your items on the belt (queueing), and only when you pay (EXEC) is everything rung up in one go, while the next customer waits.
How it works
Client Redis
MULTI ───────────▶ OK connection enters transaction state
DECRBY a 10 ───────────▶ QUEUED checked for syntax, not executed
INCRBY b 10 ───────────▶ QUEUED
EXEC ───────────▶ runs both commands back to back, then replies:
◀─────────── 1) (integer) 90
2) (integer) 110
MULTIswitches the connection into transaction state. It never fails (except when nested).- Each following command is validated (name, number of arguments) and queued. The reply is
QUEUED, not the result. EXECexecutes the queue as one unit and returns an array with one reply per command, in order.
Example
127.0.0.1:6379> SET balance:alice 100
OK
127.0.0.1:6379> SET balance:bob 100
OK
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> DECRBY balance:alice 10 # redis-cli shows (TX) while queueing
QUEUED
127.0.0.1:6379(TX)> INCRBY balance:bob 10
QUEUED
127.0.0.1:6379(TX)> LPUSH transfers "alice->bob:10"
QUEUED
127.0.0.1:6379(TX)> EXEC
1) (integer) 90
2) (integer) 110
3) (integer) 1
What you cannot do inside MULTI
Because results arrive only at EXEC, you cannot read a value and decide what to write based on it within the transaction. GET inside MULTI returns QUEUED, not the value. "Transfer only if Alice has enough money" needs either WATCH (see "Optimistic Locking with WATCH") or a script (chapter 4.4).
Other rules:
MULTIcannot be nested: a secondMULTIreturns an error, and the transaction stays open.- Blocking commands such as
BLPOPdo not block inside a transaction. They behave like their non-blocking forms, because blocking would freeze the whole server. - In Redis Cluster all keys used in a transaction must hash to the same slot (use hash tags such as
balance:{family42}:alice, chapter 7.3). - If the connection drops before
EXEC, the queue is thrown away and nothing runs.
In Python
redis-py builds transactions with a pipeline. By default r.pipeline() wraps the buffered commands in MULTI/EXEC and sends everything in one network round trip (chapter 4.3):
import redis
r = redis.Redis(decode_responses=True)
def record_transfer(src: str, dst: str, amount: int) -> list:
pipe = r.pipeline() # transaction=True by default: MULTI ... EXEC
pipe.decrby(f"balance:{src}", amount) # buffered on the client, nothing sent yet
pipe.incrby(f"balance:{dst}", amount)
pipe.lpush("transfers", f"{src}->{dst}:{amount}")
return pipe.execute() # sends MULTI, 3 commands, EXEC; returns [90, 110, 1]
Note that this function does not check whether src has enough balance. That check is the subject of WATCH.
Key Takeaways
MULTIstarts queueing, each command repliesQUEUED, andEXECruns the queue as one uninterrupted unit.- Replies are only available after
EXEC, so you cannot branch on a value read inside the transaction. - Blocking commands do not block inside
MULTI, and in a cluster all keys must share a slot. - In redis-py,
r.pipeline()is aMULTI/EXECtransaction sent in one round trip.
Practice
- (Beginner) In
redis-cli, useMULTI/EXECto create a user hash, add the user ID to ausersset and increment ausers:countcounter. RunGET users:countinside the transaction and explain the reply.- (Intermediate) Open two
redis-clisessions. In the first, startMULTIand queueINCR ctwice. In the second, runSET c 100. ThenEXECin the first. Predict and verify the final value.- (Interview) "Redis is single-threaded and every command is atomic. Why would you ever need
MULTI/EXEC?" Hint: consider what other clients can do between two of your commands.
DISCARD
Sometimes a client starts a transaction and then decides not to commit it: input validation fails halfway through building the queue, an exception is raised in application code, or the user cancels. The queued commands must be thrown away, and the connection must leave transaction state so that it can be reused.
DISCARD does exactly that: it empties the queue, ends the transaction state and removes all WATCHes on the connection. None of the queued commands are executed. It is the "put everything back on the shelf" button at the checkout.
127.0.0.1:6379> SET stock:sku1 5
OK
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> DECRBY stock:sku1 3
QUEUED
127.0.0.1:6379(TX)> DISCARD
OK
127.0.0.1:6379> GET stock:sku1
"5" # nothing ran
127.0.0.1:6379> DISCARD
(error) ERR DISCARD without MULTI
DISCARD is not a rollback. It only cancels commands that have not run yet. Once EXEC has run, there is no command that undoes its effects.
Connection hygiene
The most common bug around DISCARD is forgetting it. If application code raises an exception between MULTI and EXEC and the connection goes back to a pool still in transaction state, the next user of that connection gets QUEUED instead of real replies. Good client libraries handle this for you:
import redis
r = redis.Redis(decode_responses=True)
def create_order(order_id: int, items: dict[str, int]) -> None:
with r.pipeline() as pipe: # context manager resets the pipeline on exit
pipe.hset(f"order:{order_id}", mapping={"status": "new"})
for sku, qty in items.items():
if qty <= 0:
raise ValueError(f"invalid quantity for {sku}") # nothing was sent: queue dropped
pipe.hincrby(f"order:{order_id}:items", sku, qty)
pipe.execute()
redis-py buffers commands on the client side until execute(), so an exception before that point sends nothing at all. When the pipeline is in immediate mode after WATCH, pipe.reset() (called by the context manager) sends UNWATCH or DISCARD as needed before the connection returns to the pool.
Key Takeaways
DISCARDdrops the queued commands, leaves transaction state and clears all watched keys.- It only cancels commands that have not executed. There is no rollback after
EXEC. - Always end a transaction with
EXECorDISCARDbefore reusing a connection. Context managers in client libraries do this for you.
Practice
- (Beginner) Start a transaction, queue three writes,
DISCARDit, and confirm none of the keys exist. Then tryEXECand read the error.- (Intermediate) Write a function that opens a raw
MULTIwithr.execute_command("MULTI")on a dedicated connection, raises an exception, and returns the connection to a pool withoutDISCARD. Show what the next caller sees, then fix it.- (Interview) "What is the difference between
DISCARDand a rollback in a relational database?" Hint: think about when each one can be used relative to the moment commands are applied.
Optimistic Locking with WATCH
MULTI/EXEC alone cannot express "read a value, compute something from it, and write the result, but only if nobody changed the value in the meantime". Consider two clients doubling a counter:
time Client A Client B
1 GET n -> 10
2 GET n -> 10
3 SET n 20
4 SET n 20 B's update is based on stale data
final value 20, expected 40
This is the classic lost update. Databases often solve it with locks that make B wait. Redis instead uses optimistic locking, also called check-and-set (CAS): assume there will be no conflict, and detect it at commit time if there was one. Like editing a shared document that warns you "this file changed since you opened it, please reload" rather than locking it while you type.
How WATCH works
WATCH key [key ...] # remember these keys for this connection
UNWATCH # forget all watched keys
WATCHthe keys whose values your decision depends on.- Read them with normal commands (the replies come back immediately).
- Compute the new values in your application.
MULTI, queue the writes,EXEC.- If any watched key was modified by anyone between
WATCHandEXEC,EXECexecutes nothing and returns a null reply. Retry from step 1.
EXEC and DISCARD always clear the watch list, whether the transaction ran or not.
# Client A
127.0.0.1:6379> WATCH n
OK
127.0.0.1:6379> GET n
"10"
# Client B, meanwhile: SET n 99
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> SET n 20
QUEUED
127.0.0.1:6379(TX)> EXEC
(nil) # aborted: n changed after WATCH
127.0.0.1:6379> GET n
"99"
What counts as a modification
- Any write command that changes the key, by any client, including the watching client itself between
WATCHandMULTI. - Deleting the key, or creating it when it did not exist (watching a missing key is valid and useful).
- The key expiring (since Redis 6.0.9), being evicted, or the database being flushed or swapped (
FLUSHDB,FLUSHALL,SWAPDB). - Rewriting the same value still counts as a change:
SET n 10whennis already10aborts the transaction.
Commands queued inside the transaction itself do not count, since they run at EXEC. WATCH inside MULTI is an error.
The retry loop in Python
A safe transfer that refuses to overdraw, and the fix for the product example in chapter 2.4, where the old category was read outside the transaction:
import redis
r = redis.Redis(decode_responses=True)
class InsufficientFunds(Exception):
pass
def transfer(src: str, dst: str, amount: int, max_retries: int = 10) -> None:
src_key, dst_key = f"balance:{src}", f"balance:{dst}"
with r.pipeline() as pipe:
for _ in range(max_retries):
try:
pipe.watch(src_key) # pipeline switches to immediate mode
balance = int(pipe.get(src_key) or 0) # executed right away, returns the value
if balance < amount:
pipe.unwatch()
raise InsufficientFunds(src)
pipe.multi() # back to buffered mode: start queueing
pipe.decrby(src_key, amount)
pipe.incrby(dst_key, amount)
pipe.execute() # raises WatchError if src_key changed
return
except redis.WatchError:
continue # someone touched src_key: re-read and retry
raise RuntimeError("too much contention, giving up")
def save_product(product_id: int, fields: dict) -> None:
k = f"product:{product_id}"
def tx(pipe: redis.client.Pipeline) -> None: # called again automatically on WatchError
old_category = pipe.hget(k, "category") # immediate read under WATCH
pipe.multi()
pipe.hset(k, mapping=fields)
if old_category and old_category != fields["category"]:
pipe.srem(f"category:{old_category}", product_id)
pipe.sadd(f"category:{fields['category']}", product_id)
r.transaction(tx, k) # helper: WATCH k, run tx, EXEC, retry
When WATCH is the wrong tool
Optimistic locking works well when conflicts are rare. Under heavy contention on a single key (a hot counter, a flash-sale stock level) most attempts fail and retry, wasting round trips and CPU, and some clients may starve. In those cases:
| Situation | Better choice |
|---|---|
| Update is expressible as one command | INCRBY, HINCRBY, SET ... NX, LMOVE, ... |
| Read-decide-write logic on a hot key | Lua script or function (chapter 4.4): runs on the server, never conflicts |
| Long critical section involving external systems | A distributed lock (chapter 8.2) |
Key Takeaways
WATCHturns a transaction into a check-and-set:EXECreturns null and runs nothing if a watched key changed.- Reads between
WATCHandMULTIreturn real values, so you can decide what to write. - Any change counts, including by the same client, expiry, eviction and flushes.
EXEC/DISCARDclear all watches. - Always wrap it in a bounded retry loop. Under high contention, prefer single atomic commands or scripts.
Practice
- (Beginner) Reproduce the aborted transaction from the example above with two
redis-clisessions. Then repeat it, but have client A itself runSET n 5betweenWATCHandMULTI. DoesEXECsucceed?- (Intermediate) Implement
zpop_if_score_below(key, limit)withWATCH: pop the lowest-scored member of a sorted set only if its score is belowlimit. Hammer it from 20 threads and countWatchErrorretries.- (Interview) "Compare optimistic locking with
WATCHto pessimistic locking. When does each perform better?" Hint: think in terms of conflict probability, the cost of a retry, and the cost of waiting for a lock holder that crashed.
Error Handling in Transactions
In a relational database, if one statement in a transaction fails, the whole transaction is usually rolled back. Redis behaves differently, and the difference surprises many developers. Errors in a Redis transaction come in two kinds, depending on when they are detected.
1. Errors while queueing (before EXEC)
The command is rejected when it is sent: an unknown command name, a wrong number of arguments, or a server condition such as being out of memory under maxmemory. The server remembers the failure, and EXEC then refuses to run anything:
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> SET a 1
QUEUED
127.0.0.1:6379(TX)> INCR # missing argument
(error) ERR wrong number of arguments for 'incr' command
127.0.0.1:6379(TX)> SET b 2
QUEUED
127.0.0.1:6379(TX)> EXEC
(error) EXECABORT Transaction discarded because of previous errors.
127.0.0.1:6379> EXISTS a b
(integer) 0 # nothing ran
2. Errors while executing (during EXEC)
The command is syntactically valid, so it is queued, but it fails when it actually runs, typically because a key holds the wrong type or a value is not a number. Redis reports the error for that command and keeps executing the rest. Commands that already ran are not undone:
127.0.0.1:6379> SET name "alice"
OK
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> INCR visits
QUEUED
127.0.0.1:6379(TX)> LPUSH name "x" # name is a string, not a list
QUEUED
127.0.0.1:6379(TX)> INCR name # "alice" is not an integer
QUEUED
127.0.0.1:6379(TX)> SET last_visit 1793180000
QUEUED
127.0.0.1:6379(TX)> EXEC
1) (integer) 1
2) (error) WRONGTYPE Operation against a key holding the wrong kind of value
3) (error) ERR value is not an integer or out of range
4) OK # executed despite the errors above
| Error kind | Detected | Examples | Effect on the transaction |
|---|---|---|---|
| Queueing error | When the command is sent | Unknown command, wrong arity, OOM | EXEC returns EXECABORT, nothing runs |
| Execution error | During EXEC | WRONGTYPE, non-numeric value for INCR | Only that command fails, the others run |
| Watch conflict | At EXEC | Watched key changed | EXEC returns null, nothing runs |
Why no rollback?
Redis's designers argue that execution errors are almost always programming errors (using a key with the wrong type, writing non-numeric data to a counter) that should be caught in development, not handled at runtime. Supporting rollback would require keeping undo information for every command, which would make Redis slower and more complex. The trade-off: your code must check each reply.
Handling errors in Python
import redis
r = redis.Redis(decode_responses=True)
pipe = r.pipeline()
pipe.incr("visits")
pipe.lpush("name", "x") # will fail at execution time
pipe.set("last_visit", 1793180000)
# Default: raises the first error, but only AFTER all commands have run on the server
try:
pipe.execute()
except redis.ResponseError as e:
print("one command failed, the others were still applied:", e)
# Alternative: get every reply, with exceptions in place of failed commands
pipe.incr("visits")
pipe.lpush("name", "x")
results = pipe.execute(raise_on_error=False)
for i, res in enumerate(results):
if isinstance(res, redis.ResponseError):
print(f"command {i} failed: {res}")
To avoid execution errors altogether, validate inputs and key types beforehand, keep a consistent key naming scheme so that one name never holds two types, and put conditional logic in a script that checks before it writes.
Key Takeaways
- Queueing errors (syntax, arity, OOM) abort the whole transaction with
EXECABORT. - Execution errors (
WRONGTYPE, non-numeric values) fail only the affected command. The rest still runs and nothing is rolled back. - Always inspect every reply of
EXEC. In redis-py, an exception fromexecute()does not mean nothing was applied.
Practice
- (Beginner) Create one transaction that ends in
EXECABORTand one that partially fails at execution time. Inspect the keys afterwards in both cases.- (Intermediate) Write a Python helper
run_tx(pipe)that executes a transaction withraise_on_error=Falseand logs the index, command name and error for every failed command.- (Interview) "Redis transactions don't support rollback. Is it fair to call them atomic?" Hint: separate "all or nothing executes" and "no one sees a partial state" from "failures are undone".
Atomicity Guarantees
"Transaction" means different things in different systems. Before relying on Redis transactions for correctness, it helps to compare them precisely with the ACID properties of a relational database: Atomicity, Consistency, Isolation and Durability.
What Redis guarantees
| Property | Redis transactions | Relational database (typical) |
|---|---|---|
| Atomicity | All queued commands are executed, or none are (DISCARD, EXECABORT, WATCH conflict, connection lost before EXEC). Execution errors do not undo the other commands | All or nothing, including rollback on error |
| Isolation | Full serializability: the queue runs without any other client's command in between. No other client can observe a partial state | Configurable isolation levels |
| Consistency | No constraints, foreign keys or schemas to enforce. Your application defines what "consistent" means | Enforced constraints |
| Durability | Depends on persistence settings (chapter 6). With AOF and appendfsync always, an acknowledged EXEC is on disk | Durable on commit |
Isolation is strong for a simple reason: Redis executes commands on a single thread (chapter 5.1), so while EXEC runs its queue, nothing else runs. The price is that a long transaction blocks every other client for its duration.
Replication and persistence
The commands of a transaction are written to the AOF and sent to replicas as one MULTI/EXEC block, so replicas also apply them atomically. If the server crashes in the middle of writing that block to the AOF, Redis detects the incomplete transaction when it loads the file at startup. Depending on aof-load-truncated it either drops the partial block or refuses to start, and redis-check-aof --fix can repair the file.
Atomicity tools compared
Transactions are one of several ways to get atomic behavior in Redis. Pick the simplest one that fits:
| Tool | Atomic | Can read and decide | Round trips | Typical use |
|---|---|---|---|---|
Single command (INCR, SET NX, LMOVE, HINCRBY, ...) | Yes | Only the logic built into the command | 1 | Counters, locks, queue moves |
Multi-key command (MSET, MSETNX, SINTERSTORE) | Yes | No | 1 | Batch writes |
MULTI/EXEC | Yes (isolation, no rollback) | No | 1 when pipelined | Update several keys together |
WATCH + MULTI/EXEC | Yes, with retries | Yes, client side | 2+ per attempt | Read-modify-write with low contention |
| Lua script / function (4.4) | Yes (isolation, no rollback) | Yes, server side | 1 | Conditional logic, hot keys |
A common misconception
"I put GET and SET in MULTI, so my check-then-write is safe."
MULTI
GET stock -> QUEUED the value is not available yet,
SET stock 4 -> QUEUED so "4" was computed from an earlier read
EXEC outside the transaction: still a race
Grouping does not protect a decision made from an earlier read. Only WATCH or a script does.
Key Takeaways
- Redis transactions give strong isolation (serial execution) and all-or-nothing execution, but no rollback of failed commands.
- Consistency is the application's job, and durability depends on persistence configuration.
- Transactions are replicated and persisted as one block.
- Choose the lightest tool: a single command, then
MULTI/EXEC, thenWATCH, then a script when you need server-side decisions.
Practice
- (Beginner) For each of these, say whether it needs a single command,
MULTI/EXEC,WATCHor a script: incrementing a page view counter; creating a user hash and adding its ID to a set; decrementing stock only if it is positive.- (Intermediate) Write a load test with 50 concurrent clients that each run "read stock, decrement if positive" 1,000 times, once naively, once with
MULTIonly, once withWATCH. Report the final stock and number of oversold units in each case.- (Interview) "Explain which ACID properties Redis transactions provide and which they do not, and give a scenario where the missing property causes a bug." Hint: a transaction in which the second of three commands fails with
WRONGTYPE.
<a id="43-pipelining"></a>
4.3 Pipelining
This subchapter covers pipelining, the technique of sending many commands without waiting for each reply: why network round trips dominate Redis performance, how pipelines work in clients and in redis-cli, how pipelines differ from transactions, and how to size batches.
Round-Trip Time Costs
Redis executes a simple command such as GET or INCR in about a microsecond. Yet a naive loop of 10,000 SET commands from an application server often takes several seconds. Almost all of that time is spent waiting, not computing.
Redis uses a request/response protocol over TCP. For each command the client sends the request, the server processes it, and the reply travels back. The client sends the next command only after reading the reply. The time from sending a request to receiving its reply, excluding server work, is the round-trip time (RTT).
Client Server
|──── SET k1 v ─────────────────────────────▶|
| | ~1 µs of work
|◀─────────────────────────────────── OK ────|
| (waits a full RTT per command) |
|──── SET k2 v ─────────────────────────────▶|
|◀─────────────────────────────────── OK ────|
...
Total time ≈ N × (RTT + server time) ≈ N × RTT
Think of a delivery driver who carries one parcel per trip. The drive dominates; carrying the parcel takes a second. Speeding up the warehouse changes nothing; carrying more parcels per trip does.
What RTT costs in practice
| Network path | Typical RTT | Max sequential commands/s on one connection |
|---|---|---|
| Same host, Unix socket | ~0.02-0.05 ms | 20,000-50,000 |
| Same host, loopback TCP | ~0.05-0.1 ms | 10,000-20,000 |
| Same data center / availability zone | ~0.2-0.5 ms | 2,000-5,000 |
| Cross availability zone | ~1-2 ms | 500-1,000 |
| Cross region | 30-100 ms | 10-30 |
A server that could handle hundreds of thousands of operations per second is limited to a few thousand per connection by latency alone.
Hidden cost: system calls
Even on localhost, where RTT is tiny, each command costs the server a read() system call to receive it and a write() to reply, each with a switch between user and kernel space. With one command per round trip, these switches cost far more than executing the command. Batching commands lets one read() bring in many commands and one write() send many replies.
Measuring RTT
# Continuously sample round-trip latency to the server (min/max/avg in ms)
redis-cli -h redis.internal --latency
# Latency history in 15-second windows, useful to spot periodic spikes
redis-cli -h redis.internal --latency-history -i 15
Key Takeaways
- For simple commands, network round-trip time, not server execution, dominates total time.
- Sequential throughput per connection is roughly 1 / RTT, regardless of server speed.
- Each round trip also costs system calls on both sides, so batching helps even on localhost.
Practice
- (Beginner) Run
redis-cli --latencyagainst a local server and, if available, against a remote one. Compute the maximum sequential commands per second for each.- (Intermediate) Write a Python script that runs 10,000
SETcommands in a loop and measures the total time. Compare the measured rate with 1 / RTT from--latency.- (Interview) "Your Redis server shows 5% CPU usage, but the application can only do 3,000 Redis operations per second per worker. What is the most likely bottleneck?" Hint: compare 1 / 3,000 seconds with the network latency between the application and Redis.
Pipelining Commands
If waiting for each reply is the problem, the solution is not to wait. Pipelining means sending many commands back to back and reading all replies afterwards. Redis supports this natively: the protocol is ordered, the server processes commands in the order they arrive on a connection, and it sends replies in the same order. No special command is needed. The client simply writes several requests before reading.
Without pipelining With pipelining
C ── SET k1 ──▶ S C ── SET k1, SET k2, SET k3 ──▶ S
C ◀── OK ────── S C ◀── OK, OK, OK ────────────── S
C ── SET k2 ──▶ S
C ◀── OK ────── S 1 round trip instead of 3
C ── SET k3 ──▶ S
C ◀── OK ────── S
The driver now fills the van with parcels and makes one trip.
At the protocol level
# Three commands in one write; the three replies come back together
printf 'SET a 1\r\nINCR a\r\nGET a\r\n' | nc localhost 6379
# +OK
# :2
# $1
# 2
In Python
import time
import redis
r = redis.Redis(decode_responses=True)
N = 10_000
start = time.perf_counter()
for i in range(N):
r.set(f"k:{i}", i) # N round trips
print(f"sequential: {time.perf_counter() - start:.2f}s")
start = time.perf_counter()
pipe = r.pipeline(transaction=False) # plain pipeline, no MULTI/EXEC
for i in range(N):
pipe.set(f"k:{i}", i) # buffered locally
replies = pipe.execute() # one write, then read N replies
print(f"pipelined: {time.perf_counter() - start:.2f}s, {len(replies)} replies")
Typical results over a 0.3 ms network: about 3 s sequential versus well under 0.1 s pipelined. Results arrive as a list in command order.
Reading results
Commands in a pipeline cannot depend on each other's results, because no reply is read until the end. A common pattern is to pipeline independent reads and combine the results afterwards:
def load_dashboard(user_ids: list[int]) -> list[dict]:
pipe = r.pipeline(transaction=False)
for uid in user_ids:
pipe.hgetall(f"user:{uid}")
pipe.zscore("leaderboard", uid)
replies = pipe.execute()
# replies alternate: [hash, score, hash, score, ...]
return [{**replies[i], "score": replies[i + 1]} for i in range(0, len(replies), 2)]
Bulk loading with redis-cli
For loading large data sets, redis-cli --pipe streams commands from a file at full speed and reports how many replies and errors it received:
# Each line is a command in inline form (or raw RESP for binary-safe data)
for i in $(seq 1 1000000); do echo "SET key:$i value:$i"; done > data.txt
cat data.txt | redis-cli --pipe
# All data transferred. Waiting for the last reply...
# errors: 0, replies: 1000000
Other forms of batching
- Multi-key commands such as
MSET,MGET,HSETwith many fields,SADDwith many members, orDEL k1 k2 ...are even cheaper than a pipeline, because the server parses one command. Prefer them when they fit. - Asynchronous clients pipeline automatically: ioredis, Lettuce and node-redis send commands from concurrent requests on one connection without waiting (chapter 10.1).
redis-benchmark -P 16measures throughput with 16 pipelined commands per request.
Key Takeaways
- Pipelining sends many commands before reading replies, turning N round trips into one.
- It needs no server-side command: replies come back in the same order as the requests.
- Commands in a pipeline cannot use each other's results.
- Prefer native multi-key commands when they exist, use
redis-cli --pipefor bulk loads.
Practice
- (Beginner) Run the timing script above against your server with N = 1,000, 10,000 and 100,000. Plot or tabulate the speedup.
- (Intermediate) Rewrite a function that checks 5,000 user IDs with
SISMEMBER banned <id>one by one so that it uses a pipeline, then again usingSMISMEMBER. Compare the three versions.- (Interview) "Pipelining makes 10,000 writes 50 times faster. Does it reduce the load on the Redis server too?" Hint: think about system calls, reply buffering and per-command parsing on the server.
Pipelining vs Transactions
Pipelines and transactions both involve sending a group of commands, and in redis-py both are created with r.pipeline(). That shared API leads many developers to treat them as the same thing. They solve different problems: a pipeline is a network optimization, a transaction is an execution guarantee.
| Pipeline | Transaction (MULTI/EXEC) | |
|---|---|---|
| Purpose | Fewer round trips | No interleaving with other clients |
| Where it happens | Client and network | Server |
| Other clients' commands can run in between | Yes | No |
| Server-side command | None | MULTI, EXEC, DISCARD, WATCH |
| Replies | Each command's reply, in order | QUEUED per command, results at EXEC |
| Cost on the server | None beyond the commands | Small queueing overhead, blocks others during EXEC |
| Cluster | Clients split it per node | All keys in one slot |
Interleaving
Pipeline from client A: SET x 1, SET y 1 (arrives in one packet)
Client B: GET x, GET y
Possible order on the server: A:SET x 1 B:GET x B:GET y A:SET y 1
B sees x = 1 and y = old value.
Transaction from A: MULTI, SET x 1, SET y 1, EXEC
Order on the server: A:SET x 1 A:SET y 1 B:GET x B:GET y
B sees both or neither.
In practice a short pipeline that arrives in one packet is often processed without interruption, but nothing guarantees it. Large pipelines are read in chunks, and other clients get served between them.
Combining them
The two are orthogonal and combine well:
Not pipelined Pipelined
Not transactional N round trips, no atomicity 1 round trip, no atomicity
Transactional N + 2 round trips (each QUEUED 1 round trip, atomic
reply awaited), atomic (redis-py default)
A MULTI/EXEC sent without pipelining is the slowest option: every queued command waits for its QUEUED reply. That is why redis-py's default r.pipeline() sends MULTI, the commands and EXEC together.
import redis
r = redis.Redis(decode_responses=True)
# Transactional pipeline (default): one round trip, atomic
pipe = r.pipeline() # transaction=True
pipe.hset("user:1", "name", "Ada").sadd("users", 1) # commands can be chained
pipe.execute()
# Plain pipeline: one round trip, other clients may interleave
pipe = r.pipeline(transaction=False)
for uid in range(1, 1001):
pipe.expire(f"session:{uid}", 3600) # independent writes: atomicity not needed
pipe.execute()
Which one to use
- Independent operations that only need speed (bulk writes, many reads): plain pipeline. It avoids the queueing overhead and, in a cluster, can span many slots.
- Several writes that must appear together: transactional pipeline.
- Read-decide-write:
WATCH(4.2) or a script (4.4). A pipeline cannot help, since no reply is available until the end.
Key Takeaways
- Pipelining reduces round trips. Transactions prevent interleaving. They are independent and can be combined.
- A plain pipeline gives no atomicity: other clients' commands may run between yours.
MULTI/EXECwithout pipelining costs one round trip per command. Always pipeline transactions.- In redis-py,
pipeline()is transactional by default. Passtransaction=Falsefor a plain pipeline.
Practice
- (Beginner) Using
MONITORin a separateredis-cli, runr.pipeline()andr.pipeline(transaction=False)with the same three commands. Compare what the server receives.- (Intermediate) Write a test that runs a plain pipeline of 100,000
INCR xfrom one client while another client repeatedly readsx. Show that the reader observes intermediate values. Repeat with a transactional pipeline.- (Interview) "A colleague says: 'We use pipelines, so our updates are atomic.' How do you respond?" Hint: describe what the server does when two clients' pipelines arrive at the same time.
Batch Size Considerations
If pipelining 1,000 commands is good, is pipelining 10 million in one batch better? No. Gains flatten quickly while costs keep growing. Choosing a sensible batch size keeps the throughput benefit without hurting memory, latency or error handling.
Diminishing returns
Total time ≈ (N / batch size) × RTT + N × per-command cost. Once the batch is large enough that RTT is small compared with the time spent processing the batch, larger batches save almost nothing:
Batch size Round trips for 1M commands Time spent waiting (RTT 0.5 ms)
1 1,000,000 500 s
10 100,000 50 s
100 10,000 5 s
1,000 1,000 0.5 s
10,000 100 0.05 s
1,000,000 1 0.0005 s (negligible gain over 10,000)
What grows with batch size
| Cost | Why |
|---|---|
| Client memory | The client buffers all commands, then holds all replies until the batch completes |
| Server memory | Replies accumulate in the client's output buffer until the client reads them. Normal clients have no output buffer limit by default, and incoming commands sit in the query buffer (client-query-buffer-limit, default 1 GB) |
| Latency for the caller | Nothing is returned until the whole batch is done |
| Timeouts | A huge batch can exceed the client's socket timeout even though Redis is healthy |
| Error handling | If one batch of a million commands fails midway, it is hard to know what was applied and what to retry |
| Large replies | HGETALL or LRANGE on big keys in one batch can produce hundreds of MB of replies |
Pipelines do not block other clients the way a single slow command does: the server processes commands as they are read and serves other connections between reads. The concern is memory and recoverability, not fairness.
Guidelines
- Start with batches of 100 to 10,000 commands and measure. The Redis documentation suggests around 10,000 as a reasonable size for bulk work.
- Size batches by bytes as well as count when values are large: a few MB per batch is a good target.
- Stream with a fixed batch size instead of building one giant pipeline.
- Make writes idempotent (
SET,HSET,SADDrather thanINCR,LPUSH) so that a failed batch can simply be retried. - In Redis Cluster, cluster-aware clients split a pipeline into one sub-pipeline per node. Grouping keys by slot (or hash tag) keeps sub-pipelines large.
Chunked loading in Python
from collections.abc import Iterable, Iterator
from itertools import islice
import redis
r = redis.Redis(decode_responses=True)
def chunks(items: Iterable, size: int) -> Iterator[list]:
it = iter(items)
while batch := list(islice(it, size)):
yield batch
def load_products(products: Iterable[dict], batch_size: int = 5_000) -> int:
loaded = 0
for batch in chunks(products, batch_size):
pipe = r.pipeline(transaction=False)
for p in batch:
pipe.hset(f"product:{p['id']}", mapping=p) # idempotent: safe to retry
try:
pipe.execute()
except redis.ConnectionError:
pipe = r.pipeline(transaction=False) # retry this batch only, once
for p in batch:
pipe.hset(f"product:{p['id']}", mapping=p)
pipe.execute()
loaded += len(batch)
return loaded
Key Takeaways
- Throughput gains from pipelining flatten once the batch is large relative to the RTT. Beyond a few thousand commands the benefit is marginal.
- Large batches cost client and server memory, delay results, risk timeouts and complicate retries.
- Use fixed-size batches (100 to 10,000 commands, a few MB), keep writes idempotent and measure on your own network.
Practice
- (Beginner) Load 1,000,000 keys with batch sizes 10, 100, 1,000, 10,000 and 100,000. Record the total time for each and find where the curve flattens.
- (Intermediate) Pipeline 50,000
HGETALLcalls on hashes of about 1 KB each in one batch while watchingused_memoryandCLIENT LIST(omem) from another session. Then repeat in batches of 1,000.- (Interview) "You need to migrate 200 million keys from one Redis to another as fast as possible without affecting production traffic. How do you choose the batch size and handle failures?" Hint: consider throughput versus memory,
SCAN-based iteration, idempotent writes and checkpointing the scan cursor.
<a id="44-server-side-scripting"></a>
4.4 Server-Side Scripting
This subchapter covers running your own logic inside Redis: Lua scripts with EVAL, the script cache and EVALSHA, how keys and arguments are passed, Redis Functions as a persistent library of server-side code, and the atomicity and timeout rules that apply to both.
Lua Scripting with EVAL
Transactions cannot make decisions based on the data they touch, and WATCH turns into retry storms on hot keys. Many operations need exactly that: "decrement stock only if it is at least the requested quantity", "delete the lock only if I still own it", "allow the request only if the counter is below the limit". Doing these as read-then-write from the client takes two round trips and leaves a race in between.
Server-side scripting moves the logic to the data. Redis embeds a Lua 5.1 interpreter. You send a short script, Redis runs it atomically (no other command runs while it executes), and the script can read keys, branch on their values and write, all in one round trip. It is like handing the bank teller a written instruction "if my balance is at least 50, transfer 50 to Bob" instead of asking for your balance, walking away, and coming back to request the transfer.
Syntax
EVAL script numkeys [key [key ...]] [arg [arg ...]]
script: the Lua source code.numkeys: how many of the following arguments are key names. They are available in the script as theKEYStable, the rest asARGV(see "KEYS and ARGV").- Inside the script,
redis.call(command, ...)runs a Redis command and returns its reply.
127.0.0.1:6379> EVAL "return redis.call('SET', KEYS[1], ARGV[1])" 1 greeting hello
OK
127.0.0.1:6379> EVAL "return redis.call('GET', KEYS[1]) .. ' world'" 1 greeting
"hello world"
A real example: conditional stock reservation
-- KEYS[1] = stock key, KEYS[2] = reservations hash
-- ARGV[1] = quantity requested, ARGV[2] = order id
local stock = tonumber(redis.call('GET', KEYS[1]) or '0') -- GET returns false for a missing key
local qty = tonumber(ARGV[1])
if stock < qty then
return 0 -- not enough: change nothing
end
redis.call('DECRBY', KEYS[1], qty)
redis.call('HSET', KEYS[2], ARGV[2], qty)
return 1
127.0.0.1:6379> SET stock:sku42 5
OK
127.0.0.1:6379> EVAL "<script above>" 2 stock:sku42 reservations:sku42 3 order-1001
(integer) 1
127.0.0.1:6379> EVAL "<script above>" 2 stock:sku42 reservations:sku42 3 order-1002
(integer) 0 # only 2 left
redis.call vs redis.pcall
| Function | On a command error |
|---|---|
redis.call | Raises a Lua error. The script stops and the client receives the error |
redis.pcall | Returns a table {err = "message"}, so the script can handle it and continue |
Type conversion
Values cross the boundary between Redis replies and Lua types according to fixed rules. These cause most scripting bugs:
| Redis reply | Lua value | Lua return value | Redis reply | |
|---|---|---|---|---|
| Integer | number | number | Integer (fractional part truncated) | |
| Bulk string | string | string | Bulk string | |
| Array | table (array) | table (array) | Array, cut at the first nil | |
Status (OK) | {ok = "OK"} | {ok = "..."} / redis.status_reply(s) | Status | |
| Error | {err = "..."} (pcall) | {err = "..."} / redis.error_reply(s) | Error | |
| Nil | false | false / nil | Nil | |
true | Integer 1 |
Return a float as a string (return tostring(3.14)), otherwise the client receives 3.
The sandbox
Scripts run in a restricted environment: no file or network access, and no global variables (assigning one raises an error, so always use local). Useful libraries are available: string, table, math, cjson, cmsgpack, bit and struct, plus helpers such as redis.log and redis.sha1hex. Chapter 11.2 covers the security side of the sandbox.
Replication
Since Redis 7.0, scripts are always replicated by their effects: replicas and the AOF receive the write commands the script executed, wrapped in MULTI/EXEC, not the script itself. Scripts may therefore use non-deterministic commands such as TIME or SRANDMEMBER safely.
In Python
import redis
r = redis.Redis(decode_responses=True)
RESERVE = """
local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
if stock < tonumber(ARGV[1]) then return 0 end
redis.call('DECRBY', KEYS[1], ARGV[1])
redis.call('HSET', KEYS[2], ARGV[2], ARGV[1])
return 1
"""
ok = r.eval(RESERVE, 2, "stock:sku42", "reservations:sku42", 3, "order-1001")
In application code, prefer r.register_script() (see "EVALSHA and Script Caching").
Key Takeaways
EVALruns a Lua script atomically on the server, so read-decide-write logic needs one round trip and has no race.redis.callpropagates errors,redis.pcallreturns them as a table.- Watch the conversion rules: floats are truncated, arrays stop at
nil, a missing key reads asfalse. - Scripts are sandboxed (no globals, no I/O) and replicated by their effects.
Practice
- (Beginner) Write a script that increments a counter and sets a TTL only when the counter was just created (value 1). Test it with
EVALinredis-cli.- (Intermediate) Write a script that returns
{count, average}for the members of a sorted set whose scores fall between two bounds. Make sure the average is not truncated on the client.- (Interview) "Why is
if redis.call('GET', k) == ARGV[1] then redis.call('DEL', k) endsafe in a script but not as aGETfollowed byDELfrom the client?" Hint: consider what another client can do between the two commands in each case.
EVALSHA and Script Caching
Sending the full script source with every call wastes bandwidth, and parsing it every time wastes CPU. A 2 KB rate-limiting script called 50,000 times per second means 100 MB/s of script text alone.
Redis therefore keeps a script cache. Every script it receives is compiled once and stored under the SHA1 digest of its source. Afterwards, the client can invoke it by digest. It is like a coffee shop that, after you have spelled out a complicated order once, lets you say "the usual" the next time.
Commands
| Command | Purpose |
|---|---|
SCRIPT LOAD script | Compile and cache a script without running it. Returns its SHA1 |
EVALSHA sha1 numkeys key ... arg ... | Run a cached script. Fails with NOSCRIPT if it is not in the cache |
SCRIPT EXISTS sha1 [sha1 ...] | Check which digests are cached (1 or 0 each) |
SCRIPT FLUSH [ASYNC | SYNC] | Empty the cache |
EVAL_RO / EVALSHA_RO | Read-only variants (7.0+): rejected if the script writes. Can run on replicas |
127.0.0.1:6379> SCRIPT LOAD "return redis.call('INCRBY', KEYS[1], ARGV[1])"
"8cd00688c05c46bde4a2e60658ef20a2e5c0b248"
127.0.0.1:6379> EVALSHA 8cd00688c05c46bde4a2e60658ef20a2e5c0b248 1 counter 5
(integer) 5
127.0.0.1:6379> SCRIPT FLUSH
OK
127.0.0.1:6379> EVALSHA 8cd00688c05c46bde4a2e60658ef20a2e5c0b248 1 counter 5
(error) NOSCRIPT No matching script. Please use EVAL.
The digest is computed from the exact source text, so changing even one space produces a different SHA1.
The cache is volatile
The script cache lives in memory only. It is empty after a restart, it is per node, and a replica promoted after a failover may not have your scripts. A client must therefore always be ready for NOSCRIPT:
try EVALSHA sha ...
└─ NOSCRIPT? ──▶ EVAL script ... (runs and caches it again)
or SCRIPT LOAD, then EVALSHA
Most client libraries do this automatically:
import redis
r = redis.Redis(decode_responses=True)
# register_script computes the SHA1 locally; each call uses EVALSHA
# and falls back to loading the script on NOSCRIPT
release_lock = r.register_script("""
if redis.call('GET', KEYS[1]) == ARGV[1] then
return redis.call('DEL', KEYS[1])
end
return 0
""")
released = release_lock(keys=["lock:report"], args=["token-123"])
# Pipelines work too: the script is loaded before the pipeline is sent if needed
pipe = r.pipeline()
release_lock(keys=["lock:a"], args=["t1"], client=pipe)
release_lock(keys=["lock:b"], args=["t2"], client=pipe)
pipe.execute()
Do not generate scripts dynamically
Every distinct script text becomes a new cache entry. Building scripts with string formatting, such as f"return redis.call('GET', '{key}')", creates one entry per key and makes the cache grow without bound. Since Redis 7.4, scripts that arrive through EVAL are subject to an LRU limit (scripts loaded with SCRIPT LOAD are not), which prevents unbounded growth but evicts scripts and causes extra NOSCRIPT round trips. The rule is the same in every version: keep script text constant and pass everything variable through KEYS and ARGV. INFO memory shows the cache size in used_memory_scripts_eval and number_of_cached_scripts.
Key Takeaways
- Redis caches compiled scripts by the SHA1 of their source.
EVALSHAruns them without resending the code. - The cache is in memory, per node, and empty after restart or after some failovers. Always handle
NOSCRIPT. - Use your client's script helper (
register_scriptin redis-py), which usesEVALSHAwith an automatic fallback. - Never build script text dynamically. Pass variable data through
KEYSandARGV.
Practice
- (Beginner) Load a script with
SCRIPT LOAD, call it withEVALSHA, check it withSCRIPT EXISTS, thenSCRIPT FLUSHand observe theNOSCRIPTerror.- (Intermediate) Write your own
call_script(r, source, keys, args)helper that computes the SHA1 locally withhashlib, triesEVALSHA, and falls back toEVALonNOSCRIPT. Test it across a server restart.- (Interview) "After a Redis failover, an application starts logging thousands of
NOSCRIPTerrors per second and some operations are lost. What went wrong, and how should the client be designed?" Hint: where does the script cache live, and what should happen whenEVALSHAmisses?
KEYS and ARGV
A script receives its input in two separate lists: KEYS for the names of keys it will access, and ARGV for every other value. Technically both are just strings, and a script on a single server could read any key whose name it builds itself. The separation exists because Redis needs to know which keys a script will touch before running it:
- Redis Cluster uses
KEYSto route the call to the node that owns those keys and to check that they share a slot. - ACLs (chapter 11.1) check the declared keys against the user's key patterns.
- Proxies and tooling inspect
KEYSto understand what a call does.
A script that computes key names internally works by accident on a single server and breaks in a cluster, because the computed key may live on another node.
How the arguments are split
EVAL script 2 user:1:cart user:1:history sku42 3
│ └──── KEYS[1] ───┘└── KEYS[2] ──┘ └ARGV[1]┘└ARGV[2]┘
└ numkeys: the first 2 arguments after it are keys
Rules:
- Lua tables are 1-based: the first key is
KEYS[1]. - Everything arrives as a string. Convert numbers with
tonumber(ARGV[1])before comparing or doing arithmetic ("10" < "9"is true in a string comparison). #KEYSand#ARGVgive the counts, so a script can accept a variable number of keys.- In a cluster, all keys must hash to the same slot. Use hash tags such as
{user:1}:cartand{user:1}:history.
Example: sliding-window rate limiter
-- KEYS[1] = sorted set of request timestamps for one client
-- ARGV[1] = now (ms), ARGV[2] = window (ms), ARGV[3] = limit, ARGV[4] = unique request id
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window) -- drop requests outside the window
local count = redis.call('ZCARD', key)
if count >= limit then
return {0, count} -- rejected
end
redis.call('ZADD', key, now, ARGV[4])
redis.call('PEXPIRE', key, window) -- idle clients clean themselves up
return {1, count + 1} -- allowed, new count
import time
import uuid
import redis
r = redis.Redis(decode_responses=True)
rate_limit = r.register_script(open("rate_limit.lua").read())
def allow(client_id: str, limit: int = 100, window_ms: int = 60_000) -> bool:
allowed, count = rate_limit(
keys=[f"ratelimit:{{{client_id}}}"], # hash tag: {client_id}
args=[int(time.time() * 1000), window_ms, limit, uuid.uuid4().hex],
)
return allowed == 1
The current time is passed in ARGV rather than read with TIME inside the script. Both work since effects replication, but passing it in makes the script easier to test with fixed times.
Variable number of keys
-- Delete several keys only if all of them still hold the expected token
-- KEYS = keys to check, ARGV[1] = token
for i = 1, #KEYS do
if redis.call('GET', KEYS[i]) ~= ARGV[1] then
return 0
end
end
return redis.call('DEL', unpack(KEYS))
Key Takeaways
- Pass every key the script touches in
KEYSand everything else inARGV. Never build key names inside the script. - Declared keys let Redis Cluster route the call and let ACLs check permissions.
- Arguments are 1-based and always strings: convert with
tonumberbefore comparing numbers. - In a cluster, all keys of one call must share a slot (hash tags).
Practice
- (Beginner) Write a script that copies a hash field from
KEYS[1]toKEYS[2]under the field nameARGV[1], and call it withEVAL. What happens if you passnumkeysas 1 by mistake?- (Intermediate) Write a script that atomically moves a member from one sorted set to another, keeping its score, and returns the score. Run it on a local cluster with and without hash tags.
- (Interview) "A script works perfectly in development but fails with
CROSSSLOTor wrong results in production on Redis Cluster. What would you look for?" Hint: compare the keys declared inKEYSwith the keys the script actually accesses.
Redis Functions (FUNCTION LOAD, FCALL)
EVAL scripts have an awkward lifecycle. The code lives in the application, every client must ship the same source, the cache vanishes on restart, and there is no name or version to tell which logic is deployed. If two services use slightly different copies of "reserve stock", nothing warns you. Scripts are best thought of as part of the application, sent to Redis on demand.
Redis Functions (Redis 7.0+, also in Valkey) treat server-side code as part of the database instead. You load a named library once; its functions are then:
- Persisted in RDB and AOF, so they survive restarts.
- Replicated to replicas, so they survive failovers.
- Called by name, like built-in commands.
Think of EVAL as handing the teller a note each time, and functions as installing a new, named service at the bank that any customer can request.
Writing a library
#!lua name=inventory
-- Local helpers are private to the library
local function get_number(key)
return tonumber(redis.call('GET', key) or '0')
end
-- reserve(keys = {stock, reservations}, args = {qty, order_id})
local function reserve(keys, args)
local qty = tonumber(args[1])
if get_number(keys[1]) < qty then
return 0
end
redis.call('DECRBY', keys[1], qty)
redis.call('HSET', keys[2], args[2], qty)
return 1
end
-- available(keys = {stock}) is read-only, so it can run on replicas with FCALL_RO
local function available(keys, args)
return get_number(keys[1])
end
redis.register_function('inventory_reserve', reserve)
redis.register_function{
function_name = 'inventory_available',
callback = available,
flags = { 'no-writes' },
}
- The first line
#!lua name=<library>is mandatory: it selects the engine and names the library. - Function callbacks receive
keysandargsas parameters instead of the globalKEYS/ARGV. - Function names are global across all libraries, so prefix them with the library name.
Commands
| Command | Purpose |
|---|---|
FUNCTION LOAD [REPLACE] code | Load a library (fails if it exists, unless REPLACE) |
FCALL function numkeys key ... arg ... | Call a function |
FCALL_RO function numkeys ... | Call a function flagged no-writes, allowed on replicas |
FUNCTION LIST [LIBRARYNAME pattern] [WITHCODE] | Show libraries, functions, flags and optionally code |
FUNCTION DELETE library | Remove a library |
FUNCTION DUMP / FUNCTION RESTORE payload | Export and import all libraries (backups, migrations) |
FUNCTION STATS / FUNCTION KILL | Show the running function, stop a read-only one that runs too long |
FUNCTION FLUSH | Delete all libraries |
# Deploy (or update) the library from a file; -x passes stdin as the last argument
redis-cli -x FUNCTION LOAD REPLACE < inventory.lua
# "inventory"
redis-cli SET stock:sku42 5
redis-cli FCALL inventory_reserve 2 stock:sku42 reservations:sku42 3 order-1001
# (integer) 1
redis-cli FCALL_RO inventory_available 1 stock:sku42
# (integer) 2
import redis
r = redis.Redis(decode_responses=True)
with open("inventory.lua") as f:
r.function_load(f.read(), replace=True) # typically done by a deploy step, not per request
ok = r.fcall("inventory_reserve", 2, "stock:sku42", "reservations:sku42", 3, "order-1001")
left = r.fcall_ro("inventory_available", 1, "stock:sku42")
Scripts vs functions
EVAL scripts | Functions | |
|---|---|---|
| Where the code lives | Application, sent on demand | Database, loaded once |
| Survives restart / failover | No (cache is volatile) | Yes (persisted and replicated) |
| Invocation | By SHA1 or full source | By name |
| Code sharing | Each script self-contained | Library with shared local helpers |
| Deployment | Ships with the application | Separate step (CI, migration script) |
| Cluster | Works per call | FUNCTION LOAD must run on every primary |
| Best for | Small, application-specific logic | Shared, versioned server-side APIs |
Functions need a deployment process: load or REPLACE the library as part of releases, keep the Lua source in version control, and in a cluster run FUNCTION LOAD on every primary (for example with redis-cli --cluster call). New nodes added to a cluster do not receive libraries automatically.
Key Takeaways
- Functions are named server-side Lua code grouped in libraries, persisted and replicated with the data.
- A library starts with
#!lua name=<lib>and registers callbacks withredis.register_function. - Call them with
FCALL, orFCALL_ROfor functions flaggedno-writes. - They need an explicit deployment step, on every primary in a cluster.
Practice
- (Beginner) Load the
inventorylibrary, list it withFUNCTION LIST WITHCODE, restart Redis and confirm it is still available.- (Intermediate) Add an
inventory_release(keys, args)function that returns a reservation to stock, sharing the helper withreserve. Deploy the update withREPLACEand test both functions from Python.- (Interview) "When would you choose Redis Functions over
EVALSHA, and what new operational responsibilities do they bring?" Hint: consider persistence, failover, versioning, and how a new cluster node gets its code.
Script Atomicity and Timeouts
The property that makes scripts useful, that nothing else runs while a script executes, is also their main risk. Redis runs scripts and functions on the same single thread as every command (chapter 5.1). A script that takes 2 seconds blocks every client for 2 seconds: reads, writes, health checks, replication. Understanding the rules for atomicity, errors and timeouts is what separates safe scripts from outages.
Atomicity
A script or function is executed as one unit: no other command runs before it finishes, so no client can observe an intermediate state. Replicas and the AOF receive its writes wrapped in MULTI/EXEC, so they apply them as one unit too.
No rollback
Like transactions, scripts are atomic in isolation but do not roll back. If a script fails halfway (a redis.call raises an error, or a Lua runtime error occurs), the writes it made before the failure remain:
redis.call('SET', 'a', '1') -- applied
redis.call('LPUSH', 'a', 'x') -- WRONGTYPE error: script aborts here
redis.call('SET', 'b', '1') -- never runs
-- Result: a = "1", b missing. Nothing is undone.
Structure scripts as validate first, write last: do all reads and checks, return early on failure, and only then perform the writes.
What happens when a script runs too long
time ──────────────────────────────────────────────────────────────▶
script starts busy-reply-threshold (default 5 s)
│ other clients wait │ other clients now get: BUSY Redis is busy running a script...
▼ ▼
═══════════════════════╪═══════════════════════════════ script still running
│
├─ script has NOT written yet ──▶ SCRIPT KILL / FUNCTION KILL stops it
└─ script HAS written ──────────▶ only SHUTDOWN NOSAVE (kill server, lose
unsaved data) or wait for it to finish
busy-reply-threshold(calledlua-time-limitbefore Redis 7.0, default 5000 ms) does not stop the script. It only makes Redis start answering other clients withBUSYerrors instead of leaving them hanging, and acceptSCRIPT KILL,FUNCTION KILLandSHUTDOWN NOSAVE.- A script that has already written cannot be killed, because stopping it would expose a half-applied state and break atomicity.
SCRIPT KILLthen fails with anUNKILLABLEerror. SHUTDOWN NOSAVEstops the server without saving, so writes since the last persistence point are lost, which is why long-running write scripts are dangerous.
127.0.0.1:6379> EVAL "local i = 0 while true do i = i + 1 end" 0 # client 1: infinite loop
127.0.0.1:6379> GET foo # client 2, after 5 s
(error) BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
127.0.0.1:6379> SCRIPT KILL
OK # no writes yet: killable
# client 1 receives
(error) ERR Script killed by user with SCRIPT KILL...
Memory limits and flags
When used_memory is above maxmemory, a script that may write is rejected before it starts, unless it declares the allow-oom flag. Scripts can declare flags in a shebang line (Redis 7.0+), and functions declare them in register_function:
#!lua flags=no-writes
-- read-only script: allowed with EVAL_RO, on replicas, and when the server is over maxmemory
return redis.call('ZRANGE', KEYS[1], 0, 9)
| Flag | Effect |
|---|---|
no-writes | Script cannot write. Required for EVAL_RO/FCALL_RO and replicas |
allow-oom | May run even when memory is over maxmemory |
allow-stale | May run on a replica whose link to the primary is down (with replica-serve-stale-data no) |
no-cluster | Refuses to run in cluster mode |
allow-cross-slot-keys | May access keys from several slots on the same node (discouraged) |
Guidelines for safe scripts
- Keep scripts short and bounded: O(1) or O(log N) work, or loops with a fixed upper limit. A script that iterates over a whole large key belongs in the application, done in batches with
SCAN/HSCAN. - Check the duration of scripts in
SLOWLOG(chapter 12.1); a script appears there as oneEVAL/FCALLentry. - Validate before writing so that errors leave no partial state.
- Never wait inside a script: there is no sleep, and blocking commands such as
BLPOPdo not block in scripts. - Treat
busy-reply-thresholdas an alarm, not a safety net.
Key Takeaways
- Scripts and functions run atomically: nothing else executes until they finish, and their writes are replicated as one block.
- Errors do not roll back earlier writes. Validate first, write last.
- After
busy-reply-threshold(5 s), other clients getBUSY. A script that has not written can be killed. One that has written can only be stopped withSHUTDOWN NOSAVE. - Flags such as
no-writesandallow-oomdeclare what a script may do. Keep every script short and bounded.
Practice
- (Beginner) Run a script with an infinite loop that never writes, observe the
BUSYerror from a second client after 5 seconds, and stop it withSCRIPT KILL. Then repeat with a script that writes once before looping, and read the error fromSCRIPT KILL.- (Intermediate) Write a script that sets three keys but fails on the second one, and show which keys exist afterwards. Rewrite it so that it validates all preconditions first and either writes all three keys or none.
- (Interview) "A developer wants a Lua script that iterates over a 5-million-member set and deletes inactive users, to keep it atomic. What do you tell them?" Hint: estimate how long the script blocks every other client, and what happens after
busy-reply-thresholdif it has already deleted something.
<a id="5-internals-and-architecture"></a>