14. Multi-Cluster and Deployment Architectures
Everything so far has assumed one Kafka cluster. Real organizations rarely stop at one: they need a second cluster to survive the loss of a region, clusters close to users on several continents, and platforms that run Kafka on Kubernetes or hand it to a cloud provider altogether. This chapter covers how data moves between clusters (MirrorMaker 2, Cluster Linking, active-passive and active-active topologies, offset translation), how to design for high availability and disaster recovery (stretch clusters, failover and failback, RPO and RTO, backups), and the trade-offs of running Kafka on Kubernetes, as a managed service, or in serverless form.
<a id="141-replication-across-clusters"></a>
14.1 Replication Across Clusters
This section covers the tools that copy records from one Kafka cluster to another, the topologies built from them, and how consumers resume in the right place after switching clusters.
MirrorMaker 2
Theory
Replication inside a cluster (Section 2.4) keeps copies of each partition on several brokers, and the ISR mechanism makes writes wait for those copies. This works because brokers in one cluster sit a few milliseconds apart. Stretching that synchronous model across continents would add the round-trip time to every acks=all write, so data that must reach another region is copied asynchronously by a separate process that reads from one cluster and writes to another.
Common reasons to run more than one cluster and copy data between them:
| Reason | Example |
|---|---|
| Disaster recovery | A standby cluster in another region takes over if the primary region is lost |
| Geo-locality | Users in Europe and Asia produce and consume against a nearby cluster |
| Aggregation | Regional clusters feed a central analytics cluster |
| Isolation | A noisy analytics workload reads a copy instead of the critical production cluster |
| Migration | Moving from ZooKeeper to a new KRaft cluster, from on-premises to cloud, or between providers |
Analogy: intra-cluster replication is like several clerks writing the same entry into ledgers on the same desk before the customer leaves. Cross-cluster replication is a courier who regularly carries photocopies of the ledger to a branch office in another city. The branch is always a little behind, and if the head office burns down, whatever the courier had not yet collected is lost.
From MirrorMaker 1 to MirrorMaker 2
The original MirrorMaker was simply a consumer and a producer glued together. It copied records but nothing else: topic configurations, partition counts, ACLs, and consumer offsets stayed behind, and two-way setups looped records back and forth forever. MirrorMaker 2 (MM2, KIP-382, Kafka 2.4) replaced it. MM2 is a set of Kafka Connect connectors (Chapter 9), so it inherits Connect's scaling, task distribution, fault tolerance, and offset tracking.
| Connector | Runs against | Purpose |
|---|---|---|
MirrorSourceConnector | Consumes from source, produces to target | Copies records, creates remote topics, syncs partition counts and topic configs, writes offset syncs |
MirrorCheckpointConnector | Reads source group offsets, writes to target | Translates consumer group offsets and emits checkpoints (and optionally commits them on the target) |
MirrorHeartbeatConnector | Writes to target | Emits heartbeats so you can verify connectivity and measure replication latency |
Region east Region west
+-------------------------+ +--------------------------------+
| Cluster "east" | | Cluster "west" |
| | consume (over WAN) | |
| orders -------+------------------+ | east.orders (remote topic) |
| payments -------+---------------+ | | east.payments (remote topic) |
| __consumer_offsets ----+------------+ | | | east.checkpoints.internal |
| mm2-offset-syncs. | | | | | heartbeats |
| west.internal <-----+-------+ | | | | mm2-configs/offsets/status... |
+-------------------------+ | v v v +--------------------------------+
| +----------------------------+ ^
+--| MM2 (Connect workers), |--------+
| deployed in region west | produce (local)
+----------------------------+
Remote topics and replication policies
MM2 renames replicated topics according to a replication policy:
| Policy | Source topic orders on east becomes | Use when |
|---|---|---|
DefaultReplicationPolicy | east.orders on west | Active-active, aggregation, any topology where both clusters write. The prefix records provenance and prevents loops |
IdentityReplicationPolicy (Kafka 3.0+) | orders on west | One-way replication: migration and active-passive DR, where applications should use the same names after switching |
The prefix is what makes cycles safe: MM2 never replicates east.orders back to east, because the name says it came from there.
Example: a dedicated MM2 cluster
MM2 can run in three ways: as a dedicated MM2 cluster (connect-mirror-maker.sh, which starts Connect workers configured from one properties file), as connectors deployed on an existing Connect cluster through the REST API, or through an operator such as Strimzi (Section 14.3).
# mm2.properties
# Cluster aliases: used as remote-topic prefixes and as the prefix of per-cluster settings
clusters = east, west
east.bootstrap.servers = east-b1:9092,east-b2:9092,east-b3:9092
west.bootstrap.servers = west-b1:9092,west-b2:9092,west-b3:9092
# Security goes here too, per cluster, for example:
# east.security.protocol = SASL_SSL
# east.sasl.mechanism = SCRAM-SHA-512
# Enable one flow: east -> west
east->west.enabled = true
# Comma-separated names or regexes
east->west.topics = orders.*, payments.*
# Default excludes are .*[\-\.]internal, .*\.replica, __.*
east->west.groups = .*
# Replication factor of the remote topics MM2 creates on the target
replication.factor = 3
# MM2's own internal topics and the embedded Connect workers' topics
offset-syncs.topic.replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
config.storage.replication.factor = 3
offset.storage.replication.factor = 3
status.storage.replication.factor = 3
# What else to keep in sync besides records
sync.topic.configs.enabled = true
refresh.topics.interval.seconds = 60
emit.checkpoints.enabled = true
emit.checkpoints.interval.seconds = 30
sync.group.offsets.enabled = true
sync.group.offsets.interval.seconds = 30
# Parallelism: tasks are spread over all MM2 worker processes
tasks.max = 12
# Client overrides per cluster: the producer that writes into west
west.producer.compression.type = lz4
west.producer.linger.ms = 20
west.producer.batch.size = 262144
# Required when running several dedicated MM2 nodes (KIP-710, Kafka 3.5+)
dedicated.mode.enable.internal.rest = true
# Start one MM2 node; run the same command on 2-3 machines in region west for fault tolerance.
# --clusters west: this node only runs flows whose target is west
bin/connect-mirror-maker.sh mm2.properties --clusters west
# On the target, the remote topics and MM2's internal topics appear
bin/kafka-topics.sh --bootstrap-server west-b1:9092 --list
# east.checkpoints.internal
# east.orders.created
# east.payments.authorized
# heartbeats
# mm2-configs.east.internal
# mm2-offsets.east.internal
# mm2-status.east.internal
Deployment guidelines
- Consume remotely, produce locally. Run MM2 in the target region. A consumer tolerates WAN latency and simply fetches again, while a producer stuck on a slow link fills its buffer and blocks.
- Size for catch-up, not just steady state. After a network outage MM2 must replay the backlog faster than new data arrives. Plan for 2-3 times the normal throughput.
- Partition counts match. MM2 creates each remote topic with the source's partition count and writes each record to the same partition number, so per-key ordering is preserved.
- Offsets do not match. Records get new offsets on the target. This is why offset translation exists (see Offset Translation below).
- ACL sync is restricted.
sync.topic.acls.enabledcopies topic ACLs but not write permissions, so remote topics stay read-only for clients. Users, SCRAM credentials, and quotas are not copied at all.
Monitoring
MBean (kafka.connect.mirror:...) | Metric | Meaning |
|---|---|---|
type=MirrorSourceConnector,target=*,topic=*,partition=* | replication-latency-ms | Time from record timestamp on source to write on target |
| same | record-age-ms | Age of records when MM2 consumes them; grows when MM2 falls behind |
| same | byte-rate, record-rate | Replication throughput |
type=MirrorCheckpointConnector,source=*,target=*,group=*,topic=*,partition=* | checkpoint-latency-ms | Delay of offset checkpoints |
Also monitor the embedded consumer's records-lag-max and the Connect task states (FAILED tasks stop replication silently unless alerted on).
Limitations
- Asynchronous: anything not yet copied when the source fails is lost or stranded.
- At-least-once by default: retries and task restarts can write duplicates on the target (recent versions can enable exactly-once source support for MM2 in dedicated mode; check your version's documentation).
- No global ordering across clusters, and no transactional atomicity across clusters (Section 8.3).
- Schemas, connectors, users, and quotas live outside the topics and need their own replication plan.
Key Takeaways
- Cross-cluster replication is asynchronous by design. Synchronous replication across a WAN would put the round-trip time on every write.
- MM2 is three Connect connectors: source (records, topics, configs), checkpoint (consumer offsets), and heartbeat (liveness and latency).
DefaultReplicationPolicyprefixes remote topics with the source alias, which prevents loops.IdentityReplicationPolicykeeps names and suits one-way flows.- Run MM2 near the target cluster, size it for catch-up, and alert on
replication-latency-msand failed tasks. - Partition numbers are preserved, offsets are not.
Practice
- (Beginner) Start two single-node clusters with Docker on different ports. Configure a one-way MM2 flow
a->bfor topics matchingdemo.*, produce 1,000 records todemo.eventsona, and confirma.demo.eventsonbhas the same keys in the same partitions. Compare the end offsets of both topics.- (Intermediate) Change the partition count of
demo.eventsonafrom 3 to 6 and change itsretention.ms. Measure how long it takes MM2 to apply both changes onb, and identify which settings control that delay.- Interview question: "Why does MirrorMaker 2 rename topics by default, and when would you turn that off?" Hint: think about two clusters replicating to each other, and what an application must change after a failover in each naming scheme.
Cluster Linking
Theory
MM2 runs as a separate system: a consumer reads records, a producer writes them again, and the records get new offsets on the target. That adds components to operate, and the offset change forces offset translation. Cluster Linking is Confluent's alternative (Confluent Platform 7.0+ and Confluent Cloud, not part of Apache Kafka): the destination brokers themselves fetch from the source cluster using the same replication protocol followers use inside a cluster. The result is a mirror topic that is a byte-for-byte copy of the source topic, with identical offsets.
Analogy: MM2 is someone who reads a book aloud while another person types it out, producing a new book with different page numbers. Cluster Linking photocopies the book, so page 312 in the copy is page 312 in the original.
Source cluster (prod) Destination cluster (dr)
+----------------------+ +-------------------------------------+
| orders P0 leader | <--- replica fetch --| orders (mirror topic, read-only) |
| offsets 0..9,999 | (cluster link | offsets 0..9,987 (same offsets, |
| __consumer_offsets | <-- "prod-to-dr") | just slightly behind) |
+----------------------+ | group offsets synced periodically |
+-------------------------------------+
No Connect cluster, no translation table: offset N on dr is the record at offset N on prod.
MirrorMaker 2 versus Cluster Linking
| Aspect | MirrorMaker 2 | Cluster Linking |
|---|---|---|
| Availability | Apache Kafka, open source | Confluent Platform (commercial) and Confluent Cloud |
| Runs in | Separate Connect workers | The destination brokers |
| Offsets on target | Different, need translation | Identical |
| Target topic | Normal writable topic (name per replication policy) | Read-only mirror topic until promoted |
| Consumer offset sync | Translated checkpoints | Copied directly (optionally filtered by group) |
| Data format | Records are re-produced (re-batched, possibly re-compressed) | Batches copied as is |
| Source cluster requirements | Any Kafka-compatible cluster | Source must support the link (recent Apache Kafka or Confluent) |
| Typical use | Open-source DR, active-active with prefixes, migrations | DR and migration within the Confluent ecosystem |
Other managed or commercial replicators exist, such as Confluent Replicator (Connect-based, older) and Amazon MSK Replicator. Evaluate them with the same questions: do offsets change, how are consumer offsets carried over, and what happens to topic configurations and ACLs.
Example: creating a link and mirror topics
# link.properties: how the destination reaches the source
bootstrap.servers=prod-b1:9092,prod-b2:9092,prod-b3:9092
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="cluster-link" password="<secret>";
# Copy committed offsets of matching consumer groups every 30 s
consumer.offset.sync.enable=true
consumer.offset.sync.ms=30000
consumer.offset.group.filters={"groupFilters":[{"name":"*","patternType":"LITERAL","filterType":"INCLUDE"}]}
# Create mirror topics automatically for source topics with this prefix
auto.create.mirror.topics.enable=true
auto.create.mirror.topics.filters={"topicFilters":[{"name":"orders.","patternType":"PREFIXED","filterType":"INCLUDE"}]}
# All commands run against the DESTINATION cluster
kafka-cluster-links --bootstrap-server dr-b1:9092 --command-config admin.properties \
--create --link prod-to-dr --config-file link.properties
# Create a mirror topic explicitly (same name, same partitions, same offsets as the source)
kafka-mirrors --bootstrap-server dr-b1:9092 --command-config admin.properties \
--create --mirror-topic payments --link prod-to-dr
# Inspect state and lag per partition
kafka-mirrors --bootstrap-server dr-b1:9092 --command-config admin.properties \
--describe --topics payments
# Topic: payments LinkName: prod-to-dr State: ACTIVE ...
# Partition: 0 MaxLag: 12 ...
# Planned switchover: wait until lag is zero, then make the topic writable
kafka-mirrors --bootstrap-server dr-b1:9092 --command-config admin.properties \
--promote --topics payments
# Disaster: stop mirroring immediately and make the topic writable, accepting any lag as lost
kafka-mirrors --bootstrap-server dr-b1:9092 --command-config admin.properties \
--failover --topics payments
(Command names and options follow Confluent Platform. Confluent Cloud exposes the same operations through the confluent kafka link and confluent kafka mirror CLI commands.)
Key Takeaways
- Cluster Linking replicates broker-to-broker with the internal replication protocol, so mirror topics keep the source's offsets and no translation is needed.
- Mirror topics are read-only until promoted (
--promotewaits for zero lag,--failoverdoes not). - It removes the Connect layer but is a Confluent feature, not Apache Kafka.
- Like MM2, it is asynchronous: lag at the moment of disaster is the data you lose.
Practice
- (Beginner) Explain why a consumer that committed offset 5,000 on the source cluster can start at offset 5,000 on a Cluster Linking mirror topic, but not on an MM2 remote topic.
- (Intermediate) Build a comparison for your organization's DR plan: list the steps needed to fail over one consumer group with MM2 and with Cluster Linking, and mark which steps can produce duplicates or data loss.
- Interview question: "When would you choose MirrorMaker 2 over a broker-native replication feature even if both are available?" Hint: consider licensing, heterogeneous source clusters, topic renaming for active-active, and who operates the replication layer.
Active-Passive Topologies
Theory
The simplest multi-cluster design for disaster recovery has one active (primary) cluster that serves all producers and consumers, and one passive (standby) cluster that only receives replicated data. If the primary region fails, applications switch to the standby. Replication runs in one direction only.
Analogy: a theater with an understudy. The understudy learns every line during rehearsals (replication) but only goes on stage when the lead cannot perform (failover).
Normal operation After failover
apps ---produce/consume---> [ primary: us-east ] [ primary: us-east ] (down)
|
| MM2 / Cluster Linking (one way)
v
[ standby: us-west ] apps ---produce/consume---> [ us-west ]
(no client traffic, or
read-only workloads)
Standby temperature
| Standby type | What runs in the DR region | Recovery time | Cost |
|---|---|---|---|
| Cold | Nothing; rebuild a cluster and restore from backups (Section 14.2) | Hours | Lowest |
| Warm | DR cluster receiving replication; applications deployed but scaled to zero | Minutes | Medium |
| Hot | DR cluster and applications running, ready to take traffic | Seconds to minutes | Highest |
The standby cluster must be sized for full production load, because after failover it is production. Many teams make it earn its keep by running read-only workloads there (analytics, reporting, backfills), which also proves continuously that the replicated data is usable.
What must be replicated besides records
| Item | How |
|---|---|
| Topic data and configs | MM2 source connector or Cluster Linking |
| Consumer group offsets | MM2 checkpoints with sync.group.offsets.enabled, or Cluster Linking offset sync |
| ACLs, users, SCRAM credentials, quotas | Manage as code and apply to both clusters (Section 11.3) |
| Schemas | One Schema Registry shared by both regions, or schema replication. Records carry schema IDs, so the IDs must be identical on both sides |
| Connectors | Same configurations deployed (paused or running) in the DR Connect cluster |
Example: one-way MM2 for active-passive
Using IdentityReplicationPolicy keeps topic names identical, so applications only change their bootstrap address after failover:
clusters = primary, dr
primary.bootstrap.servers = kafka-use1.example.com:9092
dr.bootstrap.servers = kafka-usw2.example.com:9092
primary->dr.enabled = true
primary->dr.topics = .*
primary->dr.groups = .*
# Explicitly disable the reverse direction
dr->primary.enabled = false
# Same topic names on both sides (safe only because replication is one-way)
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
sync.topic.configs.enabled = true
emit.checkpoints.enabled = true
sync.group.offsets.enabled = true
sync.group.offsets.interval.seconds = 10
Applications reach the cluster through a stable name that operations can re-point during failover:
# application.properties (both producers and consumers)
# kafka.example.com is a DNS CNAME to kafka-use1 today, to kafka-usw2 after failover
bootstrap.servers=kafka.example.com:9092
# Try every IP behind the name, not only the first one
client.dns.lookup=use_all_dns_ips
Key Takeaways
- Active-passive means one cluster serves traffic and the other receives one-way replication.
- The standby must be sized for full load. Running read-only workloads on it keeps it honest.
- Records are not enough: plan replication of offsets, schemas (with identical IDs), ACLs, users, and connectors.
IdentityReplicationPolicysimplifies failover for one-way flows, because topic names do not change.
Practice
- (Beginner) List everything a payment service needs, besides topic data, to run correctly against the standby cluster after failover.
- (Intermediate) Design a warm-standby setup for a cluster handling 80 MB/s at peak with 3 days of retention. Specify the standby's broker count, the MM2 deployment, how schemas are shared, and how clients are redirected.
- Interview question: "Your DR cluster has been receiving replication for a year but has never served traffic. How confident are you that failover will work, and how would you increase that confidence?" Hint: think about what replication does not test (ACLs, schemas, client configs, capacity, runbooks) and about regular drills.
Active-Active Topologies
Theory
In an active-active topology, every cluster accepts writes from local applications, and replication runs in both directions. Users in each region get low latency, no capacity sits idle, and losing a region means redirecting that region's traffic instead of promoting a standby. The price is complexity: the same logical stream now has writers in several places, and there is no single order across them.
Analogy: two branches of a bank both open for business. Each records local deposits and sends a copy of its ledger to the other branch every few seconds. Each branch sees all transactions eventually, but not always in the same order, and if a customer withdraws in both branches at the same moment, someone has to reconcile.
Cluster "east" Cluster "west"
+---------------------------+ MM2 east->west +---------------------------+
| orders (local) ----+-------------------------->| east.orders (remote) |
| west.orders (remote) <--+---------------------------+ orders (local) |
+---------------------------+ MM2 west->east +---------------------------+
^ ^ ^ ^
east producers east consumers read west producers west consumers read
write "orders" "orders" + "west.orders" write "orders" "orders" + "east.orders"
With DefaultReplicationPolicy, each cluster has its local orders plus one remote topic per other cluster. MM2 never sends west.orders back to west, so records cannot loop. Consumers that need the global view subscribe to all of them with a pattern.
Example: MM2 in both directions and an aggregating consumer
clusters = east, west
east.bootstrap.servers = east-b1:9092
west.bootstrap.servers = west-b1:9092
east->west.enabled = true
east->west.topics = orders
west->east.enabled = true
west->east.topics = orders
# The default policy (prefixing) is mandatory here: identity naming would loop records forever
replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "east-b1:9092"); // always the local cluster
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-analytics");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
// Matches "orders" (local) and "<alias>.orders" (remote copies from any other region)
consumer.subscribe(Pattern.compile("^([a-z0-9-]+\\.)?orders$"));
while (true) {
for (ConsumerRecord<String, String> rec : consumer.poll(Duration.ofMillis(500))) {
String origin = rec.topic().contains(".")
? rec.topic().substring(0, rec.topic().indexOf('.')) // "west.orders" -> "west"
: "east"; // local topic
process(origin, rec);
}
}
}
Conflicts and ordering
Per-key ordering holds inside each topic, but orders and west.orders are separate topics. If customer 42 is updated in both regions within the replication lag, consumers in each region may see the two updates in different orders. Design so that this does not matter:
| Strategy | How it works | Example |
|---|---|---|
| Home region per key | Each key is owned by one region; writes for it are routed there | A user's profile is only modified in the region where the user signed up |
| Commutative events | Events whose order does not matter | "Add 5 points" rather than "set points to 105" |
| Version or timestamp wins | Each update carries a version; consumers keep the highest | Last-writer-wins on a updatedAt field or a vector clock |
| CRDTs | Data types that merge deterministically | Counters and sets that converge regardless of order |
| Idempotent processing | Unique event IDs so replays and duplicates are harmless | Deduplicate by eventId in the sink |
Other multi-cluster topologies
| Topology | Shape | Typical use | Watch out for |
|---|---|---|---|
| Hub and spoke (aggregation) | Many regional clusters replicate into one central cluster | Global analytics, data lake ingestion | Hub capacity, prefixes keep sources apart |
| Fan-out | One cluster replicates to many | Distributing reference data to regions | Egress cost multiplies per target |
| Full mesh | Every cluster replicates to every other | Global active-active | n x (n - 1) flows; latency and cost grow fast |
| Chain | A to B to C | Edge to regional to central | Lag adds up per hop; names become a.b.topic |
Key Takeaways
- Active-active lets every region write locally, with bidirectional replication and no idle standby.
- Use prefixing (
DefaultReplicationPolicy) so records never loop. Identity naming is not safe in active-active. - Consumers that need the global view subscribe to local and remote topics with a pattern.
- There is no global order across regions. Design with key ownership, commutative or versioned events, and idempotent consumers.
Practice
- (Beginner) Two clusters
euandusreplicate topicclicksboth ways with the default policy. List every topic that exists on each cluster and which ones a global dashboard consumer inusshould subscribe to.- (Intermediate) A shopping-cart service runs active-active in two regions, and the same cart can be modified in both. Design the event schema and consumer logic so both regions converge to the same cart contents regardless of arrival order.
- Interview question: "Why can't you simply use identity topic names in a bidirectional replication setup?" Hint: follow a single record produced in region A through both replication flows.
Offset Translation
Theory
A consumer group's progress is a set of committed offsets on one cluster. After failover, the group must continue on the other cluster, but offsets are not portable between clusters connected by MM2. The record at offset 1,500 on the source may be at offset 1,412 on the target, because:
- the remote topic started later, after the source had already deleted old segments,
- compaction removed different records on each side,
- transaction control markers occupy offsets on the source but are not copied,
- retries in MM2 may have written duplicates, shifting later records.
Without translation, a failed-over consumer has two bad options: auto.offset.reset=earliest (reprocess days of data) or latest (skip everything that was in flight).
Analogy: two editions of the same book with different page numbers. Your bookmark says "page 150" in the first edition. To continue in the second edition, you need a concordance table that maps page numbers between them.
How MM2 translates offsets
- While copying,
MirrorSourceConnectorperiodically writes offset syncs (pairs ofupstreamOffset -> downstreamOffsetper partition) tomm2-offset-syncs.<target>.internal. A new sync is emitted when the gap between the two offsets drifts by more thanoffset.lag.max(default 100), and periodically otherwise. MirrorCheckpointConnectorreads the committed offsets of each group on the source and translates them using the most recent sync at or before the committed offset.- It writes checkpoints to
<source>.checkpoints.internalon the target, and withsync.group.offsets.enabled=trueit also commits the translated offsets for the same group on the target, provided that group has no active members there.
The translation is deliberately conservative: it never jumps past a record the consumer has not processed, so the error is always on the side of duplicates.
Offset syncs for orders-0 (upstream -> downstream): 1000 -> 980 1600 -> 1575
Group "billing" committed upstream offset 1500.
Most recent sync at or before 1500: 1000 -> 980
Translated downstream offset: 981 (the sync itself + 1, since records in between are unknown)
Downstream records 981 .. ~1480 were already processed upstream -> reprocessed after failover
The duplicate window shrinks as syncs become more frequent, at the cost of more writes to the offset-syncs topic. Since Kafka 3.5 the checkpoint connector keeps several syncs per partition in memory so that translation stays correct after compaction and restarts, which also means translated offsets can lag a little behind the group's true position.
Example: simulating the translation rule
from bisect import bisect_right
def translate(syncs, upstream_committed):
"""syncs: list of (upstream, downstream) sorted by upstream. Returns downstream offset or None."""
ups = [u for u, _ in syncs]
i = bisect_right(ups, upstream_committed) - 1 # most recent sync at or before the commit
if i < 0:
return None # no sync yet: cannot translate safely
up, down = syncs[i]
return down if up == upstream_committed else down + 1
syncs = [(1000, 980), (1600, 1575), (2400, 2371)]
for committed in (999, 1000, 1500, 1600, 2500):
print(committed, "->", translate(syncs, committed))
# 999 -> None
# 1000 -> 980
# 1500 -> 981 (up to ~500 records will be reprocessed)
# 1600 -> 1575
# 2500 -> 2372
Example: applying translated offsets during failover (Java)
If automatic group offset sync is disabled, or you want to control the moment of switching, translate and commit offsets yourself with RemoteClusterUtils from the connect-mirror-client library:
// Connection properties for the TARGET cluster, where the checkpoints topic lives
Map<String, Object> targetProps = Map.of("bootstrap.servers", "west-b1:9092");
// Read the latest checkpoints written for group "billing" replicated from alias "east"
Map<TopicPartition, OffsetAndMetadata> translated =
RemoteClusterUtils.translateOffsets(targetProps, "east", "billing", Duration.ofSeconds(30));
try (Admin admin = Admin.create(targetProps)) {
// The group must have no active members on the target, or the commit is rejected
admin.alterConsumerGroupOffsets("billing", translated).all().get();
}
translated.forEach((tp, om) -> System.out.printf("%s -> %d%n", tp, om.offset()));
// east.orders-0 -> 981 (with DefaultReplicationPolicy the partitions belong to the remote topic)
Fallback: timestamp-based positioning
When checkpoints are missing or stale, position the group by time. Record timestamps are preserved by replication, so "the failure happened at 14:05" maps to the same records on both clusters:
# Rewind slightly before the failure time and accept some duplicates
bin/kafka-consumer-groups.sh --bootstrap-server west-b1:9092 --group billing \
--topic orders --reset-offsets --to-datetime 2026-10-23T14:00:00.000 --execute
With Cluster Linking, none of this is needed: offsets are identical, and committed offsets are copied as they are.
Key Takeaways
- MM2 assigns new offsets on the target, so committed offsets must be translated before consumers fail over.
- Offset syncs map upstream to downstream offsets; checkpoints are translated group offsets written on the target.
- Translation is conservative: it can cause reprocessing but never skips records. Consumers must be idempotent.
sync.group.offsets.enabledcommits translated offsets on the target only for groups without active members there.- Timestamp-based reset is a practical fallback when checkpoints are unavailable.
Practice
- (Beginner) Using the Python
translatefunction, add syncs(3000, 2950)and compute translations for committed offsets 2999, 3000, and 3001. Explain the result for 2999.- (Intermediate) In a two-cluster MM2 setup, run a consumer group on the source, stop it, and compare its committed offsets on the source with the checkpoints on the target (
kafka-consumer-groups.sh --describeon both). Loweroffset.lag.maxand repeat. How does the gap change?- Interview question: "After a regional failover, your consumers processed about 30,000 records twice. Was that a bug?" Hint: consider how offset syncs and checkpoints are produced, their intervals, and what the alternative to reprocessing would be.
<a id="142-high-availability-and-disaster-recovery"></a>
14.2 High Availability and Disaster Recovery
This section covers how to keep Kafka available when a whole data center or region fails: synchronous stretch clusters, the mechanics of failing over and back, how to set recovery objectives, and why replication alone is not a backup.
Stretch Clusters
Theory
Asynchronous replication between clusters always loses whatever was in flight. If that is unacceptable, the alternative is a single stretch cluster whose brokers and controllers are spread across several data centers (or regions), using Kafka's normal synchronous replication. With acks=all and replicas in every site, an acknowledged write already exists in more than one site, so losing a site loses no acknowledged data, and leader election moves partitions to surviving sites automatically. There is one cluster, one set of offsets, and nothing to translate.
Analogy: instead of a courier carrying copies to a branch office (asynchronous), every entry is written simultaneously into ledgers in two buildings connected by a short corridor, and the customer only leaves once both buildings have signed. This works only if the corridor is short.
The corridor is the catch. Every acknowledged write waits for at least one cross-site replica, so inter-site latency is added to produce latency, and every replicated byte crosses the inter-site link. Stretch clusters suit sites with a round-trip time in the single-digit milliseconds (availability zones, or data centers in the same metro area). Across continents, use separate clusters with asynchronous replication instead.
The quorum problem: why three sites
KRaft controllers (Section 2.3) need a majority of voters to elect a leader and change metadata. Partition data needs min.insync.replicas replicas to accept writes. Both rules punish two-site designs:
Two sites, 3 controllers: Three sites, 3 controllers:
DC1: ctrl1, ctrl2 DC2: ctrl3 DC1: ctrl1 DC2: ctrl2 DC3: ctrl3
Lose DC1 -> 1 of 3 left -> NO QUORUM Lose any DC -> 2 of 3 left -> quorum OK
Lose DC2 -> 2 of 3 left -> quorum OK
=> the cluster survives only the "right" DC failing
Common layouts:
| Layout | Controllers | Data replicas | Survives one site loss? | Notes |
|---|---|---|---|---|
| 3 sites (AZs) | 1 per site | RF=3, one per site, min.insync.replicas=2 | Yes, reads and writes continue | Standard multi-AZ cloud deployment |
| 2.5 sites | 1 per main site + 1 in a small third "tiebreaker" site | RF=4, two per main site, min.insync.replicas=3 | Data is safe; writes stop until min.insync.replicas is lowered to 2 | For on-premises with only two real data centers |
| 2 sites | Majority in one site | Any | Only if the minority site fails | Avoid |
In the 2.5-site layout, min.insync.replicas=3 with two replicas per site guarantees that every acknowledged write exists in both main sites. After losing one site only 2 replicas remain, so producers with acks=all get NotEnoughReplicas until an operator lowers min.insync.replicas (a deliberate trade of safety for availability). Without the third-site controller, the surviving site would not even have a controller quorum.
Example: a three-site stretch cluster
# broker.properties on broker 101 (DC1); brokers in DC2 and DC3 differ in node.id and broker.rack
node.id=101
process.roles=broker
broker.rack=dc1
controller.quorum.bootstrap.servers=ctrl-dc1:9093,ctrl-dc2:9093,ctrl-dc3:9093
# Safety defaults for every new topic
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
# Never elect an out-of-sync replica, even if a whole site is lost
unclean.leader.election.enable=false
# Let consumers read from a replica in their own DC (Section 2.2)
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
# Rack-aware assignment places one replica of every partition in each DC
bin/kafka-topics.sh --bootstrap-server broker101:9092 --create --topic payments \
--partitions 12 --replication-factor 3
bin/kafka-topics.sh --bootstrap-server broker101:9092 --describe --topic payments | head -3
# Topic: payments Partition: 0 Leader: 101 Replicas: 101,201,301 Isr: 101,201,301
# Topic: payments Partition: 1 Leader: 201 Replicas: 201,301,102 Isr: 201,301,102
# (1xx = dc1, 2xx = dc2, 3xx = dc3)
Producers should set acks=all (the default since Kafka 3.0) and expect produce latency of roughly one inter-site round trip plus local processing. Consumers set client.rack to read locally.
Vendor extensions
Confluent Platform's Multi-Region Clusters add observers: replicas that replicate asynchronously and are not part of the ISR, so a remote region can hold a copy without slowing writes. Replica placement is declared per topic and observers can be promoted automatically when the ISR drops too low:
{
"version": 2,
"replicas": [{"count": 2, "constraints": {"rack": "us-east"}}],
"observers": [{"count": 2, "constraints": {"rack": "us-west"}}],
"observerPromotionPolicy": "under-min-isr"
}
This file is passed as the confluent.placement.constraints topic config. It is not available in Apache Kafka.
Key Takeaways
- A stretch cluster is one Kafka cluster across sites, using synchronous replication: RPO of zero for acknowledged writes and automatic failover.
- Inter-site latency is added to every
acks=allwrite, so stretch clusters need low-latency links. - Controller quorums and
min.insync.replicasboth need a majority. Use three sites, or two sites plus a tiebreaker controller site. - Set
broker.rackper site, RF andmin.insync.replicasso every acknowledged write spans sites, and keep unclean leader election off.
Practice
- (Beginner) A cluster has 6 brokers in 3 AZs, RF=3, and
min.insync.replicas=2. Describe what happens to reads, writes, and the controller quorum when one AZ fails, and when two AZs fail.- (Intermediate) Your company has two on-premises data centers 15 km apart and a small office with a server room. Design a stretch cluster: placement of controllers and brokers, RF,
min.insync.replicas, and the manual steps to take when one data center fails.- Interview question: "Why not stretch one Kafka cluster between Frankfurt and Virginia to get zero data loss?" Hint: compute what an 85 ms round trip does to every
acks=allproduce, and consider ISR shrinking, controller elections, and network cost.
Failover and Failback
Theory
With separate clusters, surviving a regional failure requires failover: moving producers and consumers from the failed cluster to the surviving one. Later, when the original region is repaired, failback returns traffic to it (or the roles are permanently swapped). Kafka does not do either automatically across clusters, and for good reasons: deciding that a region is truly down (and not just briefly unreachable) is hard, and two clusters both accepting writes for the same topic after a false alarm creates a split brain that is painful to repair. Most organizations therefore make the decision manually and automate the execution.
Analogy: a fire drill. Deciding to evacuate is a human call made with incomplete information. Once the call is made, everyone follows a rehearsed route without improvising.
Planned versus unplanned failover
| Planned (switchover) | Unplanned (disaster) | |
|---|---|---|
| Primary reachable | Yes | No |
| Can drain replication lag | Yes: stop producers, wait for lag 0 | No: lagging records are lost or stranded |
| Data loss | None | Up to the replication lag (RPO) |
| Duplicates | Possible (offset translation) | Possible |
| Typical reason | Region maintenance, migration, DR drill | Region outage |
Failover runbook
1. DECIDE Confirm the primary is unavailable (or start a planned switchover). Record the time.
2. FENCE Stop producers writing to the primary if any can still reach it (avoid split brain).
3. DRAIN Planned only: wait until replication lag is zero for all topics.
4. CUT Stop replication (stop MM2 flows / promote or fail over mirror topics).
5. OFFSETS Make sure translated group offsets are committed on the DR cluster.
6. REDIRECT Point clients at the DR cluster (DNS, config change, service discovery).
7. CONSUMERS Start consumers first, so they are ready to read.
8. PRODUCERS Start producers.
9. VERIFY Check lag, error rates, and business metrics. Announce the new primary.
Example: a planned switchover with MM2 (one-way, identity naming)
#!/usr/bin/env bash
set -euo pipefail
PRIMARY=kafka-use1.example.com:9092
DR=kafka-usw2.example.com:9092
# 2. FENCE: scale producers to zero (here: Kubernetes deployments labeled role=producer)
kubectl scale deployment -l role=producer --replicas=0 -n apps
# 3. DRAIN: wait until MM2 has nothing left to copy. Here we compare the newest record timestamp
# on both sides for each topic; production runbooks often use the MM2 replication metrics instead.
for TOPIC in orders payments shipments; do
until python3 compare_tail.py --source $PRIMARY --target $DR --topic $TOPIC; do
echo "waiting for $TOPIC to drain"; sleep 5
done
done
# 4. CUT: stop MM2 (the dedicated MM2 deployment in the DR region)
kubectl scale deployment mm2-primary-to-dr --replicas=0 -n kafka
# 5. OFFSETS: stop consumers on the primary, then check that the synced offsets on DR look right
kubectl scale deployment -l role=consumer --replicas=0 -n apps
bin/kafka-consumer-groups.sh --bootstrap-server $DR --describe --group billing
# 6. REDIRECT: re-point the stable name used by every client
./update-dns.sh kafka.example.com CNAME kafka-usw2.example.com
# 7-8. Start consumers, then producers (they read bootstrap.servers=kafka.example.com:9092)
kubectl scale deployment -l role=consumer --replicas=3 -n apps
kubectl scale deployment -l role=producer --replicas=3 -n apps
Redirecting clients
| Method | How | Caveats |
|---|---|---|
| DNS name for bootstrap | Re-point a CNAME | Clients resolve bootstrap only when (re)bootstrapping; long-running clients may need a restart. Keep TTLs short |
| Client rebootstrap | metadata.recovery.strategy=rebootstrap (KIP-899, Kafka 3.8+) makes clients return to bootstrap.servers when no known broker is reachable | Combine with the DNS approach; verify your client library supports it |
| Configuration change | Config service, Kubernetes ConfigMap, or feature flag plus rolling restart | Most explicit and predictable |
| Service mesh / proxy | Clients talk to a proxy that routes to the active cluster | Extra hop; the proxy must understand Kafka's per-broker addressing |
Things that change with the cluster: producer IDs (idempotent producers get new ones), transactional producers must call initTransactions() again, and cached metadata becomes invalid. Restarting applications after redirecting is the safest option.
Failback
When the old primary returns, its data has diverged from the new primary:
Old primary (us-east) records 0 .. 10,000 | 10,001 .. 10,250 never replicated ("stranded")
New primary (us-west) records 0 .. 10,000 | new writes since failover, offsets 10,001 .. 95,000
| Option | Steps | Trade-off |
|---|---|---|
| Rebuild old primary | Delete topics (or the cluster) on us-east, replicate us-west -> us-east until caught up, then do a planned switchover back | Simple and consistent; stranded records are lost unless exported first |
| Reconcile stranded records | Export records after the last replicated position from us-east, have the owning teams replay or discard them, then rebuild | Recovers data; requires per-topic decisions and idempotent consumers |
| Swap roles permanently | Leave us-west as primary, rebuild us-east as the new standby | No second switchover; DR region must be equally good |
With MM2, failback means enabling the reverse flow (and, with identity naming, ensuring the old flow is off so records cannot loop). With Cluster Linking, you create a link in the opposite direction and mirror the now-writable topics back; recent Confluent versions provide commands that automate reversing a link.
Key Takeaways
- Cross-cluster failover is a decision (usually human) followed by a rehearsed, automated runbook.
- Fence old producers before redirecting clients to avoid split brain; start consumers before producers.
- Planned switchovers drain replication lag first and lose nothing. Unplanned failovers lose up to the lag.
- Failback must deal with diverged data: rebuild the old primary from the new one, reconcile stranded records, or swap roles permanently.
Practice
- (Beginner) Explain why consumers are started before producers in step 7-8 of the runbook, and what could go wrong in the opposite order.
- (Intermediate) Write a
compare_tail.pyscript (used in the runbook) that, for each partition, reads the last record on the source and target topics and exits 0 only if the target's newest timestamp is at least the source's newest timestamp.- Interview question: "Why do most companies not automate cross-region Kafka failover end to end?" Hint: consider network partitions, false positives, split brain, and the cost of failing back.
RPO and RTO Planning
Theory
Disaster recovery designs are judged by two numbers agreed with the business:
- Recovery Point Objective (RPO): how much data, measured in time, you can afford to lose. An RPO of 30 seconds means losing the last 30 seconds of writes is acceptable.
- Recovery Time Objective (RTO): how long the service can be unavailable. An RTO of 15 minutes means traffic must be flowing again within 15 minutes.
Analogy: if your laptop dies, RPO is how long ago your last backup was (the work you must redo), and RTO is how long it takes to get a working laptop and restore the backup.
Lower targets cost more: synchronous replication across sites, idle standby capacity, more automation, and more drills. Start from the business impact of losing data or being down for each data flow; payments and ledgers usually justify more than clickstream analytics.
Architectures and what they deliver
| Architecture | Typical RPO | Typical RTO | Cost and complexity |
|---|---|---|---|
| Single cluster across 3 AZs | 0 for an AZ loss; no protection for region loss | Seconds (automatic leader election) | Low |
| Stretch cluster across nearby sites | 0 for a site loss | Seconds to minutes | Medium; latency cost on writes |
| Active-passive, async replication | Replication lag (seconds to minutes) | Minutes (warm/hot standby) | Medium to high |
| Active-active, async replication | Replication lag for the failed region's unreplicated writes | Near zero for redirecting traffic | High; application design constraints |
| Backup and restore | Time since last backup (minutes to hours) | Hours | Low |
What determines each number
RPO ~= replication lag at the moment of failure (worst case during peaks or after a network blip)
bytes at risk = throughput x lag e.g. 50 MB/s x 20 s = 1 GB
RTO = detection + decision + execution + client recovery + catch-up
(alerts) (on-call) (runbook) (restarts, (consumers working
rebalances) through backlog)
Teams often plan the execution step and forget the others. A 3-minute automated runbook is meaningless if paging and the go/no-go decision take 40 minutes.
Example: estimating RPO and RTO from measurements
# Inputs from monitoring (MM2 replication-latency-ms over the last 30 days) and from the last DR drill
lag_p50_s, lag_p99_s, lag_max_s = 2.1, 9.5, 74.0 # replication lag in seconds
peak_mb_s = 60 # peak produce throughput
rto_steps_min = {
"detection (alert fires)": 3,
"decision (on-call + incident lead)": 10,
"execution (runbook)": 6,
"client recovery (restarts)": 4,
"catch-up of consumer backlog": 5,
}
print(f"RPO typical: {lag_p99_s:.0f} s (~{peak_mb_s * lag_p99_s / 1000:.1f} GB at peak)")
print(f"RPO worst observed: {lag_max_s:.0f} s (~{peak_mb_s * lag_max_s / 1000:.1f} GB at peak)")
rto = sum(rto_steps_min.values())
print(f"RTO estimate: {rto} min -> {'meets' if rto <= 30 else 'misses'} a 30 min target")
for step, minutes in sorted(rto_steps_min.items(), key=lambda kv: -kv[1]):
print(f" {minutes:>3} min {step}")
# RPO typical: 10 s (~0.6 GB at peak)
# RPO worst observed: 74 s (~4.4 GB at peak)
# RTO estimate: 28 min -> meets a 30 min target
# 10 min decision (on-call + incident lead)
# 6 min execution (runbook)
# ...
The worst observed lag (74 s) is what an RPO promise must be judged against, not the median.
Example: alerting on RPO
# Prometheus rule; the metric name depends on your JMX exporter rules for kafka.connect.mirror
groups:
- name: kafka-dr
rules:
- alert: MirrorReplicationLagAboveRPO
expr: max by (topic) (kafka_connect_mirror_mirrorsourceconnector_replication_latency_ms_max) > 30000
for: 2m
labels: { severity: page }
annotations:
summary: "Replication of {{ $labels.topic }} is behind by more than the 30 s RPO"
Proving the numbers
An RPO or RTO that has never been tested is a guess. Run regular DR drills: a planned switchover to the standby (and back), measured end to end with the same tooling and people who would handle a real incident. Record the actual detection, decision, execution, and catch-up times, and update the estimate.
Key Takeaways
- RPO is tolerable data loss in time, RTO is tolerable downtime. Both come from business impact, per data flow.
- With async replication, RPO is the replication lag at failure time. Plan against worst-case lag and alert when lag exceeds the RPO.
- RTO includes detection, decision, execution, client recovery, and catch-up, not just the runbook.
- Only DR drills turn RPO and RTO targets into known numbers.
Practice
- (Beginner) A topic receives 25 MB/s and MM2 lag is usually 5 s but reached 90 s during last month's network incident. How much data is at risk in each case, and what RPO can you honestly promise?
- (Intermediate) For an order-processing platform, write an RPO/RTO table with three tiers of topics (for example payments, orders, clickstream), choose an architecture for each tier, and justify the cost.
- Interview question: "The business asks for zero data loss and zero downtime across regions. How do you respond?" Hint: explain what synchronous replication across regions costs, what RPO 0 implies for write latency, and how to negotiate tiers.
Backup Strategies
Theory
Replication protects against hardware and site failure by copying everything, immediately. That is exactly why it is not a backup: if a script deletes a topic, a bug produces corrupted records, or an attacker with admin credentials wipes the cluster, replication faithfully copies the damage to every replica and every DR cluster. A backup is a copy that is separate in time and in control: older versions remain available, and the same mistake or credential cannot destroy both.
Analogy: RAID mirroring is not a backup for your hard disk. If you delete a file, the mirror deletes it too. A backup is the external drive you keep in a drawer.
| Threat | Replicas / stretch cluster | DR cluster (MM2, linking) | Backup |
|---|---|---|---|
| Broker or disk failure | Protected | Protected | Protected |
| Site or region loss | Stretch only | Protected | Protected (slow) |
| Accidental topic deletion | No | Often no (configs and deletions may sync) | Protected |
| Bad data written by a bug | No | No | Protected (restore an earlier point) |
| Compromised admin credentials / ransomware | No | No, if the same credentials reach it | Protected if immutable and in a separate account |
| Data aged out by retention, needed later | No | Only with longer retention | Protected |
What to back up
| Item | Where it lives | How to back it up |
|---|---|---|
| Topic data | Partitions | Sink connector to object storage, or a backup cluster with long retention |
| Topic configurations and partition counts | Cluster metadata | Topics as code (Git), or periodic export |
| ACLs, SCRAM users, quotas | Cluster metadata | Security as code, or periodic export |
| Consumer group offsets | __consumer_offsets | Periodic export (useful mainly for restoring positions by timestamp) |
| Schemas | Schema Registry (_schemas topic) | Back up the topic or export subjects through the REST API |
| Connector configurations | Connect config topic | Configs as code (Section 9.4) |
Approaches for topic data
| Approach | Point-in-time restore | Immutable | Notes |
|---|---|---|---|
| Sink connector to object storage (S3, GCS, Azure Blob) | Yes, by time range | Yes, with object lock and a separate account | Most common; cheap storage, simple retention policies |
| MM2 to a backup cluster with long retention | Limited (only what retention keeps) | Only if access is separated | Faster restore, but it is still a live Kafka cluster |
| Tiered storage (Section 7.3) | No | No | Not a backup: segments are deleted with the topic |
| Disk snapshots of broker volumes | Per broker, not consistent across brokers | Depends | Fragile; restoring a consistent cluster from them is hard |
Example: continuous backup to S3 with a sink connector
Store keys, values, and headers as raw bytes so the backup is independent of serialization formats, and partition files by Kafka partition so per-key order can be restored:
{
"name": "backup-s3-core",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"tasks.max": "8",
"topics.regex": "(orders|payments|shipments)\\..*",
"s3.bucket.name": "acme-kafka-backup-prod",
"s3.region": "eu-west-1",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
"store.kafka.keys": "true",
"store.kafka.headers": "true",
"partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
"flush.size": "50000",
"rotate.schedule.interval.ms": "300000",
"timezone": "UTC"
}
}
ByteArrayConverterkeeps the original bytes;AvroFormatwith a bytes schema writes binary-safe container files.DefaultPartitionerwrites paths liketopics/orders.created/partition=3/orders.created+3+0001250000.avro: the topic, partition, and starting offset are in the file name, which makes ordered restore straightforward.rotate.schedule.interval.mscloses files at least every 5 minutes, so the backup RPO is about 5 minutes even on quiet topics.- Protect the bucket with versioning, object lock (write once, read many), and a separate cloud account whose credentials are not available to Kafka administrators.
Example: restoring a partition range
# pip install confluent-kafka fastavro boto3
import boto3, fastavro
from confluent_kafka import Producer
s3 = boto3.client("s3")
producer = Producer({"bootstrap.servers": "restore-b1:9092", "enable.idempotence": True})
def restore_partition(bucket, topic, partition, target_topic):
prefix = f"topics/{topic}/partition={partition}/"
keys = sorted(o["Key"] for page in s3.get_paginator("list_objects_v2").paginate(
Bucket=bucket, Prefix=prefix) for o in page.get("Contents", []))
for key in keys: # file names sort by starting offset
body = s3.get_object(Bucket=bucket, Key=key)["Body"]
for value in fastavro.reader(body): # each record is the original value bytes
producer.produce(target_topic, value=value, partition=partition)
producer.poll(0)
producer.flush()
restore_partition("acme-kafka-backup-prod", "orders.created", 3, "orders.created.restored")
This sketch restores values only; a full restore also reads the matching key and header files and passes key= and headers=. Restored records get new offsets, so consumers are positioned by timestamp, not by their old offsets. Restoring into a new topic (.restored) first lets you inspect the data before switching consumers.
Example: exporting cluster metadata
#!/usr/bin/env bash
# Nightly metadata export, stored next to the data backup
BS=broker1:9092; OUT=meta-$(date -u +%F); mkdir -p $OUT
CFG="--command-config admin.properties"
bin/kafka-topics.sh --bootstrap-server $BS $CFG --describe > $OUT/topics.txt
bin/kafka-configs.sh --bootstrap-server $BS $CFG --describe --entity-type topics > $OUT/topic-configs.txt
bin/kafka-configs.sh --bootstrap-server $BS $CFG --describe --entity-type users > $OUT/users-quotas.txt
bin/kafka-acls.sh --bootstrap-server $BS $CFG --list > $OUT/acls.txt
bin/kafka-consumer-groups.sh --bootstrap-server $BS $CFG --describe --all-groups > $OUT/groups.txt
curl -s http://schema-registry:8081/schemas > $OUT/schemas.json
Exports are a safety net. The primary source of truth for topics, ACLs, and connectors should be code in version control, from which a cluster can be rebuilt.
Key Takeaways
- Replication is not backup: it copies deletions, corruption, and malicious changes immediately.
- Back up topic data (usually to object storage through a sink connector), and also metadata: topics, ACLs, users, quotas, schemas, connectors.
- Make backups immutable and keep them in a separate account. Tiered storage is not a backup.
- Restored records get new offsets. Plan to reposition consumers by timestamp, and test restores regularly.
Practice
- (Beginner) For each of these incidents, say whether a DR cluster with MM2, a stretch cluster, or an S3 backup would let you recover: a topic deleted by mistake, a producer bug writing invalid JSON for 2 hours, and an AZ outage.
- (Intermediate) Deploy the S3 sink (or a local filesystem equivalent) for one topic, delete the topic, and restore it into a new topic with keys and headers. Verify that per-key order is preserved and measure how long the restore takes per GB.
- Interview question: "We have RF=3, a DR cluster in another region, and tiered storage. Do we still need backups?" Hint: list threats that propagate through replication and tiered storage, and who holds credentials to all of them.
<a id="143-cloud-and-container-deployments"></a>
14.3 Cloud and Container Deployments
This section covers where Kafka runs today beyond hand-managed virtual machines: on Kubernetes (directly or through the Strimzi operator), as a managed cloud service, and in serverless form, together with what each option hides from you and what it still leaves you responsible for.
Kafka on Kubernetes
Theory
Many organizations run all their services on Kubernetes and want Kafka on the same platform: one deployment pipeline, one monitoring stack, one way to manage secrets and configuration. Kubernetes was designed first for stateless workloads, where any replica can replace any other. A Kafka broker is the opposite: it has a fixed identity (node.id), owns specific partitions on its own disk, must be individually addressable by clients, and must not be restarted carelessly while its partitions are under-replicated.
Analogy: Kubernetes treats ordinary pods like cattle: identical and replaceable. Kafka brokers are more like named employees with their own filing cabinets: you can replace one, but the replacement must get the same desk, the same cabinet, and the same name badge.
Mapping Kafka's needs to Kubernetes primitives
| Kafka needs | Kubernetes primitive |
|---|---|
Stable identity per broker (node.id, hostname) | StatefulSet (pods kafka-0, kafka-1, ...) or an operator-managed equivalent such as Strimzi's StrimziPodSet |
| Own disk that survives restarts and rescheduling | PersistentVolumeClaim per pod (from volumeClaimTemplates) |
| Clients reach each broker individually | Headless Service (per-pod DNS) inside the cluster; one LoadBalancer, NodePort, or Ingress route per broker outside |
| Replicas spread across failure domains | Topology spread constraints or pod anti-affinity on topology.kubernetes.io/zone, and broker.rack set from the node's zone |
| Never take down too many brokers at once | PodDisruptionBudget (maxUnavailable: 1), plus checks that partitions are in sync before each restart |
| Configuration and credentials | ConfigMaps and Secrets |
Kubernetes cluster (3 zones)
+--------------------------------------------------------------------------------+
| zone a zone b zone c |
| +--------------+ +--------------+ +--------------+ |
| | pod kafka-0 | | pod kafka-1 | | pod kafka-2 | |
| | node.id=0 | | node.id=1 | | node.id=2 | |
| | rack=zone-a | | rack=zone-b | | rack=zone-c | |
| +------+-------+ +------+-------+ +------+-------+ |
| | PVC data-kafka-0 | PVC data-kafka-1 | PVC ... |
| |
| headless Service "kafka-headless": kafka-0.kafka-headless.kafka.svc ... |
| per-broker LoadBalancers for external clients: b0.kafka.example.com ... |
+--------------------------------------------------------------------------------+
Example: a minimal StatefulSet (for understanding, not production)
Seeing the raw pieces explains what operators automate. This sketch runs three combined broker/controller nodes with the apache/kafka image and derives each node's identity from its pod name:
apiVersion: v1
kind: Service
metadata:
name: kafka-headless
namespace: kafka
spec:
clusterIP: None # headless: DNS returns individual pod addresses
selector: { app: kafka }
ports:
- { name: client, port: 9092 }
- { name: controller, port: 9093 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kafka
namespace: kafka
spec:
serviceName: kafka-headless # gives pods stable names: kafka-0.kafka-headless.kafka.svc
replicas: 3
podManagementPolicy: Parallel # controllers must start together to form a quorum
selector:
matchLabels: { app: kafka }
template:
metadata:
labels: { app: kafka }
spec:
topologySpreadConstraints: # one pod per zone
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels: { app: kafka }
containers:
- name: kafka
image: apache/kafka:4.1.0
command: ["sh", "-c"]
args:
- |
export KAFKA_NODE_ID=${HOSTNAME##*-} # kafka-2 -> 2
export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${HOSTNAME}.kafka-headless.kafka.svc:9092
exec /etc/kafka/docker/run # the image's normal start script
env:
- { name: CLUSTER_ID, value: "5L6g3nShT-eMCtK--X86sw" }
- { name: KAFKA_PROCESS_ROLES, value: "broker,controller" }
- { name: KAFKA_CONTROLLER_QUORUM_VOTERS,
value: "0@kafka-0.kafka-headless.kafka.svc:9093,1@kafka-1.kafka-headless.kafka.svc:9093,2@kafka-2.kafka-headless.kafka.svc:9093" }
- { name: KAFKA_LISTENERS, value: "PLAINTEXT://:9092,CONTROLLER://:9093" }
- { name: KAFKA_CONTROLLER_LISTENER_NAMES, value: "CONTROLLER" }
- { name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, value: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" }
- { name: KAFKA_LOG_DIRS, value: "/var/lib/kafka/data" }
- { name: KAFKA_HEAP_OPTS, value: "-Xms4g -Xmx4g" }
resources:
requests: { cpu: "2", memory: 12Gi } # 4 GiB heap + room for the page cache
limits: { memory: 12Gi } # no CPU limit: throttling hurts latency
volumeMounts:
- { name: data, mountPath: /var/lib/kafka/data }
volumeClaimTemplates:
- metadata: { name: data }
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd-wait # volumeBindingMode: WaitForFirstConsumer
resources: { requests: { storage: 500Gi } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: kafka-pdb
namespace: kafka
spec:
maxUnavailable: 1 # node drains may evict at most one broker at a time
selector:
matchLabels: { app: kafka }
What this sketch does not handle is the hard part: TLS and authentication, external access, broker.rack from the node's zone, rolling restarts that wait for the ISR to recover between pods, scaling with partition reassignment, upgrades, and certificate rotation. A PodDisruptionBudget only counts pods; it does not know whether partitions are under-replicated. That gap is why operators exist.
Kubernetes-specific pitfalls
| Pitfall | Effect | Mitigation |
|---|---|---|
| Storage class binds volumes before scheduling | Volume created in a zone where the pod cannot run | volumeBindingMode: WaitForFirstConsumer |
| Zonal volumes (EBS, PD) | A broker pod can only run in its volume's zone | Keep spare node capacity in every zone |
| Memory limit equal to heap | No room for page cache, poor read performance, OOM kills | Request heap plus several GB; Kafka relies on the page cache (Section 7.1) |
| CPU limits | CFS throttling causes latency spikes | Use requests without CPU limits, or generous limits |
| Clients outside the cluster | Advertised internal DNS names unreachable (Section 2.2) | Separate external listener with per-broker addresses |
| Node drains and cluster upgrades | Several brokers evicted in sequence before ISRs recover | PodDisruptionBudget plus ISR-aware tooling (operators, Strimzi Drain Cleaner) |
| Noisy neighbors | Disk and network contention from co-located pods | Dedicated node pools with taints and tolerations for Kafka |
Key Takeaways
- Kafka on Kubernetes needs stable identity, per-broker storage, per-broker addressing, zone spreading, and disruption control.
- StatefulSets, headless Services, PVCs, topology spread constraints, and PodDisruptionBudgets provide the building blocks, but not Kafka-aware operations.
- Size memory for heap plus page cache, avoid CPU throttling, and use zone-aware storage binding.
- For production, use an operator rather than hand-written StatefulSets.
Practice
- (Beginner) Deploy the StatefulSet above on a local cluster (kind, minikube, or k3d; adjust the storage class), create a topic with RF 3, delete pod
kafka-1, and observe that it comes back with the same name, volume, and node ID.- (Intermediate) Add an external listener so a client outside the Kubernetes cluster can produce. Explain why a single LoadBalancer Service in front of all three pods is not enough.
- Interview question: "A Kubernetes node upgrade drained three nodes in ten minutes and your Kafka cluster went offline even though you had a PodDisruptionBudget. What happened?" Hint: consider what the PDB counts, how long a restarted broker takes to rejoin the ISR, and
min.insync.replicas.
Strimzi Operator
Theory
A Kubernetes operator encodes operational knowledge in software: you declare the desired state in custom resources, and a controller continuously reconciles the real state toward it. Strimzi (a CNCF project) is the most widely used open-source Kafka operator. You write YAML describing a Kafka cluster, topics, users, Connect clusters, or MirrorMaker 2 flows, and Strimzi creates pods, services, certificates, and configuration, then keeps them correct through restarts, upgrades, and scaling.
Analogy: instead of giving a building manager a list of chores ("restart the boiler, then check the pressure"), you give them the target ("keep the building at 21 degrees"), and they decide what to do, in what order, and how carefully.
Main components and custom resources
| Component / resource | Purpose |
|---|---|
| Cluster Operator | Watches Kafka, KafkaNodePool, KafkaConnect, KafkaMirrorMaker2, KafkaBridge, KafkaRebalance and manages their pods |
Kafka | Cluster-wide settings: version, listeners, authorization, broker config, Cruise Control, Entity Operator |
KafkaNodePool | A group of nodes with the same roles (controller, broker, or both), replica count, storage, and resources |
| Entity Operator: Topic Operator | Reconciles KafkaTopic resources with real topics |
| Entity Operator: User Operator | Reconciles KafkaUser resources into credentials (Secrets) and ACLs |
KafkaConnect / KafkaConnector | Connect clusters and their connectors (Section 9.4) |
KafkaMirrorMaker2 | MM2 deployments (Section 14.1) |
KafkaRebalance | Cruise Control rebalancing proposals and approvals (Section 12.2) |
Recent Strimzi releases support only KRaft, with node pools used to separate controllers from brokers.
Example: install the operator and declare a cluster
kubectl create namespace kafka
# Installs CRDs, RBAC, and the Cluster Operator watching the "kafka" namespace
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
apiVersion: kafka.strimzi.io/v1beta2 # check which API version your Strimzi release serves
kind: KafkaNodePool
metadata:
name: controllers
namespace: kafka
labels:
strimzi.io/cluster: prod # links this pool to the Kafka resource "prod"
spec:
replicas: 3
roles: [controller]
storage:
type: jbod
volumes:
- { id: 0, type: persistent-claim, size: 20Gi, kraftMetadata: shared, deleteClaim: false }
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
name: brokers
namespace: kafka
labels:
strimzi.io/cluster: prod
spec:
replicas: 3
roles: [broker]
resources:
requests: { cpu: "4", memory: 16Gi }
limits: { memory: 16Gi }
jvmOptions: { "-Xms": "6g", "-Xmx": "6g" }
storage:
type: jbod
volumes:
- { id: 0, type: persistent-claim, size: 1Ti, class: fast-ssd, deleteClaim: false }
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: prod
namespace: kafka
spec:
kafka:
version: 4.1.0 # must be a version your Strimzi release supports
rack:
topologyKey: topology.kubernetes.io/zone # broker.rack is set from each node's zone
listeners:
- name: tls
port: 9093
type: internal # in-cluster clients
tls: true
authentication: { type: tls } # mTLS client certificates
- name: external
port: 9094
type: loadbalancer # one LoadBalancer per broker plus one bootstrap LB
tls: true
authentication: { type: scram-sha-512 }
authorization:
type: simple # ACLs, managed through KafkaUser resources
config:
default.replication.factor: 3
min.insync.replicas: 2
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
entityOperator:
topicOperator: {}
userOperator: {}
cruiseControl: {} # enables KafkaRebalance
kubectl apply -f prod-cluster.yaml -n kafka
kubectl wait kafka/prod --for=condition=Ready --timeout=600s -n kafka
kubectl get pods -n kafka
# prod-brokers-3, prod-brokers-4, prod-brokers-5, prod-controllers-0, prod-controllers-1,
# prod-controllers-2, prod-entity-operator-..., prod-cruise-control-..., strimzi-cluster-operator-...
Example: topics and users as code
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: orders.created
namespace: kafka
labels: { strimzi.io/cluster: prod }
spec:
partitions: 12
replicas: 3
config:
retention.ms: 604800000 # 7 days
min.insync.replicas: 2
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: billing-service
namespace: kafka
labels: { strimzi.io/cluster: prod }
spec:
authentication:
type: tls # operator issues a client certificate
authorization:
type: simple
acls:
- resource: { type: topic, name: orders.created, patternType: literal }
operations: [Describe, Read]
- resource: { type: group, name: billing, patternType: literal }
operations: [Read]
The User Operator writes the credentials into a Secret named billing-service (certificate, key, and a PKCS#12 keystore with its password), which the application mounts.
Example: MirrorMaker 2 as a resource
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
name: east-to-prod
namespace: kafka
spec:
version: 4.1.0
replicas: 3
connectCluster: "prod" # MM2's Connect workers run against the target
clusters:
- alias: "east"
bootstrapServers: kafka-east.example.com:9094
tls: { trustedCertificates: [{ secretName: east-ca, pattern: "*.crt" }] }
- alias: "prod"
bootstrapServers: prod-kafka-bootstrap:9093
tls: { trustedCertificates: [{ secretName: prod-cluster-ca-cert, pattern: "*.crt" }] }
mirrors:
- sourceCluster: "east"
targetCluster: "prod"
topicsPattern: "orders.*"
groupsPattern: ".*"
sourceConnector:
tasksMax: 8
config:
replication.factor: 3
offset-syncs.topic.replication.factor: 3
checkpointConnector:
config:
checkpoints.topic.replication.factor: 3
sync.group.offsets.enabled: "true"
(Newer Strimzi releases are moving these resources to a v1 API with some renamed fields; run kubectl explain kafkamirrormaker2.spec against your cluster to see the exact schema.)
Day-2 operations
| Task | How with Strimzi |
|---|---|
| Change broker config | Edit spec.kafka.config; dynamic settings are applied without restart, others trigger a rolling restart |
| Rolling restart | The operator restarts one broker at a time and waits until its partitions are back in sync; manual trigger with the strimzi.io/manual-rolling-update=true annotation |
| Upgrade Kafka | Upgrade the operator, then change spec.kafka.version; the operator rolls the cluster |
| Scale out | Increase replicas in the broker KafkaNodePool, then rebalance with a KafkaRebalance in add-brokers mode |
| Scale in | Move partitions off first with KafkaRebalance in remove-brokers mode; Strimzi refuses to remove brokers that still host replicas |
| Certificates | Cluster and clients CAs are generated and renewed automatically |
| Node drains | Strimzi Drain Cleaner intercepts evictions so the operator moves brokers safely |
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
name: add-brokers-6-7
namespace: kafka
labels: { strimzi.io/cluster: prod }
spec:
mode: add-brokers
brokers: [6, 7] # new broker IDs from the scaled-out node pool
kubectl get kafkarebalance add-brokers-6-7 -n kafka # wait for ProposalReady, review the summary
kubectl annotate kafkarebalance add-brokers-6-7 strimzi.io/rebalance=approve -n kafka
Other operators exist, notably Confluent for Kubernetes (for Confluent Platform) and vendor operators for Kafka-compatible engines. The concepts are the same: custom resources describe the cluster, and the operator owns safe rollouts.
Key Takeaways
- Strimzi turns Kafka operations into declarative custom resources reconciled by an operator.
KafkaplusKafkaNodePoolresources define a KRaft cluster; topics, users, Connect, MM2, and rebalances are resources too.- The operator's value is Kafka-aware behavior: ISR-aware rolling restarts, certificate management, safe scaling with Cruise Control.
- Keep the YAML in Git and apply it through a pipeline, so every cluster change is reviewed and reproducible.
Practice
- (Beginner) Install Strimzi on a local Kubernetes cluster, deploy a cluster with one controller pool and one broker pool (ephemeral storage is fine for a test), create a
KafkaTopic, and produce to it from a pod in the cluster.- (Intermediate) Create a
KafkaUserwith read access to one topic, mount its Secret into a consumer pod, and verify that reading another topic fails with an authorization error. Then change a broker setting that requires a restart and watch the order in which the operator restarts pods.- Interview question: "What does a Kafka operator do during a rolling restart that a plain StatefulSet rolling update does not?" Hint: consider under-replicated partitions, controller placement, and the order in which pods are restarted.
Managed Kafka Services
Theory
Running Kafka well requires skills in capacity planning, upgrades, security, and on-call response (Chapters 12 and 13). A managed Kafka service hands much of that to a provider: you create a cluster through a console or API, and the provider handles brokers, disks, patches, and failures. You still own your data model, client configuration, topic design, security policies, and costs.
Analogy: owning a car versus leasing one with full service. The leasing company handles maintenance and repairs, but you still choose where to drive, how fast, and you pay for every kilometer.
Shared responsibility
| Responsibility | Self-managed | Managed (provisioned) | Serverless |
|---|---|---|---|
| Hardware, OS, JVM, Kafka patches | You | Provider | Provider |
| Broker count, instance size, disk size | You | You choose; provider operates | Provider |
| Broker configuration | You | Partly (allowed subset) | Provider |
| Rebalancing, scaling | You | Provider tools or you | Provider |
| Topic design, partitions, retention | You | You | You (within limits) |
| Client configuration and tuning | You | You | You |
| Authentication, ACLs, network access | You | You configure provider features | You configure provider features |
| Schema governance, data contracts | You | You (or add-on service) | You (or add-on service) |
| Cost control | You | You | You |
The landscape
| Service | Engine | Notable characteristics |
|---|---|---|
| Amazon MSK (Provisioned, Serverless) | Apache Kafka | Choose broker instances (Standard or Express brokers); IAM authentication; MSK Connect; tight AWS integration |
| Confluent Cloud | Kafka-compatible engine built by Confluent | Several cluster tiers from elastic to dedicated; managed connectors, Schema Registry, Flink, Cluster Linking |
| Google Cloud Managed Service for Apache Kafka | Apache Kafka | Sized by vCPU and memory; Google Cloud IAM integration |
| Azure Event Hubs (Kafka endpoint) | Event Hubs, speaking the Kafka protocol | Not Kafka internally; most client features work, but check compatibility (for example compaction and transactions support by tier) |
| Aiven for Apache Kafka | Apache Kafka | Runs on several clouds; managed Connect, MM2, and Schema Registry (Karapace) |
| Redpanda Cloud, WarpStream, others | Kafka API-compatible engines | Different internals (C++ engine, object-storage based); verify feature compatibility and latency |
Feature sets, tiers, and limits change frequently, so treat the table as a starting point and check the providers' current documentation.
Evaluation checklist
| Question | Why it matters |
|---|---|
| Is it Apache Kafka or "Kafka-compatible"? Which APIs and versions? | Transactions, compaction, Kafka Streams, Admin API calls, and newer protocols (for example the new consumer group protocol) may be missing |
| What are the limits? | Partitions per cluster or broker, throughput per partition, max message size, connections, retention |
| Which broker and topic configs can you change? | Tuning you rely on may be locked |
| How do clients connect and authenticate? | Private networking (VPC peering, PrivateLink), IAM vs SASL/SCRAM vs mTLS; affects every client library |
| How is it billed? | Per broker-hour, per partition-hour, per GB in/out/stored, cross-AZ and egress traffic; the bill often surprises at scale |
| What is the SLA, and what is excluded? | Multi-AZ requirements, maintenance windows |
| What ecosystem comes with it? | Managed connectors, Schema Registry, stream processing, monitoring integration |
| How do you leave? | Migration needs MM2 or another replicator; proprietary features (linking, managed connectors) increase lock-in |
Example: provisioning Amazon MSK with Terraform
resource "aws_msk_cluster" "orders" {
cluster_name = "orders-prod"
kafka_version = "3.8.x" # pick from the versions MSK currently offers
number_of_broker_nodes = 3 # must be a multiple of the number of subnets (AZs)
broker_node_group_info {
instance_type = "kafka.m7g.xlarge"
client_subnets = [aws_subnet.a.id, aws_subnet.b.id, aws_subnet.c.id]
security_groups = [aws_security_group.msk.id]
storage_info {
ebs_storage_info { volume_size = 1000 } # GiB per broker
}
}
client_authentication {
sasl { iam = true } # clients authenticate with AWS IAM
}
encryption_info {
encryption_in_transit {
client_broker = "TLS"
in_cluster = true
}
}
configuration_info {
arn = aws_msk_configuration.orders.arn # broker settings such as min.insync.replicas
revision = aws_msk_configuration.orders.latest_revision
}
}
Example: a compatibility smoke test
Before committing to a service, verify the features your applications depend on. This script checks idempotent production, a compacted topic, and a transaction against any endpoint:
from confluent_kafka import Producer, Consumer
from confluent_kafka.admin import AdminClient, NewTopic
conf = {"bootstrap.servers": "<endpoint>:9092"} # add the provider's security settings here
admin = AdminClient(conf)
checks = {}
fs = admin.create_topics([NewTopic("compat.compacted", 3, 3, config={"cleanup.policy": "compact"})])
try:
fs["compat.compacted"].result(); checks["compacted topic"] = "ok"
except Exception as e:
checks["compacted topic"] = f"FAILED: {e}"
try:
p = Producer({**conf, "enable.idempotence": True, "transactional.id": "compat-test"})
p.init_transactions() # fails if transactions are unsupported
p.begin_transaction()
p.produce("compat.compacted", key=b"k1", value=b"v1")
p.commit_transaction()
checks["idempotence + transactions"] = "ok"
except Exception as e:
checks["idempotence + transactions"] = f"FAILED: {e}"
c = Consumer({**conf, "group.id": "compat-test", "isolation.level": "read_committed",
"auto.offset.reset": "earliest"})
c.subscribe(["compat.compacted"])
msg = c.poll(15.0)
checks["read_committed consume"] = "ok" if msg is not None and msg.error() is None else "FAILED"
c.close()
for name, result in checks.items():
print(f"{name:28} {result}")
Key Takeaways
- Managed services remove broker operations, not responsibility for data design, clients, security, and cost.
- "Kafka-compatible" is not "Apache Kafka": verify transactions, compaction, Streams, Admin APIs, and protocol versions.
- Read the limits and the pricing model carefully; partition counts and network traffic drive cost and feasibility.
- Plan an exit path from the start: proprietary features increase lock-in.
Practice
- (Beginner) Pick two managed services and fill in the evaluation checklist for each from their public documentation. Highlight any differences that would affect an application using Kafka Streams with exactly-once.
- (Intermediate) Estimate the monthly cost of a workload with 30 MB/s in, 90 MB/s out (three consumer groups), 7 days of retention, and 600 partitions on one provisioned and one serverless offering, using their published pricing. Identify the dominant cost item for each.
- Interview question: "Your team wants to move from self-managed Kafka to a managed service. What would you check before agreeing, and how would you migrate?" Hint: compatibility, limits, networking, cost model, and an MM2-based migration with offset translation (Section 14.1).
Serverless Kafka Considerations
Theory
"Serverless" appears in two different places around Kafka, and both deserve attention:
- Serverless Kafka clusters: the provider hides brokers entirely. You create topics and pay for usage (throughput, storage, partitions) instead of servers. Examples include MSK Serverless and the elastic tiers of Confluent Cloud.
- Serverless compute with Kafka: functions (AWS Lambda, Azure Functions, Google Cloud Run, Knative) that consume from or produce to Kafka, scaling with load and possibly to zero.
Analogy: a serverless cluster is a taxi (you pay per trip and never see the garage); serverless consumers are on-call staff who are only paid while handling a delivery, but who must still sign for each package in order.
Serverless clusters: what you give up
| Aspect | Typical constraint | Design consequence |
|---|---|---|
| Partitions | Caps per cluster and per topic | Fewer, well-chosen partitions; no per-tenant topic explosion |
| Throughput | Limits per partition and per cluster; throttling above them | Spread hot keys; watch produce-throttle-time-avg on clients |
| Retention and storage | Maximum retention or storage per partition | Archive to object storage for long-term history |
| Configuration | No broker settings, restricted topic settings | Tuning happens on clients |
| Authentication | Often only the provider's IAM | Every client library must support the mechanism |
| Connections | Limits on connections and connection rate | Reuse clients; avoid one producer per request |
| Cost model | Per GB, per partition-hour, per connection-hour | Idle partitions and chatty clients cost money |
A newer variant of the same idea is object-storage-based ("diskless") Kafka, where brokers write directly to object storage instead of local disks and replicate through it (offered by several Kafka-compatible engines and proposed for Apache Kafka in KIP-1150; see Section 16.3). It trades higher produce and end-to-end latency (hundreds of milliseconds instead of single-digit milliseconds) for much lower storage and cross-AZ replication cost.
Serverless consumers: how they work
Functions do not run a long-lived poll loop. Instead, a platform component (for example a Lambda event source mapping) polls Kafka on your behalf, groups records into batches per partition, invokes your function, and commits offsets after the invocation succeeds.
Kafka topic (6 partitions) poller (managed by the platform) function instances
+--------------------+ +--------------------------------+ +-------------+
| P0 P1 P2 P3 P4 P5 | ---- poll --->| consumer group, batches per |--------->| invocation |
+--------------------+ | partition, commit on success | up to | invocation |
+--------------------------------+ one per | ... |
partition +-------------+
Concurrency is bounded by partitions; a failing batch is retried and blocks its partition.
Consequences:
- Parallelism is capped by partitions. Adding function concurrency does not help beyond one concurrent batch per partition.
- Errors block partitions. If a record always fails (a poison pill), the batch is retried and the partition stops advancing. Catch per-record errors, send failed records to a dead letter topic, or configure the platform's on-failure destination.
- At-least-once delivery. A timeout after partial processing replays the whole batch; processing must be idempotent.
- Batch size and window (
BatchSize,MaximumBatchingWindowInSecondsfor Lambda) trade latency against invocation cost.
Example: a Lambda consumer for MSK or self-managed Kafka (Python)
import base64, json, os
from confluent_kafka import Producer
# Created once per execution environment and reused across invocations
dlq = Producer({"bootstrap.servers": os.environ["BOOTSTRAP"], "enable.idempotence": True})
def handler(event, context):
# event["records"] maps "topic-partition" to a list of records, values base64-encoded
for tp, records in event["records"].items():
for r in records:
raw = base64.b64decode(r["value"])
try:
order = json.loads(raw)
apply_order_idempotently(order) # e.g. upsert keyed by order["id"]
except Exception as exc:
# Do not fail the whole batch for one bad record: park it in a dead letter topic
dlq.produce("orders.dlq", value=raw,
headers={"error": str(exc), "source": f"{tp}@{r['offset']}"})
dlq.flush(10) # flush BEFORE returning: the environment may freeze between invocations
return {"processed": sum(len(v) for v in event["records"].values())}
Example: producing from functions
Creating a producer per invocation opens new connections each time, adds latency, and can exhaust connection limits on serverless clusters. Create it at module scope, keep linger.ms small (the function is waiting anyway), and flush before returning, because background sender threads do not run while the environment is frozen:
// Node.js function using KafkaJS; the producer lives outside the handler
const { Kafka } = require("kafkajs");
const kafka = new Kafka({ clientId: "checkout-fn", brokers: process.env.BROKERS.split(",") });
const producer = kafka.producer({ idempotent: true });
const ready = producer.connect(); // connect once per execution environment
exports.handler = async (event) => {
await ready;
const order = JSON.parse(event.body);
// send() resolves only after the broker acknowledges, so nothing is left buffered on return
await producer.send({
topic: "orders.created",
messages: [{ key: order.customerId, value: JSON.stringify(order) }],
});
return { statusCode: 202 };
};
Example: scale-to-zero consumers on Kubernetes with KEDA
KEDA scales an ordinary consumer Deployment on consumer lag, giving serverless-style scaling while keeping a normal poll loop:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: billing-consumer
spec:
scaleTargetRef:
name: billing-consumer # the Deployment running the consumer
minReplicaCount: 0 # scale to zero when there is no lag
maxReplicaCount: 12 # never more than the topic's partition count
triggers:
- type: kafka
metadata:
bootstrapServers: prod-kafka-bootstrap.kafka:9092
consumerGroup: billing
topic: orders.created
lagThreshold: "500" # target lag per replica
When serverless fits, and when it does not
| Good fit | Poor fit |
|---|---|
| Spiky or unpredictable traffic, development and test environments | Steady high throughput, where provisioned capacity is cheaper |
| Small teams without Kafka operations experience | Workloads needing broker tuning, very many partitions, or long retention |
| Event-driven glue code with modest per-partition throughput | Low-latency pipelines, large stateful stream processing (Kafka Streams state stores, long-lived transactions) |
Key Takeaways
- Serverless Kafka clusters trade control and some limits (partitions, throughput, configs, auth options) for zero broker operations and usage-based pricing.
- Function-based consumers are limited to one concurrent batch per partition, retry whole batches, and need idempotent processing and explicit poison-pill handling.
- Functions that produce must reuse clients across invocations and flush or await acknowledgment before returning.
- Object-storage-based Kafka lowers cost at the price of latency; choose it for throughput-heavy, latency-tolerant workloads.
Practice
- (Beginner) A topic has 8 partitions and a Lambda consumer takes 2 seconds per batch of 100 records. What is the maximum throughput in records per second, and what are two ways to increase it?
- (Intermediate) Modify the Python Lambda handler so that a record whose
order["id"]was already processed (tracked in a key-value store) is skipped, and write a test that invokes the handler twice with the same event and checks that orders are applied once.- Interview question: "Would you run a Kafka Streams application with large state stores on a serverless platform? Why or why not?" Hint: think about state restoration from changelog topics, rebalances on every scale event, and long-lived consumer group membership.
<a id="15-design-patterns-and-application-development"></a>