2. Core Architecture
Chapter 1 described Kafka as a distributed commit log. This chapter explains how that log is built: the data structures that hold events (records, topics, partitions, offsets, and segment files), the servers that store and serve them (brokers organized into a cluster), the system that keeps track of cluster-wide state (ZooKeeper historically, KRaft today), and the replication protocol that keeps data safe when machines fail. Almost every production issue, tuning decision, and interview question about Kafka goes back to one of these mechanisms, so later chapters assume you are comfortable with all of them.
<a id="21-fundamental-building-blocks"></a>
2.1 Fundamental Building Blocks
This section works from the smallest unit of data, the record, up to the files Kafka writes on disk, and shows how each layer builds on the one below it.
Records: Key, Value, Headers, Timestamp
Theory
Every piece of data in Kafka, whether an order, a click, or a sensor reading, travels and is stored as a record (also called a message or event). Kafka is deliberately ignorant of what a record means. To the broker a record is a few byte arrays plus some metadata. Because of this, the broker never parses your JSON or Avro, which is one reason it can move data so quickly.
A record has four parts that you control or observe:
| Field | Type (on the wire) | Required | Purpose |
|---|---|---|---|
| Key | byte[] or null | No | Identifies the entity the event is about. Drives partition choice and log compaction. |
| Value | byte[] or null | No | The payload. A null value is a tombstone (a delete marker on compacted topics). |
| Headers | List of (String, byte[]) pairs | No | Metadata about the event: trace IDs, content type, schema version, source system. |
| Timestamp | long (epoch ms) | Always present | When the event was created, or when the broker appended it. |
Kafka also assigns two values that you cannot set: the partition the record lands in and its offset within that partition (covered later in this section).
Analogy: a record is like a shipping parcel. The key is the recipient's address, which decides which truck (partition) carries it. The value is what is inside the box. Headers are the stickers on the outside ("fragile", "tracking #123"), which you can read without opening the box. The timestamp is the postmark.
The key: more than an identifier
The key has two jobs:
- Partitioning: by default, records with the same key always go to the same partition, so they keep their relative order. All events for
customer-77are read in the order they were written. - Compaction: on topics with
cleanup.policy=compact, Kafka keeps at least the latest value for each key and discards older ones (Chapter 7).
If the key is null, the producer spreads records across partitions for throughput (the sticky partitioner, Chapter 4), and you give up per-entity ordering.
Headers: metadata without touching the payload
Headers were added in Kafka 0.11 (KIP-82). They let infrastructure code, such as tracing libraries, routers, and dead-letter handlers, read and write metadata without deserializing the value. Header keys are strings, values are raw bytes, the same key may appear more than once, and order is preserved.
Timestamps: two possible meanings
The topic configuration message.timestamp.type decides what the timestamp means:
| Type | Who Sets It | Meaning | Use When |
|---|---|---|---|
CreateTime (default) | Producer (or the client library, using the current time) | When the event happened at the source | You care about event time (analytics, windowing) |
LogAppendTime | Broker, overwriting the producer's value | When the broker wrote the record | You want a trustworthy, monotonic ingestion time |
With CreateTime, a misbehaving producer can send timestamps far in the past or future. The broker settings message.timestamp.before.max.ms and message.timestamp.after.max.ms reject records outside an acceptable window. Timestamps also drive time-based retention and the offsetsForTimes lookup, so bad timestamps can cause surprising deletions or seeks.
How records are actually stored: record batches
Producers never send records one at a time. They group records for the same partition into a record batch, and the batch is the unit that is compressed, sent, written to disk, replicated, and fetched. The broker stores the batch exactly as it received it (unless it must convert formats or overwrite timestamps), which is what makes zero-copy transfer possible.
RECORD BATCH (message format v2, "magic 2")
+-------------------------------------------------------------------+
| baseOffset int64 offset of the first record |
| batchLength int32 |
| partitionLeaderEpoch int32 set by the leader (Section 2.4) |
| magic int8 = 2 |
| crc uint32 checksum of everything below |
| attributes int16 compression | timestampType | |
| isTransactional | isControl |
| lastOffsetDelta int32 |
| baseTimestamp int64 |
| maxTimestamp int64 |
| producerId int64 \ |
| producerEpoch int16 > idempotence / transactions (Ch 8)|
| baseSequence int32 / |
| recordCount int32 |
+-------------------------------------------------------------------+
| RECORDS (compressed together if compression is enabled) |
| +-------------------------------------------------------------+ |
| | length varint | |
| | attributes int8 (unused) | |
| | timestampDelta varlong relative to baseTimestamp | |
| | offsetDelta varint relative to baseOffset | |
| | keyLength/key varint + bytes (-1 = null) | |
| | valueLength/val varint + bytes (-1 = null) | |
| | headers varint count + [keyLen key valLen val]... | |
| +-------------------------------------------------------------+ |
| ... more records ... |
+-------------------------------------------------------------------+
Storing offsets and timestamps as small deltas encoded as variable-length integers keeps per-record overhead to a few bytes, which matters when you store billions of records.
Size limits
A record (more precisely, a batch) cannot exceed the broker's message.max.bytes (about 1 MB by default), overridable per topic with max.message.bytes. The producer has its own max.request.size. Kafka is designed for many small-to-medium records. For large blobs (images, videos), store the object elsewhere (for example, in S3) and send a reference to it, a technique known as the claim check pattern.
Example: producing and consuming a fully populated record
// Producer side
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
ProducerRecord<String, String> record = new ProducerRecord<>(
"orders", // topic
null, // partition: null lets the partitioner choose from the key
Instant.parse("2026-10-23T09:15:42Z").toEpochMilli(), // event time (CreateTime)
"CUST-77", // key: all of this customer's events stay in order
"{\"orderId\":\"ORD-1042\",\"total\":39.98}" // value: opaque bytes to the broker
);
record.headers()
.add("trace-id", "4bf92f3577b34da6".getBytes(StandardCharsets.UTF_8))
.add("content-type", "application/json".getBytes(StandardCharsets.UTF_8));
RecordMetadata meta = producer.send(record).get(); // block only for the demo
// Partition and offset are assigned by Kafka, not by us
System.out.printf("stored in partition %d at offset %d%n", meta.partition(), meta.offset());
}
// Consumer side
for (ConsumerRecord<String, String> rec : consumer.poll(Duration.ofMillis(500))) {
Header trace = rec.headers().lastHeader("trace-id"); // headers are read without parsing the value
System.out.printf("p=%d off=%d key=%s ts=%d (%s) trace=%s value=%s%n",
rec.partition(), rec.offset(), rec.key(),
rec.timestamp(), rec.timestampType(), // CreateTime or LogAppendTime
trace == null ? "-" : new String(trace.value(), StandardCharsets.UTF_8),
rec.value());
}
The same in Python with confluent-kafka:
from confluent_kafka import Producer
producer = Producer({"bootstrap.servers": "localhost:9092"})
producer.produce(
topic="orders",
key="CUST-77",
value='{"orderId":"ORD-1042","total":39.98}',
headers=[("trace-id", b"4bf92f3577b34da6"), ("content-type", b"application/json")],
timestamp=1761210942000, # epoch milliseconds; omit to use the current time
)
producer.flush() # wait until the batch has been delivered
Key Takeaways
- A record has a key, value, headers, and timestamp. Kafka adds the partition and offset.
- Kafka treats keys and values as opaque bytes. Serialization is the client's job (Chapter 6).
- The key decides the partition (and therefore ordering) and is the identity used by log compaction. A null value is a tombstone.
CreateTimerecords event time and is set by the producer.LogAppendTimerecords ingestion time and is set by the broker.- Records are always grouped into compressed batches, which are the real unit of storage, replication, and transfer.
Practice
- (Beginner) For a topic of payment events, pick a key and justify your choice. Then list two pieces of information that belong in headers rather than in the value.
- (Intermediate) A team uses
CreateTime, and one producer's clock is set 3 years in the past. Describe two things that could go wrong with time-based retention and with consumers that useoffsetsForTimes, and name the broker settings that would prevent it.- Interview question: "Why does Kafka store records in batches instead of individually, and what does the broker do with a compressed batch?" Hint: think about per-record overhead, compression ratios on similar data, and whether the broker needs to decompress anything to store and serve it.
Topics
Theory
If every event went into one giant log, every consumer would have to read and filter everything. A topic solves this by giving each category of events its own named stream: orders, payments, page-views. Producers write to a topic by name, and consumers subscribe to the topics they care about.
Analogy: topics are like folders in a filing cabinet. Each folder holds one kind of document. Unlike a real folder, you never remove documents when you read them, and many people can read the same folder at the same time.
A topic is a logical concept. Physically, a topic is nothing more than a set of partitions (next topic), each stored as files on some brokers. When you "write to a topic", you are really writing to one of its partitions.
Topic "orders" (logical name + configuration)
+----------------------------------------------------------+
| Partition 0 : [0][1][2][3][4][5][6] ... |
| Partition 1 : [0][1][2][3] ... | physical: one log per partition,
| Partition 2 : [0][1][2][3][4] ... | spread across brokers
+----------------------------------------------------------+
Configuration: partitions=3, replication.factor=3,
retention.ms=604800000, cleanup.policy=delete, ...
What a topic defines
| Property | Set At | Can Change Later? | Notes |
|---|---|---|---|
| Name | Creation | No (create a new topic and migrate) | Unique within the cluster |
| Topic ID | Creation (by Kafka) | No | A UUID (KIP-516) that distinguishes a deleted-and-recreated topic from the original |
| Partition count | Creation | Increase only | Increasing changes key-to-partition mapping (see Partitions) |
| Replication factor | Creation | Yes, via partition reassignment | Section 2.4 |
| Configuration overrides | Creation or any time | Yes, dynamically | retention.ms, cleanup.policy, min.insync.replicas, max.message.bytes, ... |
Topic-level configurations override broker-level defaults. For example, the broker's log.retention.hours=168 is the default, and a topic can set retention.ms=86400000 to keep data for one day.
Retention policies
A topic does not delete data when it is read. It deletes data according to its cleanup.policy:
| Policy | Behavior | Typical Use |
|---|---|---|
delete (default) | Remove old data once it is older than retention.ms or the partition exceeds retention.bytes | Event streams: clicks, logs, orders |
compact | Keep at least the latest record per key, remove older versions | Current state: customer profiles, configuration, changelogs |
compact,delete | Compact, and also delete anything past retention | Changelogs where very old keys can be dropped |
Naming rules and conventions
- Allowed characters: ASCII letters, digits,
.,_, and-. Maximum length is 249 characters. - Avoid mixing
.and_: Kafka replaces.with_in metric names, souser.eventsanduser_eventswould collide. The CLI warns about this. - Names starting with
__are reserved for internal topics such as__consumer_offsets(committed consumer offsets) and__transaction_state(transaction coordinator state). - A common convention is
<domain>.<entity>.<event-or-purpose>[.v<version>], for examplesales.orders.placed.v1. Naming strategy is covered in Section 15.2.
Automatic topic creation
When auto.create.topics.enable=true (the broker default), producing to or fetching metadata for a missing topic creates it with the broker's defaults (num.partitions=1, default.replication.factor=1). That is convenient on a laptop and dangerous in production: a typo in a topic name silently creates a new, under-replicated topic. Most production clusters set it to false and create topics explicitly.
Example: creating a topic programmatically
Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
try (Admin admin = Admin.create(props)) {
NewTopic orders = new NewTopic("sales.orders.placed.v1", 6, (short) 3) // name, partitions, replication factor
.configs(Map.of(
TopicConfig.RETENTION_MS_CONFIG, String.valueOf(Duration.ofDays(7).toMillis()),
TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_DELETE,
TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2" // see Section 2.4
));
admin.createTopics(List.of(orders)).all().get(); // throws TopicExistsException if it already exists
// Read back what the cluster actually stored
TopicDescription d = admin.describeTopics(List.of("sales.orders.placed.v1"))
.allTopicNames().get()
.get("sales.orders.placed.v1");
System.out.println("topicId=" + d.topicId() + " partitions=" + d.partitions().size());
}
The equivalent command line (Chapter 3 covers the tools in detail):
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic sales.orders.placed.v1 \
--partitions 6 --replication-factor 3 \
--config retention.ms=604800000 \
--config min.insync.replicas=2
Key Takeaways
- A topic is a named, logical stream of related events. Physically, it is a set of partitions.
- Topic configuration (retention, cleanup policy, durability settings) overrides broker defaults and can be changed at runtime.
cleanup.policy=deletekeeps data for a time or size limit.compactkeeps the latest value per key.- Reading never deletes data. Retention does.
- Disable
auto.create.topics.enablein production and create topics with deliberate partition and replication settings.
Practice
- (Beginner) Choose a
cleanup.policyfor each topic and explain why: (a) website click events, (b) the current shipping address for each customer, (c) application audit logs that must be kept for 90 days.- (Intermediate) A developer produces to
order-evnts(a typo) on a cluster with default settings. Describe exactly what the cluster does, what the resulting topic looks like, and why this is a reliability risk.- Interview question: "Why is a topic called a logical concept in Kafka? What actually exists on disk?" Hint: follow a single
send()call from the topic name down to a directory on a broker.
Partitions
Theory
A single log on a single machine has hard limits: one disk's throughput, one machine's network card, and one consumer reading it in order. To scale past those limits, Kafka splits each topic into partitions. Each partition is an independent, ordered, append-only log that can live on a different broker and be read by a different consumer.
Partitions are the most important concept in Kafka because they are the unit of:
| Unit Of | Meaning |
|---|---|
| Parallelism | Within a consumer group, each partition is read by at most one consumer. More partitions allow more parallel consumers. |
| Ordering | Order is guaranteed within a partition, never across partitions. |
| Storage and distribution | Each partition is a directory on a broker's disk. Partitions are spread across brokers. |
| Replication | Each partition is replicated independently, with its own leader (Section 2.4). |
Analogy: a supermarket with one checkout lane is simple but slow. Opening more lanes (partitions) serves more customers in parallel. Within a lane, customers are served in order, but there is no ordering between lanes: a customer in lane 3 may finish before someone who arrived earlier in lane 1.
How a record chooses its partition
The producer, not the broker, decides which partition a record goes to:
- If the record specifies a partition explicitly, that partition is used.
- If the record has a key, the Java client computes
toPositive(murmur2(keyBytes)) % numPartitions. The same key always maps to the same partition, as long as the partition count does not change. - If the key is null, the producer uses sticky partitioning: it fills a batch for one partition, then switches to another, spreading load evenly over time (Chapter 4).
Producer Topic "orders" (3 partitions)
key="CUST-77" -- murmur2 % 3 = 1 --> P1: [CUST-77 ord1][CUST-77 ord2] ... (ordered)
key="CUST-12" -- murmur2 % 3 = 0 --> P0: [CUST-12 ord1] ...
key="CUST-40" -- murmur2 % 3 = 2 --> P2: [CUST-40 ord1] ...
key=null -- sticky batch --> any partition
Ordering is per partition
If you need all events for a given order, account, or device processed in sequence, give them the same key. If you need a total order over all events in a topic, you need a single partition, which means a single consumer per group and limited throughput. Total ordering is rarely truly required, and per-entity ordering is usually enough.
Changing the partition count
You can increase a topic's partition count, but you can never decrease it. Increasing it is also risky for keyed topics: the modulo changes, so most keys move to a different partition. Old events for a key stay in the old partition while new events go to the new one, which breaks per-key ordering during the transition and breaks the co-partitioning that stream joins rely on (Chapter 10).
The common advice is to choose a partition count with room for growth when you create a keyed topic (Section 3.3 covers sizing), rather than increasing it later.
Cost of partitions
Partitions are not free. Each one uses open file handles, memory for indexes and buffers, replication traffic, and metadata entries on the controller. KRaft (Section 2.3) raised the practical limit per cluster considerably compared to ZooKeeper, but thousands of tiny, unused partitions still waste resources and slow down failover and rebalances.
Example: how keys map to partitions, and what happens when you add partitions
In Java, you can reproduce the default partitioner's choice with Kafka's own utility methods:
import org.apache.kafka.common.utils.Utils;
byte[] keyBytes = "CUST-77".getBytes(StandardCharsets.UTF_8);
int numPartitions = 6;
// The same formula the default partitioner uses for non-null keys
int partition = Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
System.out.println("CUST-77 -> partition " + partition);
This Python simulation (using CRC32 as a stand-in hash) shows how many keys move when a topic grows from 3 to 4 partitions:
import zlib
keys = [f"CUST-{i}" for i in range(1, 1001)]
def partition_for(key: str, num_partitions: int) -> int:
return zlib.crc32(key.encode()) % num_partitions # hash(key) mod N, like Kafka
before = {k: partition_for(k, 3) for k in keys}
after = {k: partition_for(k, 4) for k in keys}
moved = sum(1 for k in keys if before[k] != after[k])
print(f"{moved} of {len(keys)} keys changed partition") # roughly 75% of keys move
A cross-language pitfall: librdkafka-based clients (Python confluent-kafka, Go, .NET, and others) default to a different hash (consistent_random, based on CRC32) than the Java client's murmur2. If Java and Python services both produce keyed records to the same topic, set "partitioner": "murmur2_random" in the librdkafka clients so the same key lands in the same partition.
from confluent_kafka import Producer
producer = Producer({
"bootstrap.servers": "localhost:9092",
"partitioner": "murmur2_random", # match the Java client's key -> partition mapping
})
Key Takeaways
- A partition is an independent, ordered log. A topic is a set of partitions.
- Partitions are the unit of parallelism, ordering, storage distribution, and replication.
- Records with the same key go to the same partition, so they keep their order. There is no ordering across partitions.
- The partition count can only increase, and increasing it remaps most keys. Plan it up front for keyed topics.
- Clients in different languages may hash keys differently. Align partitioners when multiple producers share a topic.
Practice
- (Beginner) A topic has 4 partitions and a consumer group with 4 consumers is falling behind. What is the maximum number of consumers in that group that can do useful work, and what must change to scale further?
- (Intermediate) Modify the Python simulation to grow from 6 to 12 partitions. What fraction of keys move? Why is doubling the partition count gentler than going from 6 to 7 for some keys, and why does it still break ordering guarantees during the transition?
- Interview question: "How does Kafka guarantee ordering, and what would you do if the business requires all events for a customer to be processed in order?" Hint: consider what the key determines, and what could reorder records inside a single partition on the producer side (Chapter 4,
max.in.flight.requestsand idempotence).
Offsets
Theory
Each partition is a log, and every record in it has a position: its offset. An offset is a 64-bit integer, assigned by the partition leader when the record is appended, that increases with each record. Offsets are per partition: offset 42 in partition 0 and offset 42 in partition 1 are unrelated records. So a record's full address is the tuple (topic, partition, offset).
Why does this matter? Because Kafka does not track which records each consumer has read. The consumer tracks its own position as a single number per partition. That is what makes Kafka cheap to read from: thousands of consumers can read the same partition, and the broker stores no per-message delivery state at all.
Analogy: an offset is a page number in a book that never stops growing. Each reader uses their own bookmark. Closing the book and reopening it later, you just turn to your bookmark. Anyone can move their bookmark back to reread a chapter, and doing so does not affect other readers.
Important offsets in a partition
Partition "orders-0"
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 100 | 101 | ... | 107 | 108 | 109 | 110 | 111 | 112 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
^ ^ ^ ^
| | | +---- log end offset (LEO) = 113
| | +---------- high watermark (HW) = 112
| +---------------------------------- committed offset, group "billing" = 108
+---------------------------------------------------------- log start offset = 100 (0-99 deleted)
100-107 : already processed by group "billing"
108-111 : readable but not yet processed -> lag = 112 - 108 = 4
112 : on the leader, not yet on all in-sync replicas -> invisible to consumers
| Offset | Meaning |
|---|---|
| Log start offset | The oldest offset still available. Earlier records were removed by retention or compaction. |
| Committed offset | The next offset a consumer group will read after a restart. It is stored in __consumer_offsets. |
| Current position | The next offset the consumer will fetch in its current session (may be ahead of the committed offset). |
| High watermark (HW) | The offset up to which all in-sync replicas have the data. Consumers can only read below it (Section 2.4). |
| Log end offset (LEO) | The offset the next appended record will receive. |
| Consumer lag | High watermark minus committed offset: how far behind a group is. |
A common off-by-one source: the committed offset is the offset of the next record to read, not the last record processed. If you processed offset 107, you commit 108.
Offsets are not always contiguous
Offsets always increase, but consumers can see gaps:
- Compaction removes older records for a key, leaving holes.
- Transactions write control records (commit/abort markers) that take an offset but are never returned to applications.
So never assume next offset = current offset + 1 when computing counts. Use offsets as positions, not as record counts.
What happens when the committed offset is invalid
When a group has no committed offset (it is new) or its committed offset was deleted by retention, the consumer setting auto.offset.reset decides where to start:
| Value | Behavior |
|---|---|
latest (default) | Start at the end, reading only new records |
earliest | Start at the log start offset, reading all retained history |
none | Throw an exception and let the application decide |
Seeking: moving the bookmark yourself
Because the position is just a number, consumers can rewind or skip ahead on purpose: seek() to a specific offset, seekToBeginning(), seekToEnd(), or offsetsForTimes() to find the first offset at or after a timestamp. This is how you replay a day of events after fixing a bug.
Example: inspecting and moving offsets
TopicPartition tp = new TopicPartition("orders", 0);
consumer.assign(List.of(tp)); // manual assignment, no group rebalancing
// Where does the log start and end?
long start = consumer.beginningOffsets(List.of(tp)).get(tp);
long end = consumer.endOffsets(List.of(tp)).get(tp); // HW for read_uncommitted consumers
System.out.printf("available offsets: [%d, %d)%n", start, end);
// Replay everything since yesterday at 00:00 UTC
long ts = LocalDate.now(ZoneOffset.UTC).minusDays(1).atStartOfDay(ZoneOffset.UTC).toInstant().toEpochMilli();
OffsetAndTimestamp found = consumer.offsetsForTimes(Map.of(tp, ts)).get(tp);
if (found != null) {
consumer.seek(tp, found.offset()); // move this consumer's position
} else {
consumer.seekToEnd(List.of(tp)); // no record that new: nothing to replay
}
for (ConsumerRecord<String, String> rec : consumer.poll(Duration.ofSeconds(1))) {
process(rec);
// Commit "next to read" = last processed + 1
consumer.commitSync(Map.of(tp, new OffsetAndMetadata(rec.offset() + 1)));
}
Checking a group's committed offsets and lag from the command line:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 0 108 112 4 consumer-billing-1 /10.0.0.5 consumer-billing-1
billing orders 1 5530 5530 0 consumer-billing-2 /10.0.0.6 consumer-billing-2
Here CURRENT-OFFSET is the committed offset. Despite its name, the LOG-END-OFFSET column shows the offset consumers can read up to (the high watermark), which is why lag computed from it reflects only readable data.
Key Takeaways
- An offset is a per-partition, increasing 64-bit position.
(topic, partition, offset)uniquely identifies a record. - Consumers own their position. The broker keeps no per-message delivery state, which makes fan-out and replay cheap.
- The committed offset is the next record to read (last processed + 1) and is stored in
__consumer_offsets. - Offsets can have gaps (compaction, transaction markers). Do not use them as record counts.
auto.offset.resetapplies only when there is no valid committed offset.seekandoffsetsForTimeslet you replay deliberately.
Practice
- (Beginner) A consumer processes records at offsets 40, 41, and 42 in partition 3, then commits. Which offset should it commit, and what record will it read first after a restart?
- (Intermediate) A group was offline for 10 days on a topic with 7-day retention. Its committed offset is 5,000, but the log start offset is now 90,000. What happens on restart with
auto.offset.reset=latest, withearliest, and withnone? Which records are lost to this group in each case?- Interview question: "Why can Kafka support thousands of consumers on one topic while a traditional broker struggles with per-message acknowledgments?" Hint: compare the state the broker must store and update for each approach.
Segments and Log Files
Theory
If each partition were one ever-growing file, deleting old data would mean rewriting the file, and finding a record by offset would mean scanning gigabytes. Kafka avoids both problems by splitting each partition's log into segments: a sequence of files, each covering a contiguous range of offsets.
- Exactly one segment per partition is active: new records are appended only to it.
- When the active segment reaches a size or age limit, it is rolled: closed, and a new active segment is opened.
- Closed segments are immutable. Retention deletes them whole, which is just a file deletion and is very cheap.
Analogy: a diary written across numbered notebooks. You only write in the current notebook. When it is full, you start a new one. To throw away old entries, you discard entire old notebooks rather than tearing out pages. Each notebook has a small index card at the front saying roughly which page each date starts on.
What is inside a partition directory
Each partition replica is a directory named <topic>-<partition> under one of the broker's log.dirs. Each segment is a group of files sharing a name, which is the segment's base offset (its first offset) padded to 20 digits:
/var/lib/kafka/data/orders-0/
|-- 00000000000000000000.log <- segment 1: offsets 0 .. 368768 (closed)
|-- 00000000000000000000.index <- sparse offset index for segment 1
|-- 00000000000000000000.timeindex <- sparse timestamp index for segment 1
|-- 00000000000000368769.log <- segment 2: offsets 368769 .. (ACTIVE, being appended)
|-- 00000000000000368769.index
|-- 00000000000000368769.timeindex
|-- 00000000000000368769.snapshot <- producer state snapshot (idempotence, Chapter 8)
|-- leader-epoch-checkpoint <- leader epoch -> start offset history (Section 2.4)
`-- partition.metadata <- topic ID for this partition
| File | Contents | Entry Format |
|---|---|---|
.log | Record batches, exactly as received from the producer | Variable-length batches |
.index | Offset to byte position in the .log file | 8 bytes: relative offset (4) + file position (4) |
.timeindex | Timestamp to offset | 12 bytes: timestamp (8) + relative offset (4) |
.txnindex | Aborted transactions in this segment (only if transactions are used) | Used by read_committed consumers |
.snapshot | Producer IDs and sequence numbers | Used to rebuild idempotence state quickly |
Sparse indexes
The indexes do not contain every offset. By default, the broker adds an index entry roughly every 4 KB of log data (index.interval.bytes=4096). Finding offset N therefore takes three steps:
Find offset 368,900 in orders-0
1. Pick the segment: binary search over base offsets in memory
[0, 368769, 737538, ...] -> 368769 <= 368900 < 737538 -> segment 368769
2. Search that segment's .index (memory-mapped) for the greatest entry <= 368900
relative offset 131 (= 368900 - 368769)
.index: ... (96 -> pos 61440) (128 -> pos 81920) (160 -> pos 102400) ...
^ closest entry at or below 131
3. Scan the .log file forward from byte 81920 until offset 368900 is reached (at most ~4 KB)
Storing relative offsets (offset minus base offset) in 4 bytes keeps index entries small, and memory-mapping the index files means lookups usually hit RAM.
When a segment rolls
| Setting (Broker / Topic) | Default | Rolls When |
|---|---|---|
log.segment.bytes / segment.bytes | 1 GiB | The active segment reaches this size |
log.roll.hours / segment.ms | 7 days | The active segment is this old |
log.index.size.max.bytes / segment.index.bytes | 10 MiB | The offset or time index file is full |
Why segments matter for retention
Retention works on closed segments only. A segment is deleted when its newest record is older than retention.ms (or when the partition exceeds retention.bytes). The active segment is never deleted. This has a surprising consequence: on a low-traffic topic with retention.ms of 1 hour but the default 7-day segment.ms, data can live for about a week because the active segment has not rolled yet. Lowering segment.ms fixes this, at the cost of more files.
Chapter 7 covers segment internals, compaction, and tiered storage in depth.
Example: looking inside a segment and simulating the lookup
Kafka ships a tool that decodes segment files:
bin/kafka-dump-log.sh --print-data-log \
--files /var/lib/kafka/data/orders-0/00000000000000000000.log
Dumping /var/lib/kafka/data/orders-0/00000000000000000000.log
Log starting offset: 0
baseOffset: 0 lastOffset: 2 count: 3 baseSequence: 0 lastSequence: 2 producerId: 1000
producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: false isControl: false
position: 0 CreateTime: 1761210942120 size: 196 magic: 2 compresscodec: none crc: 2415896217 isvalid: true
| offset: 0 CreateTime: 1761210942120 keySize: 7 valueSize: 36 sequence: 0 headerKeys: [trace-id] key: CUST-77 payload: {"orderId":"ORD-1042","total":39.98}
| offset: 1 CreateTime: 1761210942121 keySize: 7 valueSize: 36 sequence: 1 headerKeys: [trace-id] key: CUST-12 payload: {"orderId":"ORD-1043","total":12.50}
...
(Output abridged.) Notice that metadata is stored once per batch (baseOffset, producerId, crc), while each record carries only small deltas.
A Python model of the three-step lookup:
from bisect import bisect_right
# Step 1 data: base offsets of each segment, kept sorted in memory by the broker
segment_bases = [0, 368_769, 737_538]
# Step 2 data: sparse index per segment -> list of (relative_offset, byte_position)
sparse_index = {
368_769: [(0, 0), (32, 20_480), (64, 40_960), (96, 61_440), (128, 81_920), (160, 102_400)],
}
def locate(offset: int) -> tuple[int, int]:
# 1. Find the segment: greatest base offset <= target
seg = segment_bases[bisect_right(segment_bases, offset) - 1]
# 2. Find the greatest index entry <= relative target
entries = sparse_index[seg]
rel = offset - seg
i = bisect_right([e[0] for e in entries], rel) - 1
start_pos = entries[i][1]
# 3. The broker would now scan the .log file forward from start_pos
return seg, start_pos
print(locate(368_900)) # (368769, 81920): open segment 368769, scan from byte 81920
Key Takeaways
- Each partition is stored as a series of segment files. Only the active segment is written to.
- Segment files are named by their base offset. Each
.loghas a sparse.index(offset to position) and.timeindex(time to offset). - Lookups are a binary search over segments, then over a memory-mapped sparse index, then a short sequential scan.
- Segments roll by size, age, or full index. Retention deletes whole closed segments and never the active one.
segment.msandsegment.bytesaffect how precisely retention is enforced and how many files a broker must manage.
Practice
- (Beginner) A partition directory contains
00000000000000000000.log,00000000000000052000.log, and00000000000000104000.log. Which file contains offset 60,123? Which file is most likely the active segment?- (Intermediate) A topic receives 10 records per hour and has
retention.ms=3600000(1 hour) with default segment settings. A compliance officer finds 5-day-old records. Explain why, and propose a configuration change, including its trade-off.- Interview question: "How does Kafka find a record by offset quickly without an index entry for every record?" Hint: describe the three lookup steps and explain why a full index would be wasteful given how consumers typically read.
<a id="22-brokers-and-clusters"></a>
2.2 Brokers and Clusters
This section moves from data structures to the servers that host them: what a broker does, how brokers form a cluster, which node coordinates the cluster, how clients find brokers, and how replicas are spread across failure domains.
Broker Role and Responsibilities
Theory
A broker is a single Kafka server process. It is "just" a JVM process that owns some disks and listens on some ports, but it does a lot of work:
| Responsibility | What It Means |
|---|---|
| Storage | Hosts partition replicas as segment files in its log.dirs, and enforces retention and compaction |
| Serving writes | Accepts Produce requests for partitions it leads, appends them, and acknowledges according to acks |
| Serving reads | Answers Fetch requests from consumers (and from follower replicas on other brokers) |
| Replication | Acts as a follower for partitions led elsewhere by fetching from their leaders |
| Metadata | Answers Metadata requests so clients learn which broker leads which partition |
| Coordination | Hosts coordinators: the group coordinator (consumer groups, offset commits), the transaction coordinator (Chapter 8), and the share coordinator (share groups) |
A key design choice: a broker only serves reads and writes for partitions it leads (followers can serve reads when follower fetching is enabled, see Rack Awareness). Clients therefore talk directly to the right broker for each partition. No central router sits in the data path, so adding brokers adds capacity almost linearly.
Analogy: a broker is like a branch of a library network. Each branch holds some of the collection (partitions), lends out what it holds (serves fetches), accepts donations for its own shelves (produces), and keeps photocopies of some books held at other branches (follower replicas) in case one branch burns down.
Brokers are deliberately "dumb"
Compared to traditional message brokers, Kafka brokers do very little per message:
- No per-message acknowledgments or redelivery tracking (consumers track offsets).
- No routing rules, filtering, or priority queues.
- No payload parsing: batches are written and served as opaque bytes, often straight from the page cache to the socket (zero-copy).
This "smart client, simple broker" split is the source of much of Kafka's throughput.
Inside a broker: the request path
clients / other brokers
|
v
+---------------------+
| Acceptor thread | one per listener: accepts TCP connections
+---------------------+
|
v
+---------------------+
| Network threads | num.network.threads (default 3): read requests,
| (processors) | write responses
+---------------------+
| request queue
v
+---------------------+
| I/O threads | num.io.threads (default 8): append to log,
| (request handlers) | read from log, update metadata
+---------------------+
| |
| v
| +----------------+ requests that must wait: acks=all produce
| | Purgatory | (until followers catch up), fetches with
| +----------------+ fetch.min.bytes not yet satisfied
v |
response queue <-+
|
v
network thread -> client
You rarely tune these threads early on, but knowing the path helps you read metrics like RequestQueueTimeMs and RequestHandlerAvgIdlePercent when diagnosing slowness (Chapters 12 and 16).
Listeners
A broker can expose several listeners, each a named endpoint with its own security protocol. Separating them lets you, for example, use plaintext inside a private network and TLS with authentication for external clients.
| Setting | Purpose |
|---|---|
listeners | Interfaces and ports the broker binds to |
advertised.listeners | Addresses the broker tells clients to use (must be reachable by them) |
listener.security.protocol.map | Maps each listener name to PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL |
inter.broker.listener.name | Listener used for broker-to-broker replication |
controller.listener.names | Listener(s) used to talk to the KRaft controllers |
Example: a broker configuration (KRaft mode)
# ----- Identity and role -----
process.roles=broker # this node only serves data (see Cluster Topology)
node.id=4 # unique across ALL nodes in the cluster
# ----- How to reach the controllers (Section 2.3) -----
controller.quorum.bootstrap.servers=ctrl1:9093,ctrl2:9093,ctrl3:9093
controller.listener.names=CONTROLLER
# ----- Listeners -----
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9094
advertised.listeners=INTERNAL://broker4.kafka.internal:9092,EXTERNAL://kafka4.example.com:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SASL_SSL,CONTROLLER:PLAINTEXT
inter.broker.listener.name=INTERNAL # replication traffic stays on the private network
# ----- Storage -----
log.dirs=/var/lib/kafka/data-1,/var/lib/kafka/data-2 # several disks (JBOD) are allowed
# ----- Defaults for new topics -----
num.partitions=6
default.replication.factor=3
min.insync.replicas=2
auto.create.topics.enable=false
Key Takeaways
- A broker is a Kafka server that stores partition replicas and serves produce, fetch, and metadata requests.
- Clients send data directly to the leader of each partition. No central router exists, so the cluster scales horizontally.
- Brokers do little per message (no acks per message, no routing, no parsing), which enables high throughput.
- Brokers also host the group, transaction, and share coordinators.
- Listeners separate traffic by network and security protocol.
advertised.listenersmust be reachable by clients.
Practice
- (Beginner) List four kinds of requests a broker handles, and say which ones come from producers, consumers, and other brokers.
- (Intermediate) Draw the path of an
acks=allproduce request through a broker's threads. Explain why it waits in purgatory and what event releases it.- Interview question: "Kafka brokers are often described as 'dumb' compared to RabbitMQ. Why is that a strength?" Hint: list what the broker does not do per message and what the clients take on instead.
Cluster Topology
Theory
One broker is a single point of failure with a fixed capacity ceiling. A Kafka cluster is a group of nodes that share one set of topics and one consistent view of metadata. Data is spread across brokers for capacity, and copied across brokers for fault tolerance.
In KRaft mode (the only mode in Kafka 4.x), each node has one or both of two roles, set by process.roles:
| Role | Responsibilities | Typical Count |
|---|---|---|
broker | Stores data, serves clients, replicates partitions | 3 to hundreds |
controller | Stores and manages cluster metadata via a Raft quorum (Section 2.3); elects partition leaders | 3 or 5 |
broker,controller (combined) | Both, in one JVM | Development and very small clusters only |
Two deployment shapes
COMBINED MODE (dev / small) ISOLATED MODE (production)
+-------------+ +-------------+ +-------------+ +--------+ +--------+ +--------+
| node 1 | | node 2 | | node 3 | | ctrl 1 | | ctrl 2 | | ctrl 3 | controller
| broker + | | broker + | | broker + | | (voter)| | (voter)| | (voter)| quorum
| controller | | controller | | controller | +--------+ +--------+ +--------+
+-------------+ +-------------+ +-------------+ | metadata |
v v
+--------+ +--------+ +--------+ +--------+
Simple, fewer machines, but a busy |broker 4| |broker 5| |broker 6| |broker 7| data
broker can starve the controller, and +--------+ +--------+ +--------+ +--------+ plane
controllers cannot be scaled or ^ ^ ^ ^
restarted independently +------ producers / consumers -----+
Isolated mode is recommended for production because controllers are latency-sensitive: a broker under heavy load (long GC pauses, saturated disks) could otherwise delay metadata operations for the whole cluster. Controller nodes need little disk and modest CPU, so they can be small machines.
How load is spread
- Partitions of every topic are distributed across brokers.
- Leadership is distributed too. Each partition has a preferred leader (the first replica in its assignment list), and the controller periodically moves leadership back to preferred leaders (
auto.leader.rebalance.enable=true) so that one broker does not end up leading everything after a failure. - Kafka does not automatically move partition data when you add a broker. A new broker starts empty until you run a partition reassignment or use a balancing tool such as Cruise Control (Chapter 12).
Topic "orders": 6 partitions, replication factor 3, on 4 brokers (L = leader, f = follower)
Broker 1 Broker 2 Broker 3 Broker 4
P0 L f f
P1 L f f
P2 f L f
P3 f f L
P4 L f f
P5 f L f
Leaders per broker: B1=2, B2=2, B3=1, B4=1 -> balanced enough
Cluster identity
Every cluster has a cluster ID (a base64 UUID) generated once and written into each node's storage directory when it is formatted. A node refuses to join a cluster whose ID differs from the one on its disk, which protects against accidentally pointing a broker at the wrong cluster.
Sizing rules of thumb
| Question | Rule of Thumb |
|---|---|
| Minimum brokers for production | 3, so that replication factor 3 is possible and one broker can be lost |
| Controllers | 3 (tolerates 1 failure) or 5 (tolerates 2). Never an even number (Section 2.3). |
| When to add brokers | Disk, network, or CPU consistently near capacity; partition counts per broker too high |
| Failure domains | Spread brokers and controllers across racks or availability zones (see Rack Awareness) |
Chapter 13 covers capacity planning in detail.
Example: inspecting a cluster's topology
# Which brokers exist and which API versions they support
bin/kafka-broker-api-versions.sh --bootstrap-server broker4:9092 | grep -E '^[^ ]'
# Where every replica and leader lives for a topic
bin/kafka-topics.sh --bootstrap-server broker4:9092 --describe --topic orders
Topic: orders TopicId: q8bGv1TfRXSX0n3b1n7Y2w PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2
Topic: orders Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: orders Partition: 1 Leader: 2 Replicas: 2,3,4 Isr: 2,3,4
Topic: orders Partition: 2 Leader: 3 Replicas: 3,4,1 Isr: 3,4,1
...
(Output abridged.) Replicas is the assignment, in order, so its first entry is the preferred leader. Isr is explained in Section 2.4.
Key Takeaways
- A cluster is a set of nodes sharing topics and metadata. Each node has the
brokerrole, thecontrollerrole, or both. - Use isolated mode (dedicated controllers) in production. Combined mode is for development and small setups.
- Partitions and their leaders are spread across brokers. Preferred leader election keeps leadership balanced.
- Adding a broker does not move existing data. You must reassign partitions.
- Use at least 3 brokers and an odd number of controllers (3 or 5), spread across failure domains.
Practice
- (Beginner) Explain the difference between
process.roles=broker,process.roles=controller, andprocess.roles=broker,controller, and say where each would be used.- (Intermediate) A cluster has 3 brokers and you add a fourth. A week later, the new broker has almost no disk usage. Explain why, and outline what you would do about it.
- Interview question: "Why do production KRaft clusters usually run controllers on separate nodes from brokers?" Hint: think about what happens to metadata operations, such as leader elections, when the node hosting the active controller is overloaded with client traffic.
Controller Broker
Theory
A cluster needs one place where decisions about the whole cluster are made. Who leads partition 7 now that broker 3 has died? Is broker 5 alive? Where should the replicas of a new topic go? If every broker made these decisions on its own, they would disagree. Kafka solves this by having exactly one active controller at any time.
Analogy: the controller is like an air traffic control tower. Planes (brokers) fly on their own and carry the passengers (data), but the tower alone decides who uses which runway (who leads which partition). If the tower goes dark, a standby tower must take over immediately, and planes must never take instructions from two towers at once.
What the active controller does
| Task | Description |
|---|---|
| Broker liveness | Tracks broker registrations and heartbeats. Fences brokers that stop heartbeating. |
| Leader election | Chooses new partition leaders when a leader fails, shuts down, or must move (Section 2.4) |
| ISR updates | Accepts and commits changes to in-sync replica sets requested by partition leaders |
| Topic lifecycle | Creates and deletes topics, assigns replicas to brokers, adds partitions |
| Reassignment | Coordinates moving replicas between brokers |
| Configuration, ACLs, quotas, features | Stores and propagates dynamic configuration and security metadata |
The controller is not in the data path. Producers and consumers never send records to it. It manages metadata only, which is why a small controller quorum can manage a large cluster.
The controller over time
| Aspect | ZooKeeper Era (before 4.0) | KRaft Era (3.3+ and the only option in 4.x) |
|---|---|---|
| Who is controller | One regular broker that won a race to create the /controller znode | The leader of the Raft quorum of controller nodes |
| Where metadata lives | ZooKeeper, with a cache in the controller's memory | A replicated metadata log (__cluster_metadata) on the controllers |
| Failover | New controller must read all metadata from ZooKeeper before acting (seconds to minutes on large clusters) | Standby controllers already hold the metadata log in memory, so failover takes about the election time |
| Pushing changes to brokers | Controller sends RPCs (LeaderAndIsr, UpdateMetadata) to each broker | Brokers pull the metadata log and apply changes incrementally |
| Fencing a stale controller | controller epoch stored in ZooKeeper | Raft leader epoch |
Fencing: preventing two controllers
A controller can pause (for example, during a long GC pause), lose its role, and then wake up still believing it is in charge: a zombie controller. Kafka prevents its decisions from taking effect with a monotonically increasing epoch. Every new controller term has a higher epoch, and requests stamped with an older epoch are rejected. The same idea appears again with leader epochs (Section 2.4) and producer epochs (Chapter 8).
Broker heartbeats and fencing (KRaft)
Each broker registers with the controller at startup and then sends a heartbeat every broker.heartbeat.interval.ms (default 2 seconds). If no heartbeat arrives within broker.session.timeout.ms (default 9 seconds), the controller fences the broker: it removes the broker from ISRs and moves leadership of its partitions elsewhere. A fenced broker must catch up on metadata before it is unfenced and allowed to lead again.
broker 3 active controller (ctrl 1) metadata log
| heartbeat (t=0s) | |
|---------------------->| |
| heartbeat (t=2s) | |
|---------------------->| |
X broker 3 crashes | |
| no heartbeat for 9s |
|-- FenceBrokerRecord(3) -------------->|
|-- PartitionChangeRecord(P0: leader 1->2, ISR [2,1]) ->|
| |
all brokers fetch the metadata log and learn the new leaders
Monitoring the controller
kafka.controller:type=KafkaController,name=ActiveControllerCount: summed across the cluster, this must be exactly 1. Zero means no controller (metadata changes stall). More than one indicates a serious problem.kafka.controller:type=KafkaController,name=OfflinePartitionsCount: partitions with no leader. Should be 0.- Controller event queue time and metadata error rates (Chapter 12).
Example: finding the active controller
bin/kafka-metadata-quorum.sh --bootstrap-server broker4:9092 describe --status
ClusterId: MkU3OEVBNTcwNTJENDM2Qk
LeaderId: 1
LeaderEpoch: 14
HighWatermark: 1873406
MaxFollowerLag: 0
MaxFollowerLagTimeMs: 0
CurrentVoters: [1,2,3]
CurrentObservers: [4,5,6,7]
(Output abridged; recent versions also print endpoints and directory IDs.) LeaderId is the active controller, LeaderEpoch increases on every controller change, CurrentVoters are the controllers, and CurrentObservers are the brokers that replicate the metadata log without voting.
In KRaft clients cannot reach the controllers directly, so the "controller" a client sees in a Metadata response is a randomly chosen live broker, which forwards admin requests to the real controller. Do not use a client's reported controller ID to find the active controller. Use the quorum tool or the ActiveControllerCount metric instead.
Key Takeaways
- Exactly one active controller manages cluster metadata: broker liveness, leader election, ISR changes, topics, configurations, and reassignments.
- The controller is not in the data path. Clients send records to partition leaders, not to the controller.
- In KRaft the active controller is the Raft leader. Standbys already hold the metadata, so failover is fast.
- Epochs fence zombie controllers. Heartbeats and session timeouts fence dead or stuck brokers.
- Watch
ActiveControllerCount(must sum to 1) andOfflinePartitionsCount(must be 0).
Practice
- (Beginner) Name three actions that only the active controller performs and one action it never performs.
- (Intermediate) In ZooKeeper mode, controller failover on a cluster with 200,000 partitions could take minutes. Explain what the new controller had to do during that time and why KRaft avoids this.
- Interview question: "What is a zombie controller, and how does Kafka stop it from corrupting cluster state?" Hint: think about a monotonically increasing number attached to every decision.
Broker Discovery and Bootstrap Servers
Theory
A client needs to send each record to the leader of its partition, and leaders move around. Hard-coding every broker's address would break whenever the cluster changes. Instead, a client is given a few bootstrap servers to start from and discovers the rest of the cluster by asking for metadata.
Analogy: arriving at a large office building, you do not need a directory of every employee's desk. You ask the receptionist (any bootstrap server), who hands you a floor map (metadata) showing where everyone sits. If someone changes desks, you ask again.
The discovery flow
Client bootstrap broker (any) partition leaders
| | |
| 1. TCP connect to one entry in | |
| bootstrap.servers | |
|----------------------------------->| |
| 2. MetadataRequest(topics=[orders])| |
|----------------------------------->| |
| 3. MetadataResponse: | |
| brokers = {1: b1:9092, 2: b2:9092, 3: b3:9092} |
| orders-0 leader=1, orders-1 leader=2, orders-2 leader=3 |
|<-----------------------------------| |
| 4. Produce/Fetch go DIRECTLY to each partition's leader |
|-------------------------------------------------------------------------->|
| 5. On NOT_LEADER_OR_FOLLOWER (leader moved) or every metadata.max.age.ms: |
| refresh metadata and retry |
Key points:
- Any broker can answer a metadata request, because every broker holds a copy of cluster metadata.
- The bootstrap list is only for the first contact. After that, the client uses the addresses returned in metadata.
- List several bootstrap servers (two or three is enough). If you list just one and it is down when the application starts, the client cannot connect at all. If the list is a single DNS name that resolves to several brokers, the default
client.dns.lookup=use_all_dns_ipstries each address. - Clients refresh metadata periodically (
metadata.max.age.ms, default 5 minutes) and immediately when a request fails with a leadership error. - Recent clients (3.8+) can re-run the bootstrap step if every known broker becomes unreachable (
metadata.recovery.strategy=rebootstrap), which helps when an entire cluster is replaced behind the same DNS name.
The advertised listener trap
The addresses in the metadata response come from each broker's advertised.listeners, not from the address the client used to connect. This is the single most common Kafka connectivity problem:
Client on a laptop Broker in Docker
bootstrap.servers=localhost:9092 ---> listens on 0.0.0.0:9092
advertised.listeners=PLAINTEXT://kafka:9092
<--- Metadata: "broker 1 is at kafka:9092"
connect to kafka:9092 ... FAILS (hostname "kafka" only resolves inside Docker)
The bootstrap connection succeeds, but every produce or fetch fails or hangs because the client cannot resolve or reach the advertised address. The fix is to advertise an address the client can actually reach, often by using separate listeners for internal and external clients:
# Broker inside Docker, reachable both from other containers and from the host
listeners=INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=INTERNAL://kafka:29092,EXTERNAL://localhost:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
inter.broker.listener.name=INTERNAL
controller.listener.names=CONTROLLER
# Containers use kafka:29092; applications on the host use localhost:9092
Example: seeing what a client discovers
from confluent_kafka.admin import AdminClient
admin = AdminClient({"bootstrap.servers": "broker1:9092,broker2:9092,broker3:9092"})
md = admin.list_topics(timeout=10) # sends a MetadataRequest to one bootstrap server
print("cluster id:", md.cluster_id)
for b in sorted(md.brokers.values(), key=lambda b: b.id):
# These are the ADVERTISED addresses the client will actually connect to
print(f"broker {b.id}: {b.host}:{b.port}")
for p in sorted(md.topics["orders"].partitions.values(), key=lambda p: p.id):
print(f"orders-{p.id}: leader={p.leader} replicas={p.replicas} isr={p.isrs}")
cluster id: MkU3OEVBNTcwNTJENDM2Qk
broker 1: broker1.kafka.internal:9092
broker 2: broker2.kafka.internal:9092
broker 3: broker3.kafka.internal:9092
orders-0: leader=1 replicas=[1, 2, 3] isr=[1, 2, 3]
orders-1: leader=2 replicas=[2, 3, 1] isr=[2, 3, 1]
orders-2: leader=3 replicas=[3, 1, 2] isr=[3, 1, 2]
If the printed host names are not reachable from where the client runs, you have found an advertised.listeners problem.
Key Takeaways
bootstrap.serversis only for the first connection. The client learns the full cluster layout from metadata.- Any broker can answer metadata requests. List two or three bootstrap servers for resilience.
- Clients connect directly to partition leaders and refresh metadata when leadership changes.
- Clients connect to the addresses in
advertised.listeners. Misconfigured advertised addresses are the most common cause of "connects but cannot produce" problems. - Use separate listeners for networks that need different addresses (inside Docker or Kubernetes versus outside).
Practice
- (Beginner) A cluster has 12 brokers. Does the client's
bootstrap.serversneed to list all 12? Explain what happens if it lists only one and that broker is down at startup.- (Intermediate) A Kubernetes-hosted broker advertises
kafka-0.kafka-headless.default.svc:9092. An application outside the cluster can bootstrap through a load balancer but every produce times out. Explain why and sketch a listener configuration that fixes it.- Interview question: "Walk me through what a Kafka producer does between
new KafkaProducer(props)and the first record landing on the right broker." Hint: bootstrap, metadata request, partitioner, leader lookup, direct connection, and what triggers a metadata refresh.
Rack Awareness
Theory
Replication protects you from losing a broker, but only if the replicas are not all in the same place. If all three replicas of a partition sit in one rack, one top-of-rack switch failure or one power outage takes all of them down at once. Rack awareness tells Kafka which failure domain each broker belongs to, so it can spread replicas across them. In the cloud, a "rack" is usually an availability zone (AZ).
Analogy: you would not keep your house keys, the spare set, and the neighbor's copy all in the same coat pocket. Rack awareness makes sure the copies are in different pockets.
Configuring racks
Set broker.rack on every broker:
# Brokers 1 and 4
broker.rack=eu-west-1a
# Brokers 2 and 5
broker.rack=eu-west-1b
# Brokers 3 and 6
broker.rack=eu-west-1c
When topics are created or partitions are added, the replica assignment algorithm places each partition's replicas in as many different racks as possible:
AZ eu-west-1a AZ eu-west-1b AZ eu-west-1c
+---------------+ +---------------+ +---------------+
| B1 B4 | | B2 B5 | | B3 B6 |
P0 | L | | f | | f |
P1 | f | | L | | f |
P2 | f | | f | | L |
+---------------+ +---------------+ +---------------+
Every partition has exactly one replica per AZ: losing a whole AZ leaves 2 of 3 replicas.
Rack-aware placement applies when replicas are assigned (topic creation, adding partitions, reassignment). Setting broker.rack on an existing cluster does not move existing replicas, so you must reassign them. Keep the number of brokers per rack equal: with uneven racks, the brokers in the smaller rack end up hosting more replicas than the others.
Follower fetching: reading from the closest replica (KIP-392)
By default, consumers fetch from the partition leader, which may be in another AZ. Cloud providers charge for cross-AZ traffic, and for a high-throughput topic with many consumer groups this can become a large part of the bill. Since Kafka 2.4, consumers can read from a follower in their own rack:
# On every broker
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
# On each consumer: the AZ the consumer runs in
client.rack=eu-west-1b
Without follower fetching With follower fetching
consumer (1b) --cross-AZ--> leader (1a) consumer (1b) --same AZ--> follower (1b)
(leader still handles all writes)
Trade-offs of follower fetching:
| Benefit | Cost |
|---|---|
| Lower cross-AZ network cost | Slightly higher end-to-end latency: followers only expose data once the high watermark advances, and they learn the new high watermark one fetch later |
| Load spread across more brokers | Requires accurate client.rack on every consumer (often injected from cloud instance metadata) |
Consumer groups can also take racks into account when assigning partitions, so that consumers get partitions with a replica in their own rack (KIP-881). Producers always write to the leader, so rack awareness does not reduce produce-side cross-AZ traffic.
Controllers and racks
Spread the controller quorum across failure domains too: with 3 controllers in 3 AZs, losing one AZ leaves a 2-of-3 majority. Putting 2 of 3 controllers in the same AZ means losing that AZ stops metadata operations.
Example: verifying rack placement
try (Admin admin = Admin.create(Map.of("bootstrap.servers", "broker1:9092"))) {
// Map broker id -> rack
Map<Integer, String> rackOf = admin.describeCluster().nodes().get().stream()
.collect(Collectors.toMap(Node::id, n -> n.hasRack() ? n.rack() : "none"));
TopicDescription d = admin.describeTopics(List.of("orders")).allTopicNames().get().get("orders");
for (TopicPartitionInfo p : d.partitions()) {
Set<String> racks = p.replicas().stream().map(n -> rackOf.get(n.id())).collect(Collectors.toSet());
boolean ok = racks.size() == p.replicas().size(); // one replica per distinct rack?
System.out.printf("orders-%d replicas=%s racks=%s %s%n",
p.partition(), p.replicas().stream().map(Node::id).toList(), racks, ok ? "OK" : "SHARED RACK");
}
}
Key Takeaways
broker.racktells Kafka each broker's failure domain (rack or AZ), so it spreads a partition's replicas across domains.- Placement is applied when replicas are assigned. Existing topics need reassignment after racks are added or changed.
- Keep brokers balanced across racks to avoid hot spots.
- Follower fetching (
RackAwareReplicaSelector+client.rack) lets consumers read from a same-AZ replica, which cuts cross-AZ costs at a small latency cost. - Spread controllers across failure domains so that losing one domain does not cost the quorum its majority.
Practice
- (Beginner) A cluster has 6 brokers in 3 AZs and topics use replication factor 3. How many AZs can fail before a partition loses all its replicas, assuming rack-aware placement?
- (Intermediate) A topic produces 50 MB/s and is read by 4 consumer groups, with consumers evenly spread across 3 AZs. Estimate the cross-AZ consumer traffic per second with and without follower fetching. (Assume leaders are evenly spread across AZs.)
- Interview question: "Your cloud bill shows very high inter-AZ data transfer from Kafka. What would you investigate and change?" Hint: separate replication traffic (unavoidable with replicas in different AZs) from consumer and producer traffic, and consider which of them can be kept local.
<a id="23-cluster-metadata-management"></a>
2.3 Cluster Metadata Management
Every broker must agree on which topics exist, where their replicas live, who leads each partition, and which brokers are alive. This section explains how Kafka stores and agrees on that metadata, first with ZooKeeper and now with KRaft, and how existing clusters move from one to the other.
ZooKeeper-Based Architecture
Theory
From its first release until version 3.x, Kafka did not store its own metadata. It delegated that job to Apache ZooKeeper, a separate distributed coordination service that offers a small, strongly consistent, tree-shaped key-value store. Kafka 4.0 (March 2025) removed ZooKeeper support entirely, but you still need to understand the old design: many production clusters ran on it for years, much of Kafka's documentation and tooling history refers to it, and migrating off it is a real operational task (see Migrating from ZooKeeper to KRaft).
Analogy: ZooKeeper was like an external notary office. Kafka brokers did their own work, but every important fact (who is alive, who leads what) had to be registered with the notary, and the notary's records were the official truth.
What ZooKeeper provided
ZooKeeper stores data in znodes arranged like a file system. Two features made it useful for Kafka:
- Ephemeral znodes: they exist only while the session that created them is alive. A broker that crashes loses its session, so its znode disappears automatically. This gave Kafka failure detection.
- Watches: a client can ask to be notified when a znode changes. The controller watched broker registrations and admin requests.
ZooKeeper tree used by Kafka (under an optional chroot such as /kafka)
/
|-- cluster/id cluster ID
|-- controller EPHEMERAL: {"brokerid":1} <- whoever creates it first is controller
|-- controller_epoch 17 <- fencing for zombie controllers
|-- brokers/
| |-- ids/1, ids/2, ids/3 EPHEMERAL: one per live broker (host, ports, rack)
| `-- topics/orders replica assignment {"0":[1,2,3],"1":[2,3,1],...}
| `-- partitions/0/state {"leader":1,"leader_epoch":5,"isr":[1,2,3]}
|-- config/
| |-- topics/orders topic-level config overrides
| `-- brokers/, users/, clients/ dynamic broker configs, quotas
|-- admin/
| |-- delete_topics/ pending topic deletions
| `-- reassign_partitions in-progress reassignment
`-- kafka-acl/ ACLs (ZooKeeper-based authorizer)
How the pieces worked together
- Each broker, on startup, created an ephemeral znode under
/brokers/ids/. - All brokers raced to create the ephemeral
/controllerznode. The winner became the controller. The others set a watch on it, so they could race again if it disappeared. - The controller watched
/brokers/ids. When a broker's znode disappeared, the controller chose new leaders for that broker's partitions, wrote the new state to ZooKeeper, and then sentLeaderAndIsrandUpdateMetadataRPCs to every affected broker. - Partition leaders wrote ISR changes directly to ZooKeeper, and the controller propagated them.
+-------------------------------+
| ZooKeeper ensemble |
| zk1 zk2 zk3 | source of truth for metadata
+-------------------------------+
^ watches / writes ^ ephemeral registrations, ISR writes
| |
+--------------+ +--------------+ +--------------+
| Broker 1 | | Broker 2 | | Broker 3 |
| (CONTROLLER) |------->| | | |
+--------------+ RPCs +--------------+ +--------------+
| LeaderAndIsr, UpdateMetadata ^
+-------------------------------------------+
Why Kafka moved away from ZooKeeper
| Problem | Explanation |
|---|---|
| Two distributed systems to run | Separate deployment, configuration, security (SASL, TLS, ACLs), monitoring, and upgrades for ZooKeeper |
| Slow controller failover | A new controller had to read the full state of every partition from ZooKeeper before acting. On large clusters this took tens of seconds to minutes, during which leader elections stalled. |
| Scalability ceiling | Metadata had to fit ZooKeeper's model and the controller's reload time. Practical limits were around 200,000 partitions per cluster. |
| Divergent views | Metadata lived in ZooKeeper, in the controller's memory, and in each broker's cache, updated by separate RPCs. These copies could drift apart, causing subtle bugs. |
| Inefficient propagation | Changes were pushed to brokers as full or partial snapshots, not as an ordered log of deltas. |
The insight behind the replacement (KIP-500) was: Kafka is already a replicated log system, so it should store its own metadata in a replicated log.
Example: what operating a ZooKeeper-based cluster looked like (historical, 3.x and earlier)
# server.properties of a ZooKeeper-mode broker
broker.id=1
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka # "/kafka" is a chroot path
zookeeper.session.timeout.ms=18000
listeners=PLAINTEXT://0.0.0.0:9092
log.dirs=/var/lib/kafka/data
# Inspecting cluster state directly in ZooKeeper
bin/zookeeper-shell.sh zk1:2181/kafka get /controller
# {"version":2,"brokerid":1,"timestamp":"1761210942120"}
bin/zookeeper-shell.sh zk1:2181/kafka ls /brokers/ids
# [1, 2, 3]
bin/zookeeper-shell.sh zk1:2181/kafka get /brokers/topics/orders/partitions/0/state
# {"controller_epoch":17,"leader":1,"version":1,"leader_epoch":5,"isr":[1,2,3]}
Older CLI tools even accepted --zookeeper instead of --bootstrap-server, letting them bypass the brokers. Those options were removed during the 3.x line, and none of this works in Kafka 4.x.
Key Takeaways
- Up to Kafka 3.x, ZooKeeper held all cluster metadata. Kafka 4.0 removed ZooKeeper support.
- Ephemeral znodes detected broker failures. A race on the
/controllerznode chose the controller. - The controller read state from ZooKeeper and pushed changes to brokers with RPCs.
- Its main problems were operational complexity, slow controller failover, limited partition scalability, and inconsistent copies of metadata.
- KRaft replaced it by storing metadata in a Kafka-native replicated log.
Practice
- (Beginner) Explain how ZooKeeper's ephemeral znodes allowed Kafka to detect a crashed broker.
- (Intermediate) Describe the sequence of events, from broker crash to clients producing again, in ZooKeeper mode. Identify the step that grew slower as the number of partitions grew.
- Interview question: "Why did Kafka remove its dependency on ZooKeeper?" Hint: give at least one operational reason, one performance reason, and one correctness reason.
KRaft Mode
Theory
KRaft (Kafka Raft, introduced by KIP-500) moves metadata management inside Kafka. A small set of controller nodes form a Raft quorum and store every metadata change as a record in an internal, single-partition log called __cluster_metadata. The leader of that quorum is the active controller. Brokers replicate this log too, and build their view of the cluster by applying its records in order.
Analogy: instead of registering facts with an outside notary (ZooKeeper), the company now keeps its own official ledger, maintained by a small board (the controllers). Every branch office (broker) keeps a copy of the ledger and stays current by reading new entries as they are written.
Milestones
| Version | Date | Milestone |
|---|---|---|
| 2.8 | April 2021 | KRaft available as early access |
| 3.3 | October 2022 | KRaft production-ready for new clusters |
| 3.4 | February 2023 | ZooKeeper-to-KRaft migration available as early access |
| 3.5 | June 2023 | ZooKeeper mode deprecated |
| 3.6 | October 2023 | Migration production-ready |
| 3.9 | November 2024 | Last 3.x release and the final "bridge" release for migration. Dynamic controller quorums (KIP-853). |
| 4.0 | March 2025 | ZooKeeper removed. KRaft is the only mode. |
Metadata as an event log
Every change to the cluster is a typed record appended to __cluster_metadata:
| Record Type | Written When |
|---|---|
RegisterBrokerRecord | A broker starts and registers |
FenceBrokerRecord / UnfenceBrokerRecord | A broker stops heartbeating, or catches up and becomes eligible |
TopicRecord | A topic is created |
PartitionRecord | A partition is created (replicas, leader, ISR) |
PartitionChangeRecord | Leader, ISR, or replicas of a partition change |
ConfigRecord | A dynamic configuration is set |
FeatureLevelRecord | A feature such as metadata.version is upgraded |
RemoveTopicRecord | A topic is deleted |
This is the same idea as event sourcing (Chapter 15): the current metadata is the result of replaying the log. Instead of pushing state with RPCs, the controller just appends records, and every broker pulls new records with Fetch requests, like a consumer. Consequences:
- Consistency: every node applies the same records in the same order, so all views converge. A broker knows exactly how up to date it is (by metadata offset).
- Fast failover: standby controllers are already replicating the log and hold the full metadata in memory. A new active controller can act as soon as it is elected.
- Incremental propagation: brokers receive small deltas instead of full state snapshots.
- Scale: metadata operations no longer slow down proportionally to the total partition count, which raises practical cluster limits to millions of partitions.
Controller quorum (voters) Brokers (observers)
+-----------+ +-----------+ +-----------+
| ctrl 1 | | ctrl 2 | | ctrl 3 | +----------+ +----------+
| ACTIVE | | standby | | standby | | broker 4 | | broker 5 |
| (leader) | | (follower)| | (follower)| | | | |
+-----------+ +-----------+ +-----------+ +----------+ +----------+
| __cluster_metadata (single partition) ^ ^
| offset ... 812 813 814 815 | Fetch |
+---> followers Fetch from the leader +-------------+
brokers Fetch the same log and
apply records to an in-memory cache
Snapshots
An ever-growing metadata log would make restarts slow. Controllers and brokers therefore write snapshots of their in-memory metadata periodically (controlled by metadata.log.max.record.bytes.between.snapshots and metadata.log.max.snapshot.interval.ms). A restarting node loads the latest snapshot and replays only the records after it. A node that has fallen too far behind downloads the snapshot from the leader.
Feature versioning
A KRaft cluster tracks a metadata.version feature level, which controls the metadata record formats and protocol features in use. After upgrading the software on every node, you upgrade the feature level explicitly:
bin/kafka-features.sh --bootstrap-server broker4:9092 describe
bin/kafka-features.sh --bootstrap-server broker4:9092 upgrade --release-version 4.1
This replaces the older inter.broker.protocol.version setting from ZooKeeper mode and lets you finish a rolling upgrade before turning on new features.
Example: starting a KRaft cluster
Single-node development setup (combined mode, as in the official quickstart):
# 1. Generate a cluster ID once
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
# 2. Format the storage directory: writes meta.properties (cluster ID, node ID)
# and bootstraps a one-voter controller quorum
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties
# 3. Start the node (process.roles=broker,controller in this file)
bin/kafka-server-start.sh config/server.properties
Production setup (isolated mode, static quorum shown for clarity):
# controller.properties (on ctrl1; ctrl2 and ctrl3 differ only in node.id and listener host)
process.roles=controller
node.id=1
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093 # fixed voter set
listeners=CONTROLLER://ctrl1:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT
log.dirs=/var/lib/kafka/meta # holds __cluster_metadata
# broker.properties (on broker4)
process.roles=broker
node.id=4 # must not clash with controller IDs
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://broker4:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
log.dirs=/var/lib/kafka/data
# Every node (controllers and brokers) is formatted with the SAME cluster ID
bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c controller.properties # on each controller
bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c broker.properties # on each broker
With a dynamic quorum (3.9+), you instead set controller.quorum.bootstrap.servers and format the controllers with --initial-controllers (or --standalone for the first one), which allows adding and removing controllers later (see Metadata Quorum and Raft Consensus).
Inspecting the metadata log directly:
bin/kafka-dump-log.sh --cluster-metadata-decoder \
--files /var/lib/kafka/meta/__cluster_metadata-0/00000000000000000000.log
| offset: 812 ... payload: {"type":"TOPIC_RECORD","version":0,"data":{"name":"orders","topicId":"q8bGv1TfRXSX0n3b1n7Y2w"}}
| offset: 813 ... payload: {"type":"PARTITION_RECORD","version":0,"data":{"partitionId":0,"topicId":"q8bG...","replicas":[4,5,6],"isr":[4,5,6],"leader":4,"leaderEpoch":0}}
| offset: 940 ... payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":0,"topicId":"q8bG...","isr":[5,6],"leader":5}}
(Output abridged.) Offset 940 records broker 4 losing leadership of orders-0: a single small delta, not a full state rewrite.
Key Takeaways
- KRaft stores cluster metadata in an internal Raft-replicated log,
__cluster_metadata, managed by controller nodes. The Raft leader is the active controller. - Brokers replicate the metadata log as observers and apply its records to an in-memory cache, so all nodes converge on the same state.
- Failover is fast because standby controllers already hold the metadata. Propagation is incremental and scales to far more partitions.
- Snapshots bound restart and catch-up time.
metadata.versiongates new features after an upgrade. - Every node must be formatted with
kafka-storage.shusing the same cluster ID before first start.
Practice
- (Beginner) What is stored in
__cluster_metadata, and how do brokers keep their copy up to date?- (Intermediate) Set up a three-node combined-mode KRaft cluster locally (three configuration files, different ports and
node.ids). Usekafka-metadata-quorum.sh describe --statusto find the active controller, stop it, and observe the newLeaderIdandLeaderEpoch.- Interview question: "Compare how a leader change is propagated to brokers in ZooKeeper mode and in KRaft mode." Hint: push RPCs with full or partial state versus pull of an ordered log of deltas; consider what happens to a broker that misses an update in each model.
Metadata Quorum and Raft Consensus
Theory
The controllers must agree on a single, ordered metadata log even when some of them crash or the network splits. This is the classic problem of distributed consensus. KRaft solves it with a variant of the Raft algorithm. You do not need to implement Raft to operate Kafka, but you do need its core ideas to size the quorum and understand failure behavior.
Analogy: a committee that keeps the official minutes. One member is the chair (leader) and is the only one allowed to write new entries. An entry becomes official once a majority of members have copied it into their own notebooks. If the chair disappears, the members vote for a new chair, and only someone whose notebook is fully up to date can win.
Raft in five ideas
- Roles: each voter is a leader, a follower, or a candidate. KRaft adds observers (brokers), which replicate the log but never vote.
- Epochs (terms): time is divided into numbered epochs, each with at most one leader. A higher epoch always wins, which fences old leaders.
- Leader election: if a follower hears nothing from the leader for a while, it becomes a candidate, increments the epoch, and asks for votes. A voter grants its vote only if the candidate's log is at least as up to date as its own. A candidate with votes from a majority becomes leader.
- Log replication: only the leader appends. Followers copy its log in order.
- Commitment: an entry is committed once a majority of voters have it. Committed entries are never lost, because any future leader must have them (it needed a majority's votes, and any two majorities overlap).
How KRaft differs from textbook Raft
| Aspect | Textbook Raft | KRaft |
|---|---|---|
| Replication | Leader pushes entries with AppendEntries | Followers and observers pull with Fetch, reusing Kafka's replication machinery |
| Leader liveness | Leader sends heartbeats | Followers' regular fetches double as liveness signals. A follower that cannot fetch within controller.quorum.fetch.timeout.ms starts an election. |
| Commit point | Commit index | The metadata log's high watermark |
| Non-voting replicas | Learners (an extension) | Observers: every broker |
| Divergent logs | Leader overwrites follower entries | Leader returns a "diverging epoch" in the fetch response, and the follower truncates to the right point |
Newer versions also add a pre-vote phase (KIP-996): a node first checks whether it could win before starting a real election, which prevents a node with a flaky network link from repeatedly bumping the epoch and disrupting a healthy leader.
Committing a metadata record
Voters: ctrl1 (leader), ctrl2, ctrl3 majority = 2
1. Admin creates a topic -> ctrl1 appends TopicRecord at offset 815 ctrl1 LEO=816
2. ctrl2 Fetch(offset=815) -> receives 815 ctrl2 LEO=816
3. ctrl3 is slow, still at LEO=815
4. ctrl2's next Fetch(offset=816) tells ctrl1 "I have up to 815"
-> 2 of 3 voters have 815 -> high watermark advances to 816 COMMITTED
5. Only now does ctrl1 answer the CreateTopics request successfully,
and brokers (observers) apply the record once they see HW >= 816
Sizing the quorum
| Voters | Majority Needed | Failures Tolerated | Notes |
|---|---|---|---|
| 1 | 1 | 0 | Development only |
| 2 | 2 | 0 | Worse than 1: two machines that can fail, and no extra tolerance |
| 3 | 2 | 1 | Standard choice |
| 4 | 3 | 1 | No benefit over 3, more replication work |
| 5 | 3 | 2 | Large or critical clusters, or to survive one failure during maintenance |
Use an odd number. Adding an even-numbered voter raises the majority without raising tolerance. More voters also mean every commit waits for more acknowledgments, so 3 or 5 is almost always right.
What happens when the quorum loses its majority
If a majority of controllers is unavailable, no new leader can be elected and no metadata record can be committed. Brokers keep serving reads and writes for partitions whose leaders are healthy, using their last known metadata, but nothing that needs the controller works: no leader elections after broker failures, no topic creation, no ISR changes, no configuration updates. This is why controllers must be spread across failure domains.
Dynamic quorum membership (KIP-853, Kafka 3.9+)
Originally the voter set was fixed by controller.quorum.voters, and changing it required downtime. With dynamic quorums, nodes find the quorum through controller.quorum.bootstrap.servers, and voters can be added or removed online:
# On a new, formatted controller that has started and caught up as an observer:
bin/kafka-metadata-quorum.sh --command-config new-controller.properties \
--bootstrap-controller ctrl1:9093 add-controller
# Remove a voter (identified by node ID and its storage directory ID)
bin/kafka-metadata-quorum.sh --bootstrap-controller ctrl1:9093 \
remove-controller --controller-id 3 --controller-directory-id <directory-uuid>
The directory ID identifies a specific disk, so a controller whose disk was wiped and re-formatted cannot silently rejoin as if it still had its old data, which would be unsafe for Raft.
Example: observing replication in the quorum
bin/kafka-metadata-quorum.sh --bootstrap-server broker4:9092 describe --replication
NodeId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
1 1873406 0 1761210942120 1761210942120 Leader
2 1873406 0 1761210942051 1761210942051 Follower
3 1873390 16 1761210941830 1761210941200 Follower
4 1873406 0 1761210942090 1761210942090 Observer
5 1873406 0 1761210942077 1761210942077 Observer
(Output abridged.) Node 3 is lagging slightly. If its lag grows steadily, that controller may have disk or network problems.
A small Python model of the commit rule:
def high_watermark(leader_leo: int, follower_leos: list[int]) -> int:
"""The highest offset present on a majority of voters (exclusive end, like Kafka's HW)."""
leos = sorted([leader_leo, *follower_leos], reverse=True)
majority = len(leos) // 2 + 1
return leos[majority - 1] # the majority-th largest LEO is held by >= majority voters
print(high_watermark(816, [816, 815])) # 3 voters -> 816 (ctrl1 and ctrl2 have it)
print(high_watermark(820, [816, 815, 812, 810])) # 5 voters -> 815
print(high_watermark(820, [815, 815])) # leader ahead alone -> 815, not 820
Key Takeaways
- KRaft controllers run a Raft variant: one leader per epoch, majority-based election and commitment, with followers pulling via
Fetch. - A metadata change takes effect only after a majority of voters have it (the high watermark advances).
- Use 3 or 5 voters. An odd number gives the best fault tolerance per node. Spread them across failure domains.
- Without a majority, data traffic on healthy partitions continues, but all metadata changes, including failover, stop.
- Dynamic quorums (3.9+) allow adding and removing controllers online. Directory IDs protect against unsafe rejoining with wiped disks.
Practice
- (Beginner) A quorum has 5 voters. How many can fail while the cluster still elects partition leaders? What if it has 4 voters?
- (Intermediate) Two of three controllers are in AZ
aand one is in AZb. AZafails. Describe what still works and what stops working for producers, consumers, and administrators. Propose a better placement.- Interview question: "Why can a Raft-based system never lose a committed entry, even if the leader crashes right after committing?" Hint: combine two facts: an entry is committed on a majority, and a candidate needs votes from a majority whose logs are not more up to date than its own.
Migrating from ZooKeeper to KRaft
Theory
Clusters created before KRaft must be moved to it before they can run Kafka 4.x, because 4.0 and later cannot start in ZooKeeper mode or perform the migration. The migration is done online, without downtime for clients, using a 3.x bridge release (3.9 is the last and recommended one). After migrating on 3.9, you upgrade to 4.x as a normal rolling upgrade.
Analogy: moving a shop's records from a filing cabinet to a new computer system while the shop stays open. First the new system copies everything from the cabinet. For a while, every change is written to both, so you can still go back. Once everyone uses the new system and you are confident, you stop updating the cabinet and finally remove it.
Prerequisites
- All brokers run the bridge release (for example 3.9.x) with
inter.broker.protocol.versionset to that version. - You know the existing cluster ID (stored in ZooKeeper at
/cluster/id). - Your configuration uses only features supported by migration in that release (check the release's migration documentation, for example for JBOD or specific security setups).
- Monitoring is in place for both the old and the new metadata systems, and you have rehearsed in a staging cluster.
The migration phases
Phase 0 ZooKeeper mode
[ZooKeeper] <-> brokers (one broker is the ZK controller)
Phase 1 Deploy KRaft controllers with migration enabled
[ZooKeeper] [ctrl quorum: waiting for brokers]
^
brokers (still ZK mode)
Phase 2 Enable migration on brokers (rolling restart, still ZK mode)
-> a KRaft controller becomes the active controller
-> it copies all metadata from ZooKeeper into __cluster_metadata
-> DUAL-WRITE: every metadata change is written to the KRaft log AND to ZooKeeper
[ZooKeeper] <-dual-write- [ctrl quorum ACTIVE] --RPCs--> brokers (ZK mode)
Phase 3 Restart brokers as KRaft brokers (rolling restart)
brokers now fetch __cluster_metadata; dual-write to ZooKeeper continues
[ZooKeeper] <-dual-write- [ctrl quorum] <--Fetch-- brokers (KRaft mode)
Phase 4 Finalize: restart controllers without ZooKeeper configuration
ZooKeeper no longer updated -> decommission it POINT OF NO RETURN
[ctrl quorum] <--Fetch-- brokers (KRaft mode)
Until Phase 4, ZooKeeper still receives every metadata change, so you can roll back to ZooKeeper mode by following the documented reverse procedure. After finalization, rolling back is no longer possible.
Monitoring progress
- The controller logs
Completed migration of metadata from ZooKeeper to KRaftwhen the Phase 2 copy finishes. - The metric
kafka.controller:type=KafkaController,name=ZkMigrationStateshows the current migration state. - Check that
ActiveControllerCountis 1 on the KRaft quorum, and that there are no offline or under-replicated partitions after each rolling restart.
Alternative: migrate by building a new cluster
Some teams avoid in-place migration altogether: they create a fresh KRaft cluster, replicate topics with MirrorMaker 2 or a similar tool (Chapter 14), move consumers and then producers over, and retire the old cluster. This costs more infrastructure and needs careful offset translation, but it is also a chance to fix old topic layouts and configurations.
| Approach | Advantages | Disadvantages |
|---|---|---|
| In-place migration | No data copying, same endpoints, same offsets, rollback until finalization | Several rolling restarts on a live cluster, must stay on 3.x until done |
| New cluster + replication | Clean slate, can upgrade straight to 4.x | Double infrastructure for a while, client cutover, offset translation |
Example: configuration in each phase (static quorum, 3.9)
Phase 1, controller configuration:
process.roles=controller
node.id=3000 # choose IDs that do not clash with broker.ids
controller.quorum.voters=3000@ctrl1:9093,3001@ctrl2:9093,3002@ctrl3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
# Migration
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka
# So the controller can send RPCs to brokers that are still in ZooKeeper mode
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
# Format controllers with the EXISTING cluster ID read from ZooKeeper (/cluster/id)
bin/kafka-storage.sh format -t "<existing-cluster-id>" -c controller.properties
Phase 2, broker configuration (still ZooKeeper mode, now aware of the controllers):
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker0:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
inter.broker.protocol.version=3.9
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka
zookeeper.metadata.migration.enable=true # added
controller.quorum.voters=3000@ctrl1:9093,3001@ctrl2:9093,3002@ctrl3:9093 # added
controller.listener.names=CONTROLLER # added
Phase 3, broker configuration (KRaft mode):
process.roles=broker # added
node.id=0 # same value as the old broker.id
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker0:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
controller.quorum.voters=3000@ctrl1:9093,3001@ctrl2:9093,3002@ctrl3:9093
controller.listener.names=CONTROLLER
# removed: broker.id, zookeeper.connect, zookeeper.metadata.migration.enable,
# inter.broker.protocol.version
Phase 4, controllers: remove zookeeper.metadata.migration.enable and zookeeper.connect, then restart each controller one at a time. After that, upgrade all nodes to 4.x and raise metadata.version with kafka-features.sh.
Key Takeaways
- Kafka 4.x cannot run or migrate ZooKeeper clusters. Migrate on a 3.x bridge release (3.9 recommended), then upgrade.
- The in-place migration is online: deploy migration-enabled controllers, enable migration on brokers, convert brokers to KRaft, then finalize.
- During migration, the KRaft controller dual-writes metadata to ZooKeeper, which keeps rollback possible until finalization.
- Keep the same cluster ID, and keep broker IDs as
node.ids. Controller IDs must not clash with broker IDs. - Building a new KRaft cluster and replicating data into it is a valid alternative with different trade-offs.
Practice
- (Beginner) Put these steps in order: restart brokers in KRaft mode, deploy controllers with migration enabled, remove ZooKeeper settings from controllers, enable migration on ZooKeeper-mode brokers.
- (Intermediate) During Phase 3, half of the brokers are in KRaft mode and half are still in ZooKeeper mode, and a problem appears. Explain why rollback is still possible at this point and what makes it impossible after Phase 4.
- Interview question: "You run a large Kafka 3.6 cluster on ZooKeeper and must reach Kafka 4.1. Outline your plan." Hint: version path, migration approach and its rollback window, verification at each step, and the final feature-version upgrade.
<a id="24-replication"></a>
2.4 Replication
Replication is what lets Kafka lose machines without losing data or availability. This section explains how partition copies are organized, how Kafka decides which copies are trustworthy, how it decides what is safe to show consumers, and how it picks a new leader when one fails.
Leader and Follower Replicas
Theory
Each partition is stored on several brokers. Each copy is a replica. If all replicas accepted writes independently, they would disagree about the order of records, and the core guarantee of an ordered log would break. Kafka avoids this with a simple rule: each partition has exactly one leader replica at a time, and all writes go through it.
- The leader handles all produce requests for the partition and, by default, all consumer fetches.
- Followers do not accept writes from clients. Their only job is to copy the leader's log, in the same order, so they are ready to take over.
Different partitions have leaders on different brokers, so every broker is the leader for some partitions and a follower for others, and load is spread evenly.
Analogy: a choir with one conductor per song. Everyone sings the same notes in the same order because only the conductor sets the tempo. For the next song, a different member may conduct. If the conductor faints, one of the singers who has been following closely steps up.
Replication is pull-based
Followers replicate by sending Fetch requests to the leader, the same request type consumers use. The fetch offset a follower sends doubles as an acknowledgment: "I have everything before this offset."
Partition orders-0, replication factor 3
Producer ---Produce---> +-----------------+ Consumer
| Broker 1 | <---Fetch--- (reads up to the high watermark)
| LEADER |
| 0 1 2 3 4 5 6 |
+-----------------+
^ ^
Fetch(offset=7) | | Fetch(offset=6)
| |
+-----------------+ +-----------------+
| Broker 2 | | Broker 3 |
| follower | | follower |
| 0 1 2 3 4 5 6 | | 0 1 2 3 4 5 | (one record behind)
+-----------------+ +-----------------+
Benefits of pull-based replication:
- Followers fetch at their own pace and receive large batches when they fall behind, so catching up is efficient.
- Replication reuses the log-reading code path, including zero-copy transfer.
- The leader learns each follower's progress from the fetch offsets, without separate acknowledgment messages.
Relevant broker settings: num.replica.fetchers (fetcher threads per source broker, default 1), replica.fetch.max.bytes, and replica.fetch.wait.max.ms (how long the leader holds a follower's fetch when there is no new data).
Leader epochs
Each time a partition gets a new leader, its leader epoch increases. The leader stamps the epoch into every record batch it writes (the partitionLeaderEpoch field from Section 2.1), and each replica keeps a leader-epoch-checkpoint file mapping each epoch to its starting offset.
Leader epochs solve a subtle problem: after a failover, a former leader may have records that never reached the other replicas. When it comes back as a follower, it asks the current leader, "where did epoch N end?" (OffsetsForLeaderEpoch), and truncates everything after that point before fetching again. This keeps all replicas identical. Earlier versions used the high watermark for this and could lose or diverge data in rare cases (KIP-101).
Old leader B1 returns after a failover:
B1 log: [0..9 epoch 3][10 11 epoch 3] <- 10 and 11 were never replicated
B2 log: [0..9 epoch 3][10 11 12 epoch 4] <- B2 became leader in epoch 4 at offset 10
B1 -> B2: "end offset of epoch 3?" B2 -> B1: "10"
B1 truncates to offset 10, then fetches 10, 11, 12 (epoch 4) from B2 -> logs identical
Preferred leaders
The first replica in a partition's assignment is its preferred leader. When topics are created, Kafka spreads preferred leaders evenly across brokers. After a broker restarts, though, its partitions are led elsewhere and the cluster becomes imbalanced. With auto.leader.rebalance.enable=true (the default), the controller periodically (leader.imbalance.check.interval.seconds, default 300) moves leadership back to preferred leaders if a broker's imbalance exceeds leader.imbalance.per.broker.percentage (default 10%).
Example: inspecting leaders and moving leadership back
bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders
Topic: orders TopicId: q8bGv1TfRXSX0n3b1n7Y2w PartitionCount: 3 ReplicationFactor: 3 Configs: min.insync.replicas=2
Topic: orders Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1
Topic: orders Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: orders Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Partition 0's preferred leader is broker 1 (first in Replicas), but broker 2 is leading it, probably because broker 1 restarted. To restore preferred leaders immediately instead of waiting for the automatic check:
bin/kafka-leader-election.sh --bootstrap-server broker1:9092 \
--election-type PREFERRED --topic orders --partition 0
try (Admin admin = Admin.create(Map.of("bootstrap.servers", "broker1:9092"))) {
Map<TopicPartition, Optional<Throwable>> result = admin
.electLeaders(ElectionType.PREFERRED, Set.of(new TopicPartition("orders", 0)))
.partitions().get();
// Optional.empty() means success; otherwise the Throwable says why (e.g. already preferred)
result.forEach((tp, err) -> System.out.println(tp + " -> " + err.map(Throwable::getMessage).orElse("ok")));
}
Key Takeaways
- Each partition has one leader that handles all writes (and reads by default). Followers only copy the leader's log.
- Leaders are spread across brokers, so every broker leads some partitions and follows others.
- Followers replicate by fetching from the leader. Their fetch offsets tell the leader how far they have replicated.
- Leader epochs let a returning replica truncate records that were never committed, keeping all replicas identical.
- The first replica in the assignment is the preferred leader. Automatic or manual preferred leader election keeps leadership balanced.
Practice
- (Beginner) Why does Kafka route all writes for a partition through a single leader instead of letting any replica accept writes?
- (Intermediate) Using the
kafka-topics.sh --describeoutput above, list which brokers currently lead how many partitions, and which partitions would move if you ran a preferred leader election for all partitions.- Interview question: "How does a former leader that comes back after a crash avoid keeping records that the rest of the cluster never saw?" Hint: leader epochs, the
OffsetsForLeaderEpochrequest, and truncation.
In-Sync Replicas (ISR)
Theory
Having three replicas is only useful if they are actually up to date. A follower that crashed an hour ago is technically a replica, but it is useless as a failover target because it is missing an hour of data. Kafka therefore keeps, for each partition, the set of replicas that are caught up: the in-sync replica set (ISR).
- The leader is always in the ISR.
- A follower is in the ISR if it has fully caught up with the leader's log end offset at some point within the last
replica.lag.time.max.ms(default 30 seconds). - A follower that falls behind for longer, or stops fetching, is removed from the ISR (an ISR shrink). When it catches up again, it is added back (an ISR expand).
The ISR matters for two reasons:
- Durability: a producer using
acks=allgets its acknowledgment only after every replica in the ISR has the record. - Failover: by default, only ISR members can become the new leader, because only they are guaranteed to have every committed record.
Analogy: a relay team where only runners who have kept up with the pack are allowed to receive the baton. A runner who stopped to tie their shoes is still on the team, but they cannot take the baton until they catch up.
Lag is measured in time, not messages
Older versions also removed followers that were a certain number of messages behind (replica.lag.max.messages). That caused constant ISR flapping during traffic bursts, where every follower was briefly many messages behind. The setting was removed, and today only time matters: a follower is out of sync if it has not caught up to the leader's end at any point in the last 30 seconds.
Partition payments-0, RF=3, replica.lag.time.max.ms=30000
t=0s ISR = [1, 2, 3] all followers fetching normally
t=5s broker 3's disk becomes very slow; it falls further behind
t=35s leader 1: replica 3 has not caught up for > 30s
-> leader asks the controller to shrink the ISR (AlterPartition)
-> controller commits ISR = [1, 2] IsrShrinksPerSec++
-> acks=all writes now wait for brokers 1 and 2 only
t=120s broker 3 recovers and catches up to the leader's log end offset
-> AlterPartition: ISR = [1, 2, 3] IsrExpandsPerSec++
In KRaft mode the leader proposes ISR changes, and the controller commits them to the metadata log (a PartitionChangeRecord), so every broker learns the new ISR.
ISR and min.insync.replicas
If replicas keep dropping out, the ISR could shrink to just the leader. Then acks=all would mean "acknowledged by one broker", which is no safer than acks=1. The topic or broker setting min.insync.replicas sets a floor: an acks=all write is rejected if the ISR is smaller than this value.
| RF | min.insync.replicas | ISR Size | acks=all Write |
|---|---|---|---|
| 3 | 2 | 3 | Accepted, stored on 3 brokers |
| 3 | 2 | 2 | Accepted, stored on 2 brokers |
| 3 | 2 | 1 | Rejected with NOT_ENOUGH_REPLICAS (producer retries) |
| 3 | 1 | 1 | Accepted, but stored on only 1 broker (risky) |
Writes with acks=0 or acks=1 ignore min.insync.replicas. The producer side of this is covered in Section 4.3.
Monitoring the ISR
| Metric (broker) | Healthy Value | Meaning When Not Healthy |
|---|---|---|
UnderReplicatedPartitions | 0 | Some partitions have fewer in-sync replicas than assigned replicas |
UnderMinIsrPartitionCount | 0 | Some partitions reject acks=all writes |
AtMinIsrPartitionCount | 0 (or small) | One more failure will start rejecting writes |
IsrShrinksPerSec / IsrExpandsPerSec | Near 0 | Frequent changes indicate slow followers, network trouble, or overload |
Example: finding unhealthy partitions and handling the producer error
# Partitions where ISR < replicas
bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions
# Partitions where ISR < min.insync.replicas (acks=all writes fail)
bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-min-isr-partitions
Topic: payments Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000); // total time allowed for retries
producer.send(new ProducerRecord<>("payments", paymentId, json), (metadata, exception) -> {
if (exception == null) {
return; // stored on at least min.insync.replicas brokers
}
if (exception instanceof NotEnoughReplicasException
|| exception instanceof NotEnoughReplicasAfterAppendException) {
// Retriable: the client already retried until delivery.timeout.ms ran out.
// Reaching here means the ISR stayed too small for the whole timeout.
log.error("payments ISR below min.insync.replicas for too long", exception);
}
deadLetterOrAlert(paymentId, exception); // application-specific handling
});
Key Takeaways
- The ISR is the set of replicas (including the leader) that have caught up with the leader within
replica.lag.time.max.ms. - Followers leave the ISR when they fall behind for too long and rejoin when they catch up. The controller commits these changes.
acks=allwaits for every ISR member. By default only ISR members can become leader.min.insync.replicaspreventsacks=allfrom silently degrading to a single copy by rejecting writes when the ISR is too small.- Alert on
UnderMinIsrPartitionCountandUnderReplicatedPartitions, and investigate frequent ISR shrinks and expands.
Practice
- (Beginner) A partition has
Replicas: 1,2,3andIsr: 1,3. Which broker is out of sync, and which brokers could become leader if broker 1 fails?- (Intermediate) A topic has RF=3 and
min.insync.replicas=3. Explain what happens toacks=allproducers during a routine rolling restart, and propose a better setting.- Interview question: "Why does Kafka define 'in sync' in terms of time rather than number of messages behind?" Hint: think about what happens to message-count lag during a sudden traffic spike, even on healthy followers.
Replication Factor
Theory
The replication factor (RF) is the number of copies of each partition, including the leader. It is set per topic, and it determines how many broker failures a topic can survive.
- RF=1: no copies. Losing the broker (or its disk) loses the data.
- RF=2: survives one failure, but during maintenance on one broker, you are down to a single copy.
- RF=3: the production standard. Survives one failure while still having two copies, and survives two failures without losing committed data.
Analogy: the replication factor is how many photocopies of an important document you keep in different places. One copy is fragile. Two copies are fine until one is at the repair shop. Three copies let you lose one and still have a backup for the backup.
Durability and availability depend on RF and min.insync.replicas together
With acks=all:
| RF | min.insync.replicas | Broker Failures Before Writes Are Rejected | Committed Copies Guaranteed | Typical Use |
|---|---|---|---|---|
| 1 | 1 | 0 (writes fail with the only broker) | 1 | Local development |
| 2 | 1 | 1 | 1 | Low-value data where availability matters more |
| 2 | 2 | 0 | 2 | Rarely sensible: any single failure blocks writes |
| 3 | 2 | 1 | 2 | Recommended default for production |
| 3 | 3 | 0 | 3 | Blocks writes during every restart; rarely worth it |
| 5 | 3 | 2 | 3 | Very critical data, or clusters across many failure domains |
Two simple formulas:
- Write availability with
acks=alltolerates RF -min.insync.replicasfailed replicas. - Committed data survives the loss of up to
min.insync.replicas- 1 replicas in the worst case, because every committed record is on at leastmin.insync.replicasbrokers.
Where replication factor is configured
| Setting | Default | Applies To |
|---|---|---|
--replication-factor / NewTopic(..., rf) | (explicit) | A topic created explicitly |
default.replication.factor (broker) | 1 | Auto-created topics, and creation requests that do not specify RF |
offsets.topic.replication.factor | 3 | __consumer_offsets |
transaction.state.log.replication.factor | 3 | __transaction_state |
Two common surprises: the broker default of 1 is unsafe for production (set default.replication.factor=3), and on a single-broker development cluster, the internal topics' default of 3 cannot be satisfied, so local setups override these to 1.
RF cannot exceed the number of brokers. Creating a topic with RF=3 on a two-broker cluster fails with InvalidReplicationFactorException.
The cost of replication
| Cost | Scales With |
|---|---|
| Disk space | RF times the data size |
| Network | Every byte produced is sent RF-1 more times between brokers (and across AZs if racks are spread) |
Latency with acks=all | Waiting for the slowest ISR member |
Changing the replication factor
There is no "alter RF" command. You change RF by reassigning each partition to a new replica list with kafka-reassign-partitions.sh. Kafka copies data to the new replicas in the background and switches over once they join the ISR.
Example: increasing a topic's RF from 2 to 3
{
"version": 1,
"partitions": [
{ "topic": "payments", "partition": 0, "replicas": [1, 2, 3] },
{ "topic": "payments", "partition": 1, "replicas": [2, 3, 1] },
{ "topic": "payments", "partition": 2, "replicas": [3, 1, 2] }
]
}
# Start the reassignment, limiting replication traffic to 50 MB/s so clients are not starved
bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
--reassignment-json-file increase-rf.json --execute --throttle 50000000
# Check progress; once complete, --verify also removes the throttle
bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
--reassignment-json-file increase-rf.json --verify
Keep the current leader as the first entry where possible, and give each partition a different first replica so preferred leaders stay balanced.
Key Takeaways
- RF is the number of copies of each partition, including the leader. RF=3 is the production standard.
- Durability and write availability depend on RF and
min.insync.replicastogether. RF=3 withmin.insync.replicas=2tolerates one failure without blockingacks=allwrites. - The broker default
default.replication.factor=1is unsafe. Set it explicitly in production. - Replication multiplies disk and network usage, and
acks=alllatency depends on the slowest in-sync follower. - Change RF with a partition reassignment, throttled to protect client traffic.
Practice
- (Beginner) A topic has RF=3 and
min.insync.replicas=2. How many brokers hosting a given partition can be down whileacks=allproducers can still write to it?- (Intermediate) Write the reassignment JSON to change a 4-partition topic from RF=3 to RF=2 on brokers 1 to 4, keeping preferred leaders balanced. What risk are you accepting?
- Interview question: "Explain the relationship between replication factor,
min.insync.replicas, andacks. Which combination would you use for financial transactions, and why?" Hint: separate how many copies exist, how many must confirm a write, and what happens to writes when brokers fail.
High Watermark and Log End Offset
Theory
A record written to the leader is not yet safe: if the leader crashes before followers copy it, the record may disappear when another replica takes over. If consumers had already read such a record, different consumers could see different histories. Kafka prevents this by tracking two positions per replica and only exposing records that are safely replicated.
- Log end offset (LEO): the offset the next record will get on a given replica. Each replica has its own LEO.
- High watermark (HW): the offset up to which all ISR members have the data. Records below the HW are committed. Only committed records are visible to consumers.
In short, HW = min(LEO of all ISR members), maintained by the leader.
Analogy: a newspaper's print room. Reporters (the leader) may have drafted articles up to page 12 (the LEO), but the editor only releases pages that every proofreader (each ISR member) has checked, so far up to page 9 (the HW). Readers only ever see released pages, so nobody reads an article that might be pulled.
Partition orders-0, ISR = [1, 2, 3]
0 1 2 3 4 5 6 7 8 9
Leader 1: [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] LEO = 10
Follower 2:[x] [x] [x] [x] [x] [x] [x] [x] LEO = 8
Follower 3:[x] [x] [x] [x] [x] [x] [x] LEO = 7
^
HW = min(10, 8, 7) = 7
Consumers can read offsets 0..6. Offsets 7..9 exist on the leader but are not yet committed.
How the high watermark advances
The leader learns each follower's LEO from the offsets in that follower's fetch requests, and tells followers the current HW in fetch responses. So the followers' copy of the HW lags one fetch round behind the leader's.
Step by step, for a single acks=all write with RF=3 (all replicas start at LEO 5, HW 5):
| Step | Event | Leader LEO | F2 LEO | F3 LEO | Leader HW | Follower HW |
|---|---|---|---|---|---|---|
| 1 | Producer sends a record. Leader appends it at offset 5. | 6 | 5 | 5 | 5 | 5 |
| 2 | Followers fetch from offset 5 and receive the record. | 6 | 6 | 6 | 5 | 5 |
| 3 | Followers fetch from offset 6. The leader now knows both have offset 5. | 6 | 6 | 6 | 6 | 5 |
| 4 | The produce request is acknowledged. Consumers can read offset 5. | 6 | 6 | 6 | 6 | 5 |
| 5 | Fetch responses carry HW=6 to the followers. | 6 | 6 | 6 | 6 | 6 |
Why this matters to you
- Consumer visibility: consumers (with
read_uncommitted, the default) only see records below the HW. A consumer cannot read a record before it is fully replicated, which adds a small amount of latency with RF > 1. - Lag numbers: tools such as
kafka-consumer-groups.shreport lag against the HW, not the leader's LEO. - Transactions: consumers with
isolation.level=read_committedread only up to the last stable offset (LSO), the first offset of the earliest still-open transaction (or the HW if none is open). So LSO is always less than or equal to HW, which is less than or equal to LEO (Chapter 8). - Truncation: when a new leader takes over, records above the new leader's log end on other replicas were never committed. Those replicas truncate them (using leader epochs, as described earlier), and this is safe because no consumer ever saw them.
log start LSO HW LEO
| | | |
+-------------------+--------------------+------------------+
|<- read_committed->| | |
|<------- read_uncommitted ------------->| |
|<- starts with an ->|<- not yet on --->|
open transaction all ISR members
Example: a small replication simulation
class Replica:
def __init__(self, broker_id: int):
self.broker_id = broker_id
self.log: list[str] = []
self.hw = 0 # follower's view of the high watermark
@property
def leo(self) -> int: # log end offset = next offset to be written
return len(self.log)
class PartitionLeader(Replica):
def __init__(self, broker_id: int, followers: list[Replica]):
super().__init__(broker_id)
self.followers = followers
self.follower_leo = {f.broker_id: 0 for f in followers} # learned from fetch offsets
self.isr = {broker_id, *self.follower_leo}
def append(self, value: str) -> int:
self.log.append(value)
return self.leo - 1 # offset of the new record
def handle_fetch(self, follower: Replica) -> None:
# The fetch offset tells the leader what the follower already has
self.follower_leo[follower.broker_id] = follower.leo
self._advance_hw()
# Send records from the fetch offset, plus the current HW
follower.log.extend(self.log[follower.leo:])
follower.hw = self.hw
def _advance_hw(self) -> None:
isr_leos = [self.leo] + [self.follower_leo[b] for b in self.isr if b != self.broker_id]
self.hw = min(isr_leos) # HW = min LEO across the ISR
def readable(self) -> list[str]:
return self.log[: self.hw] # consumers only see committed records
f2, f3 = Replica(2), Replica(3)
leader = PartitionLeader(1, [f2, f3])
leader.append("order-1")
print(leader.hw, leader.readable()) # 0 [] written, not yet committed
leader.handle_fetch(f2); leader.handle_fetch(f3) # followers copy offset 0
print(leader.hw, leader.readable()) # 0 [] leader does not know yet
leader.handle_fetch(f2); leader.handle_fetch(f3) # next fetches report LEO=1
print(leader.hw, leader.readable()) # 1 ['order-1'] committed and visible
print(f2.hw, f3.hw) # 0 1 f2 fetched before the HW moved
leader.handle_fetch(f2)
print(f2.hw) # 1 followers learn the HW one fetch later
Key Takeaways
- LEO is per replica: the next offset to be written. HW is per partition (maintained by the leader): the minimum LEO across the ISR.
- Records below the HW are committed and visible to consumers. Records at or above it are not, and may be truncated after a failover.
- The leader advances the HW from followers' fetch offsets. Followers learn it one fetch round later.
read_committedconsumers stop at the last stable offset, so LSO is at most HW, which is at most LEO.- Consumer lag is measured against the HW.
Practice
- (Beginner) A partition with ISR
[1, 2, 3]has LEOs of 120, 118, and 119. What is the HW, and which offsets can consumers read?- (Intermediate) Extend the Python simulation so that follower 3 stops fetching, and remove it from the ISR after a configurable number of rounds. Show that the HW stalls until follower 3 is removed and then advances.
- Interview question: "Why doesn't Kafka let consumers read a record as soon as the leader has written it?" Hint: consider what a consumer would have seen if the leader crashed and a follower without that record became leader.
Leader Election
Theory
When a partition's leader becomes unavailable, its partition cannot accept writes until a new leader is chosen. Leader election is the process of choosing that new leader, and it is performed by the active controller. Making it fast and safe is central to Kafka's availability.
Analogy: when a ship's captain is incapacitated, command passes to the next officer in a predefined chain, but only to someone who has been on the bridge and knows the current course. A crew member who just woke up from a long sleep is skipped, even if they rank higher.
When elections happen
| Trigger | Cause |
|---|---|
| Broker failure | The controller fences a broker that stopped heartbeating, and every partition it led needs a new leader |
| Controlled shutdown | A broker that is stopping asks the controller to move its leaderships before it exits |
| Preferred leader election | Automatic rebalancing or kafka-leader-election.sh --election-type PREFERRED |
| Reassignment | The current leader is removed from the partition's replica list |
| Unclean election | Explicitly enabled or triggered when no ISR member is available (next topic) |
How the controller chooses
For a normal (clean) election, the controller takes the partition's replica list in assignment order and picks the first replica that is alive and in the ISR. It then:
- Increments the partition's leader epoch.
- Writes a
PartitionChangeRecord(new leader, new ISR, new epoch) to the metadata log. - Brokers apply the record: the new leader starts accepting writes, and the other replicas start fetching from it, truncating any uncommitted tail first.
Because the new leader is an ISR member, it has every committed record (everything below the HW), so no committed data is lost.
Partition orders-0: Replicas [1, 2, 3], ISR [1, 2, 3], leader 1, epoch 7
t=0.0s broker 1 crashes
t=0-9s producers to orders-0 get timeouts; controller waits for broker.session.timeout.ms
t=9.0s controller fences broker 1
-> first replica in [1, 2, 3] that is alive and in ISR = 2
-> PartitionChangeRecord: leader=2, ISR=[2, 3], leaderEpoch=8
t=9.1s brokers apply the record; broker 2 now leads orders-0
t=9.1s+ producers get NOT_LEADER_OR_FOLLOWER or time out, refresh metadata, retry against broker 2
Controlled shutdown makes planned restarts almost invisible
When you stop a broker normally (SIGTERM), it does not just disappear. It tells the controller it wants to shut down, the controller moves its leaderships to other ISR members, and only then does the broker exit. Clients see a quick leadership change instead of a timeout. A hard kill (SIGKILL, power loss) skips this step, so failover waits for the session timeout. This is why rolling restarts should always use a graceful stop and should wait until there are no under-replicated partitions before moving on to the next broker.
What clients experience
- Producers receive
NOT_LEADER_OR_FOLLOWER(or time out), refresh metadata, and retry. With the defaultretriesanddelivery.timeout.ms(2 minutes), and idempotence enabled (the default since 3.0), a failover is usually invisible to application code apart from a latency spike, and retries do not create duplicates (Chapter 8). - Consumers also refresh metadata and resume fetching from the new leader. They include the leader epoch in their fetches, so if the log was truncated during failover they can detect it and reset their position correctly (KIP-320).
Example: watching an election and triggering one
# Before: leader is broker 1
bin/kafka-topics.sh --bootstrap-server broker2:9092 --describe --topic orders | grep "Partition: 0"
# Topic: orders Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
# Stop broker 1 gracefully (controlled shutdown), then describe again
# Topic: orders Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2,3
# After broker 1 is back in the ISR, move leadership back to the preferred replica
bin/kafka-leader-election.sh --bootstrap-server broker2:9092 \
--election-type PREFERRED --all-topic-partitions
A producer callback that logs failover effects:
producer.send(record, (metadata, e) -> {
if (e instanceof NotLeaderOrFollowerException) {
// Normally retried internally; seeing it here means retries were exhausted or disabled
log.warn("leader moved for {} and retries ran out", record.topic(), e);
} else if (e instanceof TimeoutException) {
// No leader was reachable within delivery.timeout.ms: e.g. no ISR member available
log.error("no available leader for {}", record.topic(), e);
}
});
Key Takeaways
- The active controller elects partition leaders. A clean election picks the first live ISR replica in assignment order.
- Each election increments the leader epoch, and the change is committed to the metadata log.
- Because the new leader comes from the ISR, committed data is never lost in a clean election.
- Controlled shutdown moves leadership before a broker stops, which makes planned restarts nearly invisible to clients.
- Producers and consumers recover automatically by refreshing metadata and retrying. Idempotence prevents duplicates from retries.
Practice
- (Beginner) A partition has
Replicas: 3,1,2,Isr: 3,2, and leader 3. Broker 3 fails. Which broker becomes the leader, and why not broker 1?- (Intermediate) Compare the client-visible unavailability of a partition when its leader's broker is (a) stopped with SIGTERM and (b) killed with SIGKILL. Which configuration value mostly determines the duration in case (b)?
- Interview question: "Walk me through what happens, end to end, when the broker leading a partition crashes while a producer is sending to it with
acks=all." Hint: cover detection, the controller's choice, the metadata update, what the producer sees, and why no acknowledged record is lost.
Unclean Leader Election
Theory
A clean election needs at least one live ISR member. What if every ISR replica is down, and the only surviving replica is one that had fallen out of sync? Kafka must choose between two bad options:
- Wait for an ISR member to come back. No data is lost, but the partition is unavailable (no reads or writes) until then, possibly for hours.
- Elect the out-of-sync replica anyway: an unclean leader election. The partition is available again immediately, but every record the new leader is missing is permanently lost, including records that producers were told were committed.
This is the consistency-versus-availability trade-off from the CAP theorem, and Kafka lets you pick per topic with unclean.leader.election.enable. The default is false: Kafka prefers consistency.
Analogy: the only two people who memorized the latest version of a speech are both sick, and one assistant memorized an older draft. You can postpone the event (unavailable) or let the assistant give the old version (available, but the latest changes are gone and the audience never knows).
What data loss looks like
Partition payments-0: Replicas [1, 2, 3], min.insync.replicas=2
t0 ISR = [1, 2, 3], all at LEO 100, HW 100
t1 broker 3 falls behind at LEO 80 and is removed: ISR = [1, 2]
t2 producers write offsets 100..119 with acks=all HW = 120 (on brokers 1 and 2)
t3 brokers 1 AND 2 fail. Only broker 3 is alive (LEO 80, not in ISR).
unclean.leader.election.enable=false (default):
partition OFFLINE until broker 1 or 2 returns. No data lost.
unclean.leader.election.enable=true:
broker 3 becomes leader at LEO 80.
Offsets 80..119 (40 acknowledged records) are LOST.
New writes reuse offsets 80, 81, ... with different content.
When brokers 1 and 2 return, they truncate to 80 and follow broker 3.
Consumers that committed offset 120 are now beyond the log end (offset out of range).
The danger is not only lost records. The same offsets now refer to different records, which can confuse any system that stored offsets as references, such as consumers, stream processors, or external checkpoints.
Choosing a setting
false (default) | true | |
|---|---|---|
| Priority | Consistency and durability | Availability |
| When all ISR members are down | Partition offline until one returns | New leader from out-of-sync replicas |
| Acknowledged writes | Never lost | Can be lost |
| Good fit | Payments, orders, changelogs, anything stateful | Metrics, logs, telemetry where gaps are acceptable but downtime is not |
The setting exists at the broker level (default for all topics) and the topic level (override). Rather than enabling it permanently, many teams leave it off and, in a real emergency, trigger a one-time unclean election for specific partitions after deciding that availability matters more than the missing data:
bin/kafka-leader-election.sh --bootstrap-server broker3:9092 \
--election-type UNCLEAN --topic payments --partition 0
Reducing the need for it
- Use RF=3 and
min.insync.replicas=2so that acknowledged data is always on at least two brokers. - Spread replicas across racks or AZs so that correlated failures are less likely to take down every ISR member.
- Alert on
UnderMinIsrPartitionCountandOfflinePartitionsCountto act before or as soon as a partition goes offline. - Recent Kafka 4.x releases add Eligible Leader Replicas (ELR) (KIP-966). When the ISR shrinks below
min.insync.replicas, the high watermark cannot advance, so replicas removed from the ISR in that state still hold every committed record. Kafka records them in a separate ELR set and can elect one of them cleanly if all ISR members fail, which avoids many situations that previously forced a choice between waiting and an unclean election. Check whether theeligible.leader.replicas.versionfeature is enabled in your cluster.
Example: making the trade-off explicit per topic
try (Admin admin = Admin.create(Map.of("bootstrap.servers", "broker1:9092"))) {
// Telemetry: availability first
ConfigResource metrics = new ConfigResource(ConfigResource.Type.TOPIC, "host-metrics");
// Payments: consistency first (also the default, but stated explicitly for clarity)
ConfigResource payments = new ConfigResource(ConfigResource.Type.TOPIC, "payments");
admin.incrementalAlterConfigs(Map.of(
metrics, List.of(new AlterConfigOp(
new ConfigEntry(TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "true"), AlterConfigOp.OpType.SET)),
payments, List.of(new AlterConfigOp(
new ConfigEntry(TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "false"), AlterConfigOp.OpType.SET))
)).all().get();
}
In KRaft mode, the controller reacts to this setting dynamically: enabling it on a topic with offline partitions lets the controller perform the unclean election without restarting anything.
Key Takeaways
- An unclean leader election makes an out-of-sync replica the leader when no ISR member is available.
- It restores availability immediately but permanently loses records the new leader is missing, including acknowledged ones, and reuses their offsets.
unclean.leader.election.enabledefaults tofalse(consistency first). Enable it only for data where gaps are acceptable.- In an emergency, prefer a deliberate one-time unclean election with
kafka-leader-election.shover enabling it permanently. - RF=3,
min.insync.replicas=2, rack awareness, good alerting, and (in recent releases) Eligible Leader Replicas make the situation much rarer.
Practice
- (Beginner) For each topic, decide whether to allow unclean leader election: (a) application debug logs, (b) bank transfers, (c) a Kafka Streams changelog topic for a state store.
- (Intermediate) In the data loss timeline above, a consumer group had committed offset 115 before the failure. After the unclean election, the new leader's log end is 80 and producers have written 10 new records. Describe what the consumer sees on restart with
auto.offset.reset=earliestand withlatest, and which records it never processes.- Interview question: "Explain unclean leader election in terms of the CAP theorem, and describe a situation where you would deliberately trigger one." Hint: identify which property each setting gives up during a partition failure, and consider data whose value is mostly in being current rather than complete.
<a id="3-getting-started-and-operations-basics"></a>