16. Advanced Internals
Earlier chapters explain what Kafka does. This chapter shows how it does it: the binary protocol that every client, broker, and controller speaks, the threads and data structures that turn requests into replicated logs, and the extension points and proposals through which Kafka keeps changing. These internals explain metrics you have already used (RequestQueueTimeMs, PurgatorySize, MaxLag). They also let you debug at the protocol level, plan client and broker upgrades, and judge new designs such as diskless topics.
<a id="161-kafka-protocol"></a>
16.1 Kafka Protocol
Every interaction with a Kafka cluster is a request and a response in one binary protocol over TCP, from a producer's send() to a controller election. This section covers how messages are framed, which APIs exist, how versions are negotiated, and what that means for keeping clients and brokers compatible.
Binary Wire Protocol
Theory
Kafka could have used HTTP or gRPC, but it was built for a workload where those would get in the way. A broker receives millions of records per second, and it wants to store a producer's batch exactly as it arrived and later send those same bytes to consumers with zero-copy transfer (Section 7.1). A compact binary format with length-prefixed fields lets the broker read a small header, validate the batch, and treat the records as an opaque blob. Long-lived TCP connections avoid per-request connection setup, and several requests can be in flight on one connection.
Analogy: a container port. Every container has the same standardized frame (size, labels) so cranes can move it without opening it. The cargo inside (record batches) is only unpacked at its destination (the consumer).
Framing
Every message on the wire is a 4-byte size followed by that many bytes:
Request
+--------------+-----------------------------------------+------------------------------+
| size: int32 | request header | request body |
| (bytes that | api_key int16 which API | fields defined by |
| follow) | api_version int16 which schema | (api_key, api_version) |
| | correlation_id int32 echoed back | |
| | client_id nullable_string | |
| | tagged_fields (header v2 only) | |
+--------------+-----------------------------------------+------------------------------+
Response
+--------------+-----------------------------------------+------------------------------+
| size: int32 | correlation_id int32 [+ tagged_fields] | response body |
+--------------+-----------------------------------------+------------------------------+
Rules that follow from this layout:
- All fixed-width integers are big-endian (network byte order).
- The response does not repeat the API key or version. The client matches a response to its request by
correlation_id, so it must remember what it sent. - On one connection, the broker processes one request at a time and returns responses in the order the requests were sent. Clients may still pipeline several requests (
max.in.flight.requests.per.connection); the extra ones wait in the socket buffer. - A
Producerequest withacks=0gets no response at all.
Primitive types
| Type | Encoding | Typical use |
|---|---|---|
int8, int16, int32, int64 | Fixed width, big-endian | Error codes, API keys, partition IDs, offsets |
varint, varlong | ZigZag, then 7 bits per byte (as in Protocol Buffers) | Fields inside records: lengths, offset and timestamp deltas |
unsigned_varint | 7 bits per byte, high bit = "more bytes follow" | Lengths in compact types, tags |
uuid | 16 bytes | Topic IDs |
string, nullable_string | int16 length (-1 = null) + UTF-8 | Strings in older (non-flexible) versions |
compact_string | unsigned_varint of length + 1 (0 = null) + UTF-8 | Strings in flexible versions |
bytes, compact_bytes | int32 length, or unsigned_varint length + 1 | Opaque data |
array, compact_array | int32 count (-1 = null), or unsigned_varint count + 1 | Lists of topics, partitions, brokers |
records | Length-prefixed blob of record batches (Section 7.1) | Produce and Fetch payloads |
tagged_fields | unsigned_varint count, then for each: tag, size, bytes | Optional fields added without a new version |
Varints keep small numbers small. For example, the unsigned varint of 300 is two bytes:
300 = 0b1_0010_1100
low 7 bits = 010_1100 = 0x2C, more bytes follow -> 0xAC
next 7 bits = 000_0010 = 0x02, last byte -> 0x02
encoding: AC 02 (an int32 would always take 4 bytes)
Flexible versions
Since Kafka 2.4 (KIP-482), newer versions of each API are flexible: they use the compact types and allow tagged fields at the end of every structure. A tagged field can be added later without changing the version number, and old readers skip tags they do not know. Flexible requests use request header v2, and flexible responses use response header v1 (each adds a tagged-field section). Two exceptions keep negotiation safe:
client_idstays a classicint16-length string even in header v2, so the broker can always parse it.- The
ApiVersionsresponse always uses response header v0, because the client reads it before it knows which versions the broker supports.
Example: talking to a broker by hand (Python)
This script sends an ApiVersions v3 request (API key 18) using only the standard library and prints which API versions the broker supports. It shows framing, the v2 header, compact strings, and tagged fields in about 60 lines.
import socket
import struct
API_NAMES = {0: "Produce", 1: "Fetch", 2: "ListOffsets", 3: "Metadata", 8: "OffsetCommit",
10: "FindCoordinator", 18: "ApiVersions", 19: "CreateTopics", 68: "ConsumerGroupHeartbeat"}
def uvarint(n: int) -> bytes:
out = bytearray()
while True:
byte, n = n & 0x7F, n >> 7
if n:
out.append(byte | 0x80) # high bit set: more bytes follow
else:
out.append(byte)
return bytes(out)
def compact_string(s: str) -> bytes:
data = s.encode("utf-8")
return uvarint(len(data) + 1) + data # length + 1, because 0 means null
def read_uvarint(buf: bytes, pos: int) -> tuple[int, int]:
result = shift = 0
while True:
byte = buf[pos]
pos += 1
result |= (byte & 0x7F) << shift
if not byte & 0x80:
return result, pos
shift += 7
def skip_tagged_fields(buf: bytes, pos: int) -> int:
count, pos = read_uvarint(buf, pos)
for _ in range(count):
_tag, pos = read_uvarint(buf, pos)
size, pos = read_uvarint(buf, pos)
pos += size # unknown tags are skipped, never parsed
return pos
def recv_exact(sock: socket.socket, n: int) -> bytes:
data = b""
while len(data) < n:
chunk = sock.recv(n - len(data))
if not chunk:
raise ConnectionError("broker closed the connection")
data += chunk
return data
client_id = b"wire-demo"
header = (struct.pack(">hhih", 18, 3, 42, len(client_id)) # api_key, api_version, correlation_id, client_id length
+ client_id
+ uvarint(0)) # header tagged fields: none
body = compact_string("wire-demo") + compact_string("1.0") + uvarint(0) # software name, version, no tags
payload = header + body
with socket.create_connection(("localhost", 9092)) as sock:
sock.sendall(struct.pack(">i", len(payload)) + payload) # size prefix, then the message
size = struct.unpack(">i", recv_exact(sock, 4))[0]
resp = recv_exact(sock, size)
correlation_id, error_code = struct.unpack_from(">ih", resp, 0) # response header v0, then error_code
print(f"correlation_id={correlation_id} error_code={error_code}")
pos = 6
count, pos = read_uvarint(resp, pos)
for _ in range(count - 1): # compact array: stored count is N + 1
key, vmin, vmax = struct.unpack_from(">hhh", resp, pos)
pos = skip_tagged_fields(resp, pos + 6)
if key in API_NAMES:
print(f"{key:>3} {API_NAMES[key]:<24} {vmin}..{vmax}")
correlation_id=42 error_code=0
0 Produce 3..12
1 Fetch 4..17
18 ApiVersions 0..4
...
(Illustrative output. The exact ranges depend on the broker release.)
The request bytes on the wire, annotated:
00 00 00 23 size: 35 bytes follow
00 12 api_key 18 = ApiVersions
00 03 api_version 3
00 00 00 2a correlation_id 42
00 09 77 69 72 65 2d 64 65 6d 6f client_id "wire-demo" (int16 length, even in header v2)
00 header tagged fields: 0
0a 77 69 72 65 2d 64 65 6d 6f client_software_name: uvarint 10 = length 9 + 1
04 31 2e 30 client_software_version "1.0"
00 body tagged fields: 0
Key Takeaways
- Every Kafka message is a 4-byte size followed by a header and a body. Integers are big-endian.
- Responses are matched to requests by
correlation_id. A broker handles one request per connection at a time and answers in order. - Flexible versions (KIP-482) use compact varint-length types and tagged fields, so optional fields can be added without new versions.
- Record batches travel as an opaque
recordsblob, which lets the broker store and serve them without re-encoding.
Practice
- (Beginner) Encode the unsigned varint of 127, 128, and 16,384 by hand. How many bytes does each take?
- (Intermediate) Extend the Python script to send
ApiVersionswith version 99. Decode the error code and the version range the broker returns, and explain why the broker can still answer a request it does not understand.- Interview question: "Why does Kafka use its own binary protocol instead of HTTP?" Hint: consider connection reuse, pipelining, and what the broker must do with a batch of records between receiving it from a producer and sending it to a consumer.
Request and Response Types
Theory
Each operation in Kafka is an API, identified by a 16-bit API key and defined by a pair of schemas: one request, one response. Kafka 4.x defines about 90 APIs. Most clients use about a dozen of them, and the rest serve administration, coordination, replication, and the KRaft quorum.
It helps to group them by who talks to whom:
Produce, Fetch, Metadata, ...
clients ----------------------------------> brokers
(incl. admin) | \
| \ Fetch (replication)
| -----------------> other brokers
BrokerRegistration, |
BrokerHeartbeat, AlterPartition,
Envelope (forwarded admin) v
controllers <-- Vote, BeginQuorumEpoch,
Fetch, FetchSnapshot (Raft)
Each schema declares which listeners expose it ("listeners": ["broker"], ["controller"], or both). A broker only advertises broker-listener APIs in its ApiVersions response, so clients never see Raft APIs.
API families
| Family | APIs (key) | Used by |
|---|---|---|
| Data path | Produce (0), Fetch (1), ListOffsets (2), Metadata (3) | Producers, consumers, followers (Fetch) |
| Connection setup | ApiVersions (18), SaslHandshake (17), SaslAuthenticate (36) | Every client, first thing after connecting |
| Classic consumer groups | FindCoordinator (10), JoinGroup (11), SyncGroup (14), Heartbeat (12), LeaveGroup (13), OffsetCommit (8), OffsetFetch (9) | Classic protocol consumers, Connect, Streams |
| New group protocols | ConsumerGroupHeartbeat (68), ShareGroupHeartbeat (76), ShareFetch (78), ShareAcknowledge (79) | KIP-848 consumers (Section 5.2), share consumers (Section 5.4) |
| Transactions | InitProducerId (22), AddPartitionsToTxn (24), AddOffsetsToTxn (25), EndTxn (26), TxnOffsetCommit (28), WriteTxnMarkers (27) | Idempotent and transactional producers, coordinators (Chapter 8) |
| Administration | CreateTopics (19), DeleteTopics (20), CreatePartitions (37), IncrementalAlterConfigs (44), DescribeConfigs (32), CreateAcls (30), ElectLeaders (43), AlterPartitionReassignments (45), DescribeCluster (60) | Admin client and CLI tools |
| Broker to controller | BrokerRegistration (62), BrokerHeartbeat (63), AlterPartition (56), Envelope (58) | Brokers in KRaft mode |
| KRaft quorum | Vote (52), BeginQuorumEpoch (53), EndQuorumEpoch (54), DescribeQuorum (55), FetchSnapshot (59), AddRaftVoter (80), RemoveRaftVoter (81) | Controllers |
The ZooKeeper-era controller-to-broker RPCs (LeaderAndIsr, UpdateMetadata, StopReplica) are not used in KRaft. Brokers learn the same facts by replaying the metadata log (Section 16.2).
Errors are usually per item
Most responses carry an error code per topic or per partition, not one for the whole request. A Produce to three partitions can succeed on two and fail on one, and the client retries only the failed one.
| Code | Name | Retriable | Typical cause |
|---|---|---|---|
| 0 | NONE | n/a | Success |
| 1 | OFFSET_OUT_OF_RANGE | No | Consumer position deleted by retention |
| 3 | UNKNOWN_TOPIC_OR_PARTITION | Yes | Topic not created yet, or metadata not propagated |
| 6 | NOT_LEADER_OR_FOLLOWER | Yes | Leadership moved; refresh metadata |
| 7 | REQUEST_TIMED_OUT | Yes | acks=all waited longer than the request timeout |
| 16 | NOT_COORDINATOR | Yes | Group or transaction coordinator moved |
| 19 | NOT_ENOUGH_REPLICAS | Yes | ISR below min.insync.replicas |
| 29 | TOPIC_AUTHORIZATION_FAILED | No | Missing ACL |
| 35 | UNSUPPORTED_VERSION | No | Client and broker share no usable version |
| 74 | FENCED_LEADER_EPOCH | Yes | Client used a leader epoch older than the broker's |
Example: anatomy of a Produce exchange
A flexible Produce request and response, with the fields that matter:
ProduceRequest
transactional_id compact_nullable_string null unless the producer is transactional
acks int16 -1 = all, 1 = leader only, 0 = no response
timeout_ms int32 how long the leader may wait for followers
topic_data[]
name compact_string
partition_data[]
index int32 partition number
records compact_records one or more record batches, stored as received
ProduceResponse
responses[]
name
partition_responses[]
index, error_code
base_offset offset assigned to the first record of the batch
log_append_time_ms -1 unless the topic uses LogAppendTime
log_start_offset
record_errors[] which records in the batch were invalid, and why
current_leader (tagged) new leader if error_code = NOT_LEADER_OR_FOLLOWER (KIP-951)
throttle_time_ms quota delay the client must honor
node_endpoints (tagged) address of that new leader
Example: the conversation behind a simple consumer
What a classic-protocol consumer sends from startup to its first commit, in order:
| Step | API | Sent to | Purpose |
|---|---|---|---|
| 1 | ApiVersions | Bootstrap broker | Negotiate versions on this connection |
| 2 | Metadata | Bootstrap broker | Brokers, partitions, leaders for subscribed topics |
| 3 | FindCoordinator | Any broker | Which broker coordinates group billing |
| 4 | JoinGroup, then SyncGroup | Coordinator | Membership and partition assignment |
| 5 | OffsetFetch | Coordinator | Last committed offsets for assigned partitions |
| 6 | ListOffsets | Partition leaders | Only for partitions with no committed offset (auto.offset.reset) |
| 7 | Fetch (repeated) | Partition leaders | Records |
| 8 | Heartbeat (periodic) | Coordinator | Stay in the group |
| 9 | OffsetCommit | Coordinator | Save progress |
With group.protocol=consumer (KIP-848), steps 4 and 8 collapse into periodic ConsumerGroupHeartbeat calls, and the coordinator computes the assignment.
To watch this on a real cluster, use the Kafka dissector built into Wireshark on plaintext listeners, or temporarily set the broker logger kafka.request.logger to DEBUG, which logs every request and is very verbose.
Key Takeaways
- An API is identified by its key and has a request schema and a response schema. Kafka 4.x has about 90 APIs.
- APIs are scoped to broker or controller listeners. Clients only ever see broker APIs.
- Errors are reported per topic or partition, so clients retry only the parts that failed.
- Knowing the sequence of APIs behind a client operation helps you read broker request metrics and debug hangs ("stuck at
FindCoordinator" points to a different problem than "stuck atFetch").
Practice
- (Beginner) Run
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092and find the API keys forProduce,Fetch, andOffsetCommit. Which APIs from the KRaft quorum family appear, and why?- (Intermediate) List, in order, the APIs a transactional producer sends to process one transaction that writes to two partitions and commits consumer offsets. Mark which go to the transaction coordinator, which to the group coordinator, and which to partition leaders.
- Interview question: "A
Producerequest to three partitions returnsNOT_LEADER_OR_FOLLOWERfor one of them. What does the client do, and what changed in Kafka 3.7 to make this faster?" Hint: per-partition errors, metadata refresh, and the tagged fields in the response.
API Versioning
Theory
A Kafka cluster serves clients written in many languages and released over many years, and brokers are upgraded one at a time. No one can upgrade everything at once, so every API carries its own version number, increased whenever its schema or meaning changes. A client and a broker agree on a version per API, per connection.
Analogy: two people who share several languages. Before talking business they list the languages each speaks (
ApiVersions), then each topic is discussed in the most recent dialect both understand.
Where versions are defined
Every message is defined in a JSON file in the Kafka source tree (clients/src/main/resources/common/message/). A code generator turns them into Java classes such as ProduceRequestData. An abridged excerpt:
{
"apiKey": 0,
"type": "request",
"listeners": ["broker"],
"name": "ProduceRequest",
"validVersions": "3-12",
"flexibleVersions": "9+",
"fields": [
{ "name": "TransactionalId", "type": "string", "versions": "3+", "nullableVersions": "3+",
"default": "null", "entityType": "transactionalId" },
{ "name": "Acks", "type": "int16", "versions": "0+" },
{ "name": "TimeoutMs", "type": "int32", "versions": "0+" },
{ "name": "TopicData", "type": "[]TopicProduceData", "versions": "0+", "fields": [ "..." ] }
]
}
validVersions: the range this release can read and write. Old versions are removed from the bottom over time.flexibleVersions: from which version compact types and tagged fields are used.versionson a field: the versions in which the field exists."3+"means it was added in v3.taggedVersionsandtag(not shown): an optional tagged field.
What forces a new version
| Change | New version needed? |
|---|---|
| Add a field that the receiver must understand | Yes |
| Add an optional field in a flexible version | No: use a tagged field |
| Change the meaning of a field or of a response | Yes (for example, Fetch v4 added isolation_level for transactions) |
| A new error code that old clients would not handle | Yes |
| Remove a field | Yes: limit its versions to the old range |
Negotiation
ApiVersions (KIP-35, Kafka 0.10.0) is the first request on every connection, sent even before SASL authentication. The broker returns, for each API, the minimum and maximum versions it supports. If the client's own ApiVersions version is too new, the broker answers with UNSUPPORTED_VERSION and its ranges in a v0 body, and the client retries with an older version.
The client then picks, for each API, the highest version both sides support. The logic in the Java client reduces to this:
// For one API on one connection (simplified from NodeApiVersions.latestUsableVersion)
short chooseVersion(ApiKeys api, ApiVersion broker) {
short low = (short) Math.max(api.oldestVersion(), broker.minVersion()); // both must still support it
short high = (short) Math.min(api.latestVersion(), broker.maxVersion()); // neither may exceed its max
if (low > high) {
// No overlap: the feature that needs this API cannot work against this broker
throw new UnsupportedVersionException("No usable version of " + api);
}
return high;
}
client supports broker supports
Produce 3 ................ 12 3 ............ 11
overlap = 3..11 -> use 11
Fetch 4 ............. 17 0 ...... 12
overlap = 4..12 -> use 12
ConsumerGroupHeartbeat 0..1 (not supported)
no overlap -> group.protocol=consumer fails
Because negotiation is per connection, a client can use different versions with different brokers during a rolling upgrade.
Removing old versions
Supporting every version forever has a cost in code and testing. Kafka 4.0 (KIP-896) removed protocol versions older than those used by Kafka 2.1 clients. Before upgrading, find clients that still use them with the broker metric kafka.network:type=RequestMetrics,name=DeprecatedRequestsPerSec,request=<api>,version=<v>,clientSoftwareName=<name>,clientSoftwareVersion=<version> (available in late 3.x releases).
API versions and feature versions are different things
API versions describe message formats on one connection. Feature versions (metadata.version, kraft.version, transaction.version, group.version, set with kafka-features.sh, Section 2.3) describe behavior the whole cluster has agreed to turn on. Some newer API versions are only advertised once the corresponding feature level is finalized, so after an upgrade, a client may only see the new version after the operator runs kafka-features.sh upgrade.
Key Takeaways
- Each API has its own version, defined in JSON schemas with
validVersionsand per-fieldversions. ApiVersionsis the first request on every connection. Clients then use the highest version both sides support, per API.- Tagged fields allow optional additions without new versions. Semantic changes always need a new version.
- Kafka 4.0 removed versions older than the Kafka 2.1 baseline (KIP-896). Feature versions gate cluster-wide behavior separately.
Practice
- (Beginner) Open
ProduceRequest.jsonandFetchRequest.jsonin the Kafka source on GitHub. What is the oldest valid version of each in your release, and from which version is each flexible?- (Intermediate) Pick one field added to
FetchRequestin a version above 10. Find the KIP that added it (the JSON comments usually name it) and explain why it needed a version bump instead of a tagged field, or why it did not.- Interview question: "How can a Kafka 4.1 client work with a 3.5 broker, given that the client was written years later?" Hint: per-connection negotiation, choosing the maximum common version, and what happens to features that require APIs the broker does not have.
Metadata Requests
Theory
Section 2.2 showed how a client uses metadata to find partition leaders. This topic looks at the Metadata API (key 3) itself: what it contains, how clients cache it, and why it can become expensive on large clusters.
Request and response (current versions)
MetadataRequest
topics[] null = ALL topics; empty = none (brokers and cluster only)
topic_id uuid (or name)
name
allow_auto_topic_creation may the broker create missing topics (if auto.create.topics.enable=true)
include_topic_authorized_operations
MetadataResponse
throttle_time_ms
brokers[] node_id, host, port, rack only endpoints of the listener the request arrived on
cluster_id
controller_id in KRaft: a random live broker, not the real controller (Section 2.2)
topics[]
error_code, name, topic_id, is_internal
partitions[]
error_code, partition_index
leader_id, leader_epoch
replica_nodes, isr_nodes, offline_replicas
topic_authorized_operations
Points worth knowing:
- Any broker answers from its local cache, which in KRaft is the in-memory metadata image built from
__cluster_metadata(Section 16.2). Different brokers may be a few milliseconds apart. leader_epochprotects against stale answers. The client remembers the highest leader epoch it has seen for each partition and ignores metadata with a lower one (KIP-320). Otherwise, a lagging broker could send the client back to a former leader.- Unauthorized topics return
TOPIC_AUTHORIZATION_FAILEDwhen asked for by name, and are silently omitted when the client asks for all topics. - The listener matters. The broker returns the addresses of the same listener name the request came in on, which is why
advertised.listenersmust be right for each listener (Section 2.2).
How clients cache and refresh
| Client | What it requests | When it refreshes |
|---|---|---|
| Producer | Only topics it has sent to recently; topics unused for metadata.max.idle.ms (5 minutes) are dropped | Every metadata.max.age.ms (5 minutes), on leadership errors, and when send() needs an unknown topic (it blocks up to max.block.ms) |
| Consumer, topic list | Subscribed topics | Same triggers |
| Consumer, regex (classic protocol) | All topics, matched against the pattern on the client | Every metadata.max.age.ms |
Consumer, SubscriptionPattern (KIP-848) | The broker evaluates the regex | Driven by the group coordinator |
| Admin client | Whatever each call needs | Per call |
Since Kafka 3.7, a Produce or Fetch error caused by a leader change can carry the new leader directly (KIP-951), so clients often avoid a full metadata round trip. When all known brokers become unreachable, clients can go back to bootstrap.servers (metadata.recovery.strategy=rebootstrap, Section 14.2).
Why metadata can get expensive
A request for all topics on a cluster with 200,000 partitions returns a response of many megabytes that the broker must build on an I/O thread. A few hundred clients refreshing it often can load the brokers noticeably. Common causes:
- Classic consumers with regex subscriptions (each refresh fetches every topic).
- Monitoring scripts or UIs that list all topics every few seconds.
- A very low
metadata.max.age.ms.
Watch kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Metadata and the corresponding TotalTimeMs. For describing very large topics, newer tools use DescribeTopicPartitions (key 75, KIP-966), which returns partitions in pages.
Example: reading metadata from Python
# pip install confluent-kafka
from confluent_kafka.admin import AdminClient
admin = AdminClient({"bootstrap.servers": "localhost:9092"})
md = admin.list_topics(topic="orders", timeout=10) # a Metadata request for one topic
print(f"cluster={md.cluster_id} controller(as reported)={md.controller_id}")
for b in sorted(md.brokers.values(), key=lambda b: b.id):
print(f" broker {b.id}: {b.host}:{b.port}") # comes from advertised.listeners
topic = md.topics["orders"]
if topic.error is not None:
print("topic error:", topic.error) # e.g. UNKNOWN_TOPIC_OR_PART
for p in sorted(topic.partitions.values(), key=lambda p: p.id):
under = len(p.isrs) < len(p.replicas)
print(f" orders-{p.id}: leader={p.leader} replicas={p.replicas} isr={p.isrs}"
+ (" UNDER-REPLICATED" if under else ""))
cluster=MkU3OEVBNTcwNTJENDM2Qk controller(as reported)=2
broker 1: kafka-1.internal:9092
broker 2: kafka-2.internal:9092
broker 3: kafka-3.internal:9092
orders-0: leader=1 replicas=[1, 2, 3] isr=[1, 2, 3]
orders-1: leader=2 replicas=[2, 3, 1] isr=[2, 3] UNDER-REPLICATED
orders-2: leader=3 replicas=[3, 1, 2] isr=[3, 1, 2]
Example: the metadata wait behind a producer's first send
t=0 producer.send(topic="payments") topic not in the cache
t=0 Metadata(topics=[payments], allow_auto_topic_creation=true)
t=3 ms response: payments error=UNKNOWN_TOPIC_OR_PARTITION (auto-creation disabled)
t=103 ms retry after backoff ... same answer
...
t=60 s send() throws TimeoutException:
"Topic payments not present in metadata after 60000 ms." (max.block.ms)
This message almost always means the topic does not exist (and cannot be auto-created), the client lacks Describe on it, or the client cannot reach the advertised listener. It is rarely a performance problem.
Key Takeaways
Metadatareturns brokers (for the requesting listener), cluster ID, and per-partition leader, leader epoch, replicas, and ISR.- Any broker answers from its local metadata image. Leader epochs let clients reject stale responses.
- Producers fetch metadata only for topics in use; classic regex consumers fetch all topics, which is costly on big clusters.
- "Topic not present in metadata after 60000 ms" means the topic is missing, not authorized, or unreachable, not that Kafka is slow.
Practice
- (Beginner) Run the Python example against a topic that does not exist, with
auto.create.topics.enable=falseon the broker. What error do you see? Repeat with it enabled.- (Intermediate) Write a script that calls
list_topics()with no topic (all topics) on your cluster and measures the response time and the number of partitions returned. Estimate how many such requests per second your brokers could handle.- Interview question: "A team's monitoring shows Metadata requests are 40% of all broker requests. What would you look for?" Hint: which clients send them (client IDs in request logs or quota metrics), regex subscriptions,
metadata.max.age.ms, and tools that poll all topics.
Client-Broker Compatibility
Theory
Because versions are negotiated per API, Kafka clients and brokers are bidirectionally compatible: an older client can talk to a newer broker, and a newer client can talk to an older broker. That is what makes independent upgrades possible. Compatibility has limits, though: there is a minimum supported version on each side, and new features need both sides to support them.
The compatibility window
| Combination | Works? | What to expect |
|---|---|---|
| Older client (2.1 or newer) with a 4.x broker | Yes | The client uses the versions it knows. New broker features are simply unused. |
| 4.x Java client with an older broker (2.1 or newer) | Yes | The client lowers versions per API. Features needing APIs the broker lacks fail with UnsupportedVersionException. |
| Client older than 2.1 with a 4.x broker | No | KIP-896 removed the versions these clients use. Requests fail with UNSUPPORTED_VERSION. |
| 4.x Java client with a broker older than 2.1 | No | The client refuses to use the removed versions. |
Features that need a minimum broker version
| Client feature | Needs on the broker side |
|---|---|
| Idempotent producer (default since client 3.0) | 0.11+ (and permission to write, Section 11.3) |
processing.guarantee=exactly_once_v2 (Streams) | 2.5+ |
Fetch from follower (client.rack) | 2.4+ and replica.selector.class set |
| Client metrics push (KIP-714) | 3.7+ and a configured metrics receiver |
group.protocol=consumer (KIP-848) | 4.0+ |
| Share groups (KIP-932) | 4.x with share groups enabled |
Rebootstrap (metadata.recovery.strategy) | Nothing: client-side only |
Non-Java clients
Most other clients also negotiate, but check how:
| Client | Behavior |
|---|---|
| librdkafka (C/C++, and through it confluent-kafka for Python, Go, and .NET) | Negotiates with ApiVersions by default (api.version.request=true) |
| franz-go | Negotiates per broker |
| Sarama (Go) | Uses the protocol level you set in config.Version rather than fully negotiating. Set it to your broker version, or new features stay off. |
Upgrade order
- Upgrade brokers and controllers (rolling).
- Finalize feature versions with
kafka-features.sh upgrade --release-version <v>once you no longer need to downgrade. - Upgrade clients, Connect workers, and Streams applications at their own pace.
- Only then turn on features that need the new versions (for example
group.protocol=consumer).
Finding out who is connected
Since KIP-511 (Kafka 2.4), Java and librdkafka clients send their software name and version in ApiVersions (the client_software_name field from the earlier example). Brokers expose a connection count per name and version:
kafka.server:type=socket-server-metrics,clientSoftwareName=apache-kafka-java,clientSoftwareVersion=2.0.1,listener=PLAINTEXT,networkProcessor=3
connections = 14
Example: pre-flight checks before an upgrade
BS=broker1:9092
# 1. Does the cluster support the APIs a new feature needs? "UNSUPPORTED" means no.
bin/kafka-broker-api-versions.sh --bootstrap-server $BS \
| grep -E "ConsumerGroupHeartbeat|ShareGroupHeartbeat|DescribeTopicPartitions"
# ConsumerGroupHeartbeat(68): 0 to 1 [usable: 1],
# ShareGroupHeartbeat(76): UNSUPPORTED,
# 2. Which client versions are connected right now? (JMX on each broker)
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
--jmx-url service:jmx:rmi:///jndi/rmi://broker1:9999/jmxrmi \
--object-name 'kafka.server:type=socket-server-metrics,clientSoftwareName=*,clientSoftwareVersion=*,listener=*,networkProcessor=*' \
--attributes connections --one-time true
# 3. Is anyone using protocol versions that the next release removes?
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
--jmx-url service:jmx:rmi:///jndi/rmi://broker1:9999/jmxrmi \
--object-name 'kafka.network:type=RequestMetrics,name=DeprecatedRequestsPerSec,*' \
--attributes Count --one-time true
Key Takeaways
- Clients and brokers are compatible in both directions within a window. In Kafka 4.x that window starts at version 2.1 on both sides.
- Compatibility means "can talk", not "every feature works". Features that need new APIs require new brokers too.
- Upgrade servers first, finalize features, then clients. Turn on new client features last.
- Use client software name and version metrics, and deprecated-request metrics, to find old clients before an upgrade.
Practice
- (Beginner) Connect a Kafka 4.x console consumer to a 3.x broker with
--consumer-property group.protocol=consumer. What error do you get, and which API is missing?- (Intermediate) Write a script that scrapes the
socket-server-metricsMBeans from all brokers and prints a table of client software name, version, and total connections, sorted by version. Flag anything older than 2.1.- Interview question: "You must upgrade a 3.4 cluster to 4.1, but a legacy application uses a 0.10.2 Java client that nobody can rebuild. What are your options?" Hint: the 4.x compatibility window, where that client could connect instead (a separate cluster, MirrorMaker 2, a proxy), and how long you can stay on 3.x.
<a id="162-broker-internals"></a>
16.2 Broker Internals
Inside each broker, a few thread pools and data structures turn protocol requests into durable, replicated logs. This section follows a request through them, then looks at the components that work in the background: purgatory, replica fetchers, the log cleaner, and the KRaft controller.
Request Processing Pipeline
Theory
A broker may hold tens of thousands of client connections, and it cannot dedicate a thread to each one. It also must not let a slow disk read stall network I/O for every other client. Kafka therefore uses a staged pipeline (the reactor pattern): a few threads do non-blocking network I/O, a separate pool does the actual work, and queues connect the stages. Section 2.2 introduced the stages. Here we look at what happens in each and how they interact.
Analogy: a busy restaurant. A host seats guests (acceptor), waiters take orders and carry plates but never cook (network threads), orders go on a ticket rail (request queue), cooks prepare them (I/O threads), and dishes that need something else first wait on a shelf (purgatory) so the cook can move on.
The components (Kafka 4.x)
SocketServer
+------------------------------------------------------------------+
| Acceptor (1 per listener endpoint) |
| accept TCP connection, hand it to a Processor (round robin) |
| |
| Processor x num.network.threads (per listener) |
| own NIO Selector; TLS and SASL work; reads size-prefixed |
| requests; parses the header; MUTES the connection |
+--------------------------|---------------------------^-----------+
v |
RequestChannel: shared request queue | per-processor response queue
(bounded by queued.max.requests = 500) |
| |
+--------------------------v---------------------------|-----------+
| KafkaRequestHandler x num.io.threads (shared by all listeners) |
| KafkaApis.handle(request): switch on api_key |
| Produce -> ReplicaManager.appendRecords |
| Fetch -> ReplicaManager.fetchMessages |
| OffsetCommit, JoinGroup ... -> GroupCoordinator |
| CreateTopics ... -> forwarded to the active controller |
+--------------------------|---------------------------------------+
v
DelayedOperationPurgatory (Produce, Fetch, ...)
Controllers run the same SocketServer and handler pool, dispatching to ControllerApis instead of KafkaApis on the listeners named in controller.listener.names.
Life of an acks=all produce request
- Processor reads bytes until a full request has arrived (at most
socket.request.max.bytes, 100 MiB), decrypts TLS, parses the header, and builds a request object carrying the principal, listener, and client address. - It puts the request on the shared request queue and mutes the connection: it stops reading further requests from that socket. This is what guarantees one-request-at-a-time, in-order processing per connection.
- A free I/O thread takes the request.
KafkaApisauthorizes it, andReplicaManagervalidates each batch (CRC, record format, compression, sequence numbers for idempotence), assigns offsets, and appends to the active segment, which lands in the page cache. - The followers have not copied the data yet, so the handler creates a
DelayedProduceand parks it in purgatory. The I/O thread is now free for other work. - When follower fetches move the high watermark past the batch, the delayed operation completes, and its callback puts the response on the response queue of the processor that owns the connection.
- The processor sends the response and unmutes the connection. Only now is the next request from that client read.
Backpressure and throttling
- If the request queue is full (
queued.max.requests), processors block on putting requests, which stops them reading from sockets. TCP backpressure then slows clients down.queued.max.request.bytescan also limit queued memory. - When a client exceeds a quota, the broker computes a throttle time, sends the response immediately with
throttle_time_msset, and keeps the connection muted for that time (KIP-219). Well-behaved clients also wait on their side.
Example: the handler loop (simplified Java)
The core of an I/O thread is a short loop. This is a simplification of KafkaRequestHandler.run():
while (running) {
// Wait up to 300 ms for work; idle time here is what RequestHandlerAvgIdlePercent measures
RequestChannel.BaseRequest req = requestChannel.receiveRequest(300);
if (req == null) continue;
if (req instanceof RequestChannel.ShutdownRequest) {
running = false; // broker is stopping
} else if (req instanceof RequestChannel.Request request) {
request.requestDequeueTimeNanos = time.nanoseconds(); // end of RequestQueueTimeMs
try {
apis.handle(request, requestLocal); // dispatch by api_key; may park work in purgatory
} catch (Throwable t) {
// An unexpected exception becomes an error response, never a dead thread
requestChannel.sendErrorResponse(request, t);
}
}
}
The key design rule is visible here: nothing in handle may block for long. Anything that waits (for followers, for more data, for a rebalance) goes to purgatory instead.
Example: finding what busy I/O threads are doing
PID=$(pgrep -f 'kafka\.Kafka')
jcmd "$PID" Thread.print > /tmp/threads.txt
# How many request handler and network threads exist
grep -c '"data-plane-kafka-request-handler' /tmp/threads.txt
grep -c '"data-plane-kafka-network-thread' /tmp/threads.txt
# What the handlers are doing right now (repeat 3-4 times, a few seconds apart)
grep -A 12 '"data-plane-kafka-request-handler' /tmp/threads.txt \
| grep -E '^\s+at ' | sort | uniq -c | sort -rn | head -15
How to read it:
| Most frequent frames | Meaning |
|---|---|
RequestChannel.receiveRequest / poll | Threads are idle. The bottleneck is elsewhere. |
Partition.appendRecordsToLeader waiting on a lock | Many producers writing to the same hot partition |
FileRecords.read, FileChannel.read | Fetches reading from disk: consumers far behind, page cache misses |
LogValidator with compression classes | Broker recompressing batches: topic compression.type differs from the producer's |
| Authorizer classes | A slow custom authorizer on the hot path (Section 16.3) |
Key Takeaways
- Acceptors accept connections, processors do network I/O, I/O threads do the work, and queues connect them.
- A connection is muted from the moment a request is read until its response is sent, which gives per-connection ordering.
- I/O threads must never block for long. Waiting work moves to purgatory.
- Thread dumps of request handler threads tell you which stage is saturated and why.
Practice
- (Beginner) On a local broker, list the acceptor, network, and request handler threads in a thread dump. Change
num.io.threadsdynamically and confirm the number of handler threads changes.- (Intermediate) Explain why a client with
max.in.flight.requests.per.connection=5still has its requests processed one at a time by the broker. Where do requests 2 to 5 wait, and what does this mean for latency?- Interview question: "
RequestQueueTimeMsfor Produce is high, butRequestHandlerAvgIdlePercentis 0.6. How can both be true?" Hint: the request queue is shared by all request types; consider bursts, a few slow requests holding threads, and how averages hide spikes.
Purgatory and Delayed Operations
Theory
Many requests cannot be answered immediately:
- An
acks=allproduce must wait until the followers have the data. - A fetch with
fetch.min.bytes=64KBmust wait until that much data exists, or untilfetch.max.wait.mspasses. - A
DeleteRecordsrequest must wait until every replica has moved its log start offset.
If an I/O thread waited for each of these, eight threads would be exhausted almost immediately. Instead, the broker wraps the waiting request in a delayed operation and puts it in a purgatory: a holding area that completes the operation when its condition becomes true, or expires it after a timeout. The I/O thread is free in microseconds.
Analogy: a coat check. You do not stand at the counter while your coat is fetched from the back. You get a ticket, and you are called when the coat is ready or told after a while that it cannot be found.
The pieces
| Piece | Role |
|---|---|
DelayedOperation | One waiting request. Implements tryComplete() (check the condition), onComplete() (send the response), onExpiration() (record the timeout) |
| Watch keys | What the operation waits on, usually a partition. One operation may watch several keys (a produce to 10 partitions). |
| Watcher lists | Per key, the operations to re-check when that key changes |
| Timer | Expires operations at their deadline, using a hierarchical timing wheel |
| Expiration reaper thread | Advances the timer and runs expirations (ExpirationReaper-<brokerId>-Produce, -Fetch, ...) |
The protocol between a request and the purgatory
tryCompleteElseWatch(op, keys):
if op.tryComplete(): return # common case: already satisfiable
for key in keys: watch(key, op)
if op.tryComplete(): return # re-check: the condition may have changed
# between the first check and watching
timer.add(op) # otherwise wait for an event or the deadline
checkAndComplete(key): # called by whoever changes the state behind `key`
for op in watchers(key): op.tryComplete()
forceComplete(): # used by both completion and expiration
atomically set completed = true; only the winner runs onComplete()
Events that call checkAndComplete:
| Event | Wakes up |
|---|---|
| High watermark of a partition advances | Delayed produces for that partition; delayed consumer fetches (consumers only see data below the HW) |
| New data appended to the leader's log | Delayed follower fetches |
| Leadership change | Both, so they complete with NOT_LEADER_OR_FOLLOWER instead of waiting for the timeout |
The purgatories in a broker
| Purgatory | Completes when | Expires after |
|---|---|---|
Produce (DelayedProduce) | HW passes the last appended offset in every partition, or a partition fails | The request's timeout_ms (the producer's request.timeout.ms), then REQUEST_TIMED_OUT |
Fetch (DelayedFetch) | At least fetch.min.bytes available, or an error | fetch.max.wait.ms (consumers), replica.fetch.wait.max.ms (followers) |
| DeleteRecords | All replicas' log start offsets reach the requested offset | Request timeout |
| ElectLeader | Requested leaders are elected | Request timeout |
| RemoteFetch | A tiered-storage read finishes (Section 7.3) | remote.fetch.max.wait.ms |
| ShareFetch | Records available for a share group (Section 5.4) | Request wait time |
Why a timing wheel
A busy broker may have hundreds of thousands of pending operations: every idle consumer has a fetch waiting, and most of them are completed early and never expire. A priority queue costs O(log n) per insert and keeps cancelled entries around. A timing wheel inserts and removes in O(1).
Level 0: tick 1 ms, 20 slots -> covers 20 ms
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| | |op| | | |op| | | | | | | | | | | | | | each slot = list of operations
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
^ current time
Level 1: tick 20 ms, 20 slots -> covers 400 ms (overflow wheel, created on demand)
Level 2: tick 400 ms, 20 slots -> covers 8 s
Level 3: tick 8 s, 20 slots -> covers 160 s ...
- An operation due in 7 ms goes into a level-0 slot; one due in 30 s goes into a level-3 slot.
- Only non-empty SLOTS go into a DelayQueue, so the queue is tiny.
- When a higher-level slot expires, its operations are re-inserted into lower levels
(like a clock's hour hand handing over to the minute hand).
- Completing an operation early just unlinks it from its slot: O(1).
Example: a delayed produce (simplified Java)
class DelayedProduce extends DelayedOperation {
private final Map<TopicPartition, Long> requiredOffsets; // last appended offset + 1, per partition
private final Map<TopicPartition, Errors> status = new HashMap<>();
private final Consumer<Map<TopicPartition, Errors>> responseCallback;
@Override
public boolean tryComplete() {
for (var entry : requiredOffsets.entrySet()) {
TopicPartition tp = entry.getKey();
if (status.containsKey(tp)) continue; // already decided for this partition
Partition partition = replicaManager.onlinePartition(tp);
if (partition == null || !partition.isLeader()) {
status.put(tp, Errors.NOT_LEADER_OR_FOLLOWER); // an error also counts as "done"
} else if (partition.highWatermark() >= entry.getValue()) {
status.put(tp, Errors.NONE); // replicated to the whole ISR
} else {
return false; // still waiting: stay in purgatory
}
}
return forceComplete(); // exactly one of completion or expiration wins
}
@Override
public void onExpiration() {
// Partitions not decided yet timed out; the producer will retry them
requiredOffsets.keySet().forEach(tp -> status.putIfAbsent(tp, Errors.REQUEST_TIMED_OUT));
}
@Override
public void onComplete() {
responseCallback.accept(status); // builds the ProduceResponse and queues it for the processor
}
}
Reading the metrics
| Metric | Normal | Investigate when |
|---|---|---|
kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Fetch | Roughly the number of connected consumers and followers | Never alarming alone; idle consumers always wait here |
Same with delayedOperation=Produce | Near 0 to low hundreds | Growing steadily: followers are slow, so acks=all requests wait |
kafka.server:type=DelayedOperationPurgatory,name=NumDelayedOperations,... | Operations still pending | Compare with PurgatorySize, which also counts completed operations not yet purged from watcher lists |
Produce RemoteTimeMs | Low milliseconds | High: time spent in the produce purgatory, so look at replication |
For example, 2,000 consumers on quiet topics with fetch.max.wait.ms=500 keep about 2,000 delayed fetches in purgatory and cause about 4,000 expirations per second. With a timing wheel this costs almost nothing, so a large fetch purgatory is normal.
Key Takeaways
- Purgatory holds requests that must wait, so I/O threads never block on replication or empty partitions.
- A delayed operation completes when an event makes its condition true (HW advances, new data, leader change) or expires at its deadline.
forceCompleteensures it completes exactly once. - Hierarchical timing wheels make adding, completing, and expiring operations O(1).
- A large fetch purgatory is normal. A growing produce purgatory means slow followers.
Practice
- (Beginner) Start a consumer on an empty topic with
fetch.min.bytes=1048576andfetch.max.wait.ms=5000. Produce one small record. How long does the consumer take to receive it, and why?- (Intermediate) Implement a single-level timing wheel in Python (20 slots, 1 ms ticks) with
add(task, delay_ms)andadvance(now_ms). Then add an overflow wheel and test a task with a 300 ms delay.- Interview question: "Why does
tryCompleteElseWatchcalltryCompletetwice?" Hint: imagine the high watermark advancing on another thread after the first check returns false but before the operation is added to the watcher list.
Replica Fetcher Threads
Theory
Followers copy data by pulling from leaders with Fetch requests (Section 2.4). The replica fetcher threads do this work, and their organization explains replication lag patterns, thread counts, and why some partitions fall behind while others on the same broker keep up.
How fetchers are organized
- The
ReplicaFetcherManageron each broker creates fetcher threads per source broker (the leader being copied from):num.replica.fetchersthreads (default 1) for each broker that leads at least one partition this broker follows. - Each thread is named
ReplicaFetcherThread-<fetcherId>-<leaderBrokerId>, which is also theclientIdin fetcher metrics. - A partition is assigned to a fetcher by hash:
fetcherId = abs(31 * topic.hashCode() + partition) % num.replica.fetchers
Consecutive partitions of one topic go to consecutive fetchers, so a large topic spreads evenly. Skew comes from many small topics hashing to the same fetcher.
- Thread count per broker =
num.replica.fetchersx (number of other brokers it follows). On a 30-broker cluster withnum.replica.fetchers=4, a broker can run up to 116 fetcher threads.
One fetcher thread's loop
loop:
1. Build one FetchRequest for all partitions this thread owns (an incremental fetch session,
so unchanged partitions are omitted; Chapter 13)
replica_state = (this broker's id, its broker epoch)
max_wait_ms = replica.fetch.wait.max.ms (500)
min_bytes = replica.fetch.min.bytes (1)
max_bytes = replica.fetch.response.max.bytes (10 MiB)
per partition: fetch_offset = my LEO, current_leader_epoch, partition_max_bytes
= replica.fetch.max.bytes (1 MiB)
2. Send it to the leader. The leader may park it in the fetch purgatory until data arrives.
3. For each partition in the response:
- append the records at the LEADER's offsets (followers never assign offsets)
- follower HW = min(leader HW in the response, my LEO)
- update the log start offset (retention on the leader propagates this way)
- on a diverging epoch: truncate the log to the point where it diverged, then continue
- on OFFSET_MOVED_TO_TIERED_STORAGE: rebuild state from remote storage (Section 7.3)
- on NOT_LEADER / FENCED_LEADER_EPOCH: pause the partition until new metadata arrives
4. On errors, back off replica.fetch.backoff.ms (1 s) for the affected partitions.
On the leader, each follower fetch carries the follower's log end offset. The leader records it, may add the follower back to the ISR (an AlterPartition request to the controller), and may advance the high watermark. Advancing the HW then completes delayed produces and consumer fetches.
Why acks=all costs about one extra fetch round trip
time leader (LEO / HW) follower producer
t0 append 100 -> LEO 101 (waiting in fetch Produce(acks=all) parked in purgatory
HW 100 purgatory, offset 100)
t1 new data completes the receives offset 100,
follower's delayed fetch appends -> LEO 101
t2 Fetch(offset=101) arrives: follower has 100
-> HW 101 (gets HW 101 in this DelayedProduce completes -> response
response or the next)
The follower's next fetch is what tells the leader it has the data. Because the follower's request is already waiting in the leader's purgatory, the record leaves immediately, and replica.fetch.wait.max.ms adds no latency while data is flowing. The follower's own HW is always slightly behind the leader's, which is harmless because only the leader's HW decides what consumers can read.
Failures and throttling
- If one partition hits an unexpected error (for example a corrupt local log), the fetcher marks that partition failed and keeps replicating the others (KIP-461). The partition stays under-replicated until its leadership changes or the broker restarts. Watch
FailedPartitionsCount. - Replication quotas (
leader.replication.throttled.rate,follower.replication.throttled.rate, and the throttled replica lists) limit fetchers during reassignments (Section 12.2). - The same machinery, as
ReplicaAlterLogDirsThread, moves a replica between disks on one broker.
Example: predicting fetcher skew (Python)
This script reproduces the broker's assignment for the partitions broker 2 follows from leader 1, and shows how load spreads over fetchers:
from collections import Counter
def to_int32(n: int) -> int:
n &= 0xFFFFFFFF
return n - (1 << 32) if n >= 1 << 31 else n
def java_string_hash(s: str) -> int:
h = 0
for ch in s: # ASCII topic names: one UTF-16 code unit per character
h = to_int32(31 * h + ord(ch))
return h
def fetcher_id(topic: str, partition: int, num_fetchers: int) -> int:
v = to_int32(31 * java_string_hash(topic) + partition)
v = 0 if v == -(1 << 31) else abs(v) # Kafka's Utils.abs
return v % num_fetchers
# Partitions led by broker 1 and followed by broker 2, with their inbound MB/s
followed = [("clicks", p, 8.0) for p in range(0, 24, 3)] + \
[("audit", 0, 25.0), ("billing", 0, 2.0), ("billing", 3, 2.0), ("geo", 1, 30.0)]
for n in (1, 2, 4):
load, parts = Counter(), Counter()
for topic, partition, mb_s in followed:
f = fetcher_id(topic, partition, n)
load[f] += mb_s
parts[f] += 1
print(f"num.replica.fetchers={n}: " +
", ".join(f"fetcher {f}: {parts[f]} partitions {load[f]:.0f} MB/s" for f in sorted(load)))
num.replica.fetchers=1: fetcher 0: 12 partitions 123 MB/s
num.replica.fetchers=2: fetcher 0: 6 partitions 64 MB/s, fetcher 1: 6 partitions 59 MB/s
num.replica.fetchers=4: fetcher 0: 4 partitions 48 MB/s, fetcher 1: 3 partitions 18 MB/s, fetcher 2: 2 partitions 16 MB/s, fetcher 3: 3 partitions 41 MB/s
The clicks partitions spread evenly, but with four threads the load is still uneven (48 MB/s against 16 MB/s), because the heavy single-partition topics geo and audit land on fetchers 0 and 3 on top of their share of clicks. Each thread handles its partitions in sequence, so the busiest thread determines the lag. Adding threads helps only as far as it separates the heavy partitions.
Metrics
| Metric | Meaning |
|---|---|
kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica | Maximum lag in messages over all followed partitions on this broker |
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=ReplicaFetcherThread-0-1,topic=...,partition=... | Lag of one partition, which identifies the fetcher thread |
kafka.server:type=FetcherStats,name=BytesPerSec,clientId=ReplicaFetcherThread-0-1,... | Throughput per fetcher thread: look for one saturated thread |
kafka.server:type=ReplicaFetcherManager,name=FailedPartitionsCount,clientId=Replica | Partitions the fetchers gave up on. Must be 0. |
Key Takeaways
- Each broker runs
num.replica.fetchersfetcher threads per leader broker it follows. Partitions are hashed to threads by topic and partition number. - A fetcher sends one fetch for all its partitions, appends at the leader's offsets, and takes its HW from the leader.
- Follower fetches drive the leader's HW, which completes
acks=allproduces. Their cost is roughly one extra fetch round trip. - Lag concentrated on one fetcher thread means skewed assignment. Failed partitions stay under-replicated until leadership changes.
Practice
- (Beginner) On a three-broker cluster, count the
ReplicaFetcherThreadthreads on one broker withnum.replica.fetchers=1, then with2. Explain the numbers.- (Intermediate) Extend the Python script to read real data: use
kafka-topics.sh --describeoutput to find which partitions each broker follows from each leader, and print the expected partitions per fetcher thread.- Interview question: "One follower keeps dropping out of the ISR for only two partitions, although its disk and network are fine. How would you investigate?" Hint: per-partition
ConsumerLagmetrics to identify the fetcher thread, that thread's throughput, what else hashes to it, andFailedPartitionsCount.
Log Cleaner Internals
Theory
Section 7.2 explained what compaction guarantees and the basic algorithm: build an offset map, then recopy segments keeping only the latest record per key. This topic follows the cleaner through one full cycle: how it picks a partition, which part of the log it may touch, how it treats tombstones, transactions, and producer state, and what happens when it fails.
Components
| Component | Role |
|---|---|
LogCleaner | Owns log.cleaner.threads threads, named kafka-log-cleaner-thread-<n> |
LogCleanerManager | Tracks partition state (in progress, paused, uncleanable) and the cleaner-offset-checkpoint file in each log directory |
Cleaner | Runs the algorithm for one partition |
Offset map (SkimpyOffsetMap) | Open-addressing hash table: 16-byte MD5 of the key to an 8-byte offset |
| Throttler | Enforces log.cleaner.io.max.bytes.per.second |
Step 1: choose a partition
Each cleaner thread repeatedly picks the dirtiest eligible partition:
cleanable range = [firstDirtyOffset, firstUncleanableOffset)
firstDirtyOffset = cleaner checkpoint (or the log start offset)
firstUncleanableOffset = min( base offset of the active segment, never cleaned
first unstable offset (LSO), open transactions stay
first segment newer than min.compaction.lag.ms )
dirty ratio = bytes in cleanable range / (bytes already clean + bytes in cleanable range)
eligible if dirty ratio > min.cleanable.dirty.ratio
or some record has waited longer than max.compaction.lag.ms (these go first)
Partitions are skipped while paused (for example during deletion or a log directory move) or after being marked uncleanable.
Step 2: build the offset map
The cleaner scans the cleanable range and stores, for each key, the latest offset. Records of aborted transactions and control batches are not added. If the map fills up (its capacity is log.cleaner.dedupe.buffer.size / log.cleaner.threads x the load factor 0.9 / 24 bytes per entry), the cleaner stops at a segment boundary and cleans only up to there. The rest waits for the next pass.
Step 3: recopy segments
Segments from the log start up to the end of the mapped range are grouped so that each group fits in one new segment (segment.bytes, index size limits). Each group is rewritten into a .cleaned file. For every record:
| Record | Kept? |
|---|---|
| Key not in the map, or offset >= the map's offset for that key | Yes: it is the latest value |
| Older value of a key that appears later | No |
| Tombstone (null value) that is the latest for its key | Yes, until its delete horizon passes. The first pass that retains it stamps the batch with a horizon of now + delete.retention.ms (the "delete horizon" attribute from Section 7.1). A later pass drops it. |
| Record from an aborted transaction | No |
| Transaction marker (commit or abort) | Yes, until all data of that transaction is gone and its horizon passes |
| Last batch written by each active producer ID | The batch is kept (possibly emptied of records) so the broker can rebuild idempotent producer state after a restart |
Step 4: swap and checkpoint
The cleaned file is renamed .cleaned to .swap, swapped in place of the old segments, and the old files are renamed .deleted and removed after file.delete.delay.ms. Finally, the checkpoint is advanced to the end of the cleaned range.
When cleaning fails
If the cleaner hits an unexpected error on a partition (for example a corrupt batch), it marks that partition uncleanable, logs the error to log-cleaner.log, and continues with other partitions. The partition then grows without bound until the cause is fixed and the broker restarts, or the partition is moved. Older releases could lose the whole cleaner thread instead, which is why DeadThreadCount still exists.
Example: one cleaning pass in Python
This model applies steps 1 to 3 to a small log. It ignores transactions and producer state to keep the core logic visible.
from dataclasses import dataclass
@dataclass
class Rec:
offset: int
key: str
value: str | None # None = tombstone
horizon: int | None = None # delete horizon stamped by an earlier pass (ms)
def clean(log, active_base, lso, now, delete_retention_ms, checkpoint):
first_uncleanable = min(active_base, lso) # min.compaction.lag ignored here
# Step 2: offset map over the dirty range [checkpoint, first_uncleanable)
latest = {}
for r in log:
if checkpoint <= r.offset < first_uncleanable:
latest[r.key] = r.offset # later offsets overwrite earlier ones
# Step 3: recopy everything below first_uncleanable
out = []
for r in log:
if r.offset >= first_uncleanable:
out.append(r) # active segment / unstable data: untouched
continue
if r.key in latest and r.offset < latest[r.key]:
continue # superseded by a newer record
if r.value is None: # a tombstone that is the latest for its key
if r.horizon is None:
r.horizon = now + delete_retention_ms # first pass: keep it, start the countdown
elif now >= r.horizon:
continue # horizon passed: drop it for good
out.append(r)
return out, first_uncleanable # new log, new checkpoint
log = [Rec(0, "a", "1"), Rec(1, "b", "1"), Rec(2, "a", "2"), Rec(3, "b", None),
Rec(4, "c", "1"), Rec(5, "a", "3")]
log, cp = clean(log, active_base=5, lso=6, now=0, delete_retention_ms=86_400_000, checkpoint=0)
print([(r.offset, r.key, r.value) for r in log], "checkpoint", cp)
# [(2,'a','2'), (3,'b',None), (4,'c','1'), (5,'a','3')] checkpoint 5
# offset 5 is in the active segment, so 'a'=2 survives this pass
log, cp = clean(log, active_base=6, lso=6, now=90_000_000, delete_retention_ms=86_400_000, checkpoint=cp)
print([(r.offset, r.key, r.value) for r in log], "checkpoint", cp)
# [(4,'c','1'), (5,'a','3')] checkpoint 6
# 'a'=2 is now superseded, and the tombstone for 'b' is past its horizon
Example: reading log-cleaner.log
[kafka-log-cleaner-thread-0]:
Log cleaner thread 0 cleaned log customer-profiles-3 (dirty section = [1200000, 2400000])
1,024.0 MB of log processed in 6.3 seconds (162.5 MB/sec).
Indexed 1,000.0 MB in 3.1 seconds (322.6 Mb/sec, 49.2% of total time)
Buffer utilization: 42.0%
Cleaned 1,024.0 MB in 3.2 seconds (320.0 Mb/sec, 50.8% of total time)
Start size: 1,024.0 MB (2,400,000 messages)
End size: 512.0 MB (1,200,000 messages)
50.0% size reduction (50.0% fewer messages)
(Abridged.) "Buffer utilization" near 100% means the offset map limited this pass. A low size reduction on every pass means most keys are unique, so compaction is costing I/O without saving much space.
| Metric | Investigate when |
|---|---|
kafka.log:type=LogCleanerManager,name=max-dirty-percent | Stays high: cleaners cannot keep up |
kafka.log:type=LogCleanerManager,name=time-since-last-run-ms | Keeps growing: cleaner stuck or dead |
kafka.log:type=LogCleanerManager,name=uncleanable-partitions-count,logDirectory=... | Above 0: check log-cleaner.log for the error |
kafka.log:type=LogCleaner,name=max-buffer-utilization-percent | Near 100: raise log.cleaner.dedupe.buffer.size |
kafka.log:type=LogCleaner,name=cleaner-recopy-percent | High: most work recopies data that was already clean |
kafka.log:type=LogCleaner,name=DeadThreadCount | Above 0 |
Key Takeaways
- The cleaner picks the dirtiest partition and cleans only
[checkpoint, min(active segment, LSO, compaction lag)). - The offset map's size limits how much one pass can clean. Large key sets need a bigger dedupe buffer or more passes.
- Tombstones survive one horizon (
delete.retention.ms) after first being cleaned. Aborted data is removed, and each producer's last batch is kept for idempotence. - A failing partition becomes uncleanable and grows without bound. Alert on
uncleanable-partitions-countandtime-since-last-run-ms.
Practice
- (Beginner) Using the Python model, add a record
Rec(6, "b", "2")after the tombstone and show how the outcome for keybchanges.- (Intermediate) Add transactions to the model: give records a producer ID and a transaction state (committed, aborted, open). Implement the rules from the table, including keeping open transactions untouched via the LSO.
- Interview question: "A compacted topic keeps growing although
cleanup.policy=compactis set and keys repeat often. List the possible causes." Hint: the active segment never being rolled, an open transaction holding the LSO,min.compaction.lag.ms, uncleanable partitions, and cleaner throughput versus write rate.
KRaft Controller Internals
Theory
Section 2.3 explained KRaft as a concept: metadata is a Raft-replicated log, and brokers replay it. This topic looks inside the active controller, which must turn concurrent admin requests and broker heartbeats into a single ordered log, and inside the broker, which must apply that log quickly and safely.
Analogy: a court clerk. Requests arrive from many people, but the clerk handles them one at a time, writes each decision in the official record, and only tells the requester "done" once the record has been countersigned by a majority of the other clerks.
A single-threaded event loop
The QuorumController processes every operation on one thread (quorum-controller-<id>-event-handler) fed by an event queue. Serial processing removes the need for locks and makes every decision deterministic. Each write operation goes through the same steps:
CreateTopics / AlterPartition / BrokerHeartbeat / ... (from the network, via ControllerApis)
|
v
+---------------+ 1. run the operation against in-memory state
| event queue | -> ControllerResult = (records to write, response)
+-------+-------+ 2. apply the records to in-memory "timeline" structures
| immediately (speculative state)
v 3. append the records to the Raft log as one batch
Raft log (leader) 4. park the response in a deferred queue
| 5. when the high watermark passes the batch (majority has it),
v complete the deferred response
committed 6. if leadership is lost first, revert the in-memory state
to the last committed offset
Two ideas make this fast and safe:
- Pipelining. The controller does not wait for a batch to commit before processing the next event, so many operations are in flight at once.
- Timeline data structures. In-memory state (
TimelineHashMapand similar, managed by aSnapshotRegistry) can be read "as of" any recent offset and reverted to the committed offset. That allows speculative updates without risking uncommitted state if the controller loses leadership.
The work is split across managers that share the event thread:
| Manager | Owns |
|---|---|
ClusterControlManager | Broker registrations, epochs, fencing |
ReplicationControlManager | Topics, partitions, leaders, ISR, ELR, reassignments |
ConfigurationControlManager | Dynamic configurations |
FeatureControlManager | Finalized feature levels |
AclControlManager, ClientQuotaControlManager, ScramControlManager | ACLs, quotas, SCRAM credentials |
ProducerIdControlManager | Blocks of producer IDs handed to brokers |
Operations whose records do not fit in one batch (for example a topic with thousands of partitions) are wrapped in BeginTransactionRecord and EndTransactionRecord (KIP-868), and brokers apply them only once the end record arrives, so nobody sees half a topic.
Broker lifecycle seen from the controller
broker starts --BrokerRegistration--> controller writes RegisterBrokerRecord (fenced)
broker replays metadata; each BrokerHeartbeat reports its current metadata offset
caught up? --> controller writes UnfenceBrokerRecord: broker may lead partitions
heartbeats every broker.heartbeat.interval.ms (2 s)
no heartbeat for broker.session.timeout.ms (9 s) --> FenceBrokerRecord + leader elections
controlled shutdown: heartbeat with wantShutDown=true --> controller moves leadership away,
then answers shouldShutDown=true
Heartbeats go through the same event queue as everything else. A controller overloaded by a burst of admin operations therefore processes heartbeats late, which can fence healthy brokers. That is why EventQueueTimeMs is the first controller metric to watch.
Applying metadata on brokers
Brokers replicate __cluster_metadata as Raft observers. A MetadataLoader turns each committed batch into a MetadataDelta, combines it with the current immutable MetadataImage to produce a new image, and hands both to publishers:
| Publisher | Effect of a delta |
|---|---|
| Replica manager | Become leader or follower for changed partitions, start or stop fetchers |
| Metadata cache | Answer Metadata requests from the new image |
| Dynamic config publisher | Apply changed broker and topic configurations |
| ACL publisher | Update StandardAuthorizer |
| Feature and quota publishers | New feature levels, quota values |
Because images are immutable, request threads read metadata without locks: they simply use whichever image is current.
Snapshots and log size
Every node (controllers and brokers) independently writes a snapshot of its in-memory state after metadata.log.max.record.bytes.between.snapshots (20 MiB of new records) or metadata.log.max.snapshot.interval.ms (1 hour). Snapshot files are named <offset>-<epoch>.checkpoint. Log segments older than the latest snapshot are deleted according to metadata.max.retention.bytes and metadata.max.retention.ms. A restarting node loads the newest snapshot and replays only what came after.
Example: exploring the metadata image
kafka-metadata-shell.sh mounts a snapshot (or a log directory) as a read-only file system. Run it on a copy of the files or on a stopped node:
bin/kafka-metadata-shell.sh \
--snapshot /var/lib/kafka/meta/__cluster_metadata-0/00000000000000412337-0000000014.checkpoint
>> ls /image
acls clientQuotas cluster configs delegationTokens features producerIds provenance scram topics
>> ls /image/cluster/brokers
1 2 3
>> cat /image/cluster/brokers/2
BrokerRegistration(id=2, epoch=88, incarnationId=..., listeners=[...], rack=Optional[use1-az2],
fenced=false, inControlledShutdown=false, ...)
>> ls /image/topics/byName/orders
0 1 2 id name
>> cat /image/topics/byName/orders/1
PartitionRegistration(replicas=[2, 3, 1], isr=[2, 3], leader=2, leaderEpoch=17, partitionEpoch=41, ...)
(Output abridged; node names vary slightly between releases.) The snapshot file name tells you this image is as of offset 412337 in epoch 14.
Example: controller health metrics
| Metric | Healthy | Meaning when unhealthy |
|---|---|---|
kafka.controller:type=KafkaController,name=ActiveControllerCount | Sums to 1 across controllers | 0: no leader (quorum lost); above 1 briefly during elections only |
kafka.controller:type=ControllerEventManager,name=EventQueueTimeMs | Low milliseconds at p99 | Events wait for the single thread: heartbeats delayed, fencing risk |
kafka.controller:type=ControllerEventManager,name=EventQueueProcessingTimeMs | Low milliseconds | Individual operations are slow (huge topic creation, large reassignments) |
kafka.controller:type=KafkaController,name=TimedOutBrokerHeartbeatCount | Flat | Rising: brokers fenced because heartbeats timed out |
kafka.server:type=raft-metrics, attribute commit-latency-avg | Low milliseconds | Slow disks or network among controllers |
kafka.server:type=broker-metadata-metrics, attributes metadata-apply-error-count, metadata-load-error-count | 0 | A broker failed to apply a metadata record: its view may be wrong |
Key Takeaways
- The active controller processes all operations on one event thread, applies records speculatively, and answers only after the records commit.
- Timeline data structures allow pipelining and roll back uncommitted state if leadership is lost.
- Brokers turn committed batches into immutable metadata images and publish deltas to the replica manager, caches, configs, and the authorizer.
- Snapshots bound restart time.
EventQueueTimeMsis the early warning for an overloaded controller.
Practice
- (Beginner) Open a snapshot with
kafka-metadata-shell.shand find the leader, ISR, and leader epoch of one partition. Compare withkafka-topics.sh --describe.- (Intermediate) Create a topic with 5,000 partitions while watching
EventQueueTimeMsandEventQueueProcessingTimeMs. Then usekafka-dump-log.sh --cluster-metadata-decoderto find the transaction records that wrapped the creation.- Interview question: "Why does the KRaft controller apply records to memory before they are committed, and how does it avoid serving or keeping wrong state?" Hint: pipelining throughput, deferred responses until the high watermark passes, and reverting timeline structures on leadership loss.
<a id="163-extensibility-and-emerging-features"></a>
16.3 Extensibility and Emerging Features
Kafka changes in two ways: through plug-in code that runs inside clients and servers via stable public interfaces, and through changes to Kafka itself via the KIP process. This section covers both, and ends with the largest architectural change currently being debated: moving the log onto object storage.
Producer and Consumer Interceptors
Theory
Some logic belongs to the platform, not to any one application: adding a trace or origin header to every record, counting failed sends, measuring end-to-end latency. Copying that code into every producer and consumer is repetitive and easy to forget. Interceptors are classes that the client calls at fixed points in its pipeline, configured with one property (interceptor.classes) and invisible to application code.
Analogy: the security and stamping desk in a mail room. Every outgoing letter passes it before being sorted, and every incoming letter passes it before being delivered, without the writer or reader doing anything.
Where they run
The pipelines in Sections 4.1 and 5.1 show interceptors as the first stage of the producer and the last stage of the consumer:
| Method | Called | Thread | Can change |
|---|---|---|---|
ProducerInterceptor.onSend(record) | In send(), before serialization and partitioning | The application thread calling send() | Topic, key, value, headers, partition. Changing the key changes the partition. |
ProducerInterceptor.onAcknowledgement(metadata, exception) | When the broker acknowledges or the send fails | Usually the producer's I/O thread | Nothing; observe only |
ConsumerInterceptor.onConsume(records) | Just before poll() returns | The application thread calling poll() | The records returned to the application |
ConsumerInterceptor.onCommit(offsets) | After offsets are committed successfully | The thread that completes the commit | Nothing; observe only |
Rules that follow from the contract:
- Order: interceptors run in the order listed in
interceptor.classes, and each receives the output of the previous one. - Exceptions are swallowed: if an interceptor throws, the client logs it and continues (with the record from the previous interceptor for
onSend). An interceptor cannot stop a send by throwing. onAcknowledgementmust be fast. It runs on the thread that sends every batch for the producer, so blocking it delays all sends.- Thread safety: a producer is used from many threads, so a producer interceptor must be thread-safe.
- Configuration: interceptors implement
Configurableand receive the client's full configuration, including custom keys you add for them. ImplementingClusterResourceListenergives them the cluster ID once metadata arrives. - Headers are writable only in
onSend. The producer makes headers read-only right after interceptors run.
What interceptors are and are not for
| Good fit | Poor fit |
|---|---|
| Adding tracing, origin, or schema hint headers | Business logic (hidden from the code that owns the behavior) |
| Counting failures, measuring latency, auditing | Encryption of values: interceptors see objects, not serialized bytes; use a serializer instead |
| Enforcing simple conventions (required headers) | Filtering records in onConsume: committed offsets still move past them, so they are silently skipped |
| Monitoring agents (many vendors ship interceptors) | Anything slow or blocking (network calls, synchronous logging to remote systems) |
Other places with similar hooks:
- Kafka Streams and Connect pass interceptors through prefixed client configurations, for example
StreamsConfig.producerPrefix(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG). - librdkafka supports interceptors written in C, loaded with
plugin.library.paths. - Brokers have no interceptors in Apache Kafka. Broker-side checks use policies and authorizers (next topic), or a Kafka-protocol proxy such as Kroxylicious that runs filters between clients and brokers.
Example: origin headers and failure counting (Java)
package com.acme.kafka.interceptors;
import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.atomic.LongAdder;
public class OriginHeaderInterceptor implements ProducerInterceptor<Object, Object> {
private byte[] origin;
private final LongAdder failures = new LongAdder(); // contention-free counter for many threads
@Override
public void configure(Map<String, ?> configs) {
// Custom keys in the producer config are passed through to interceptors
Object value = configs.get("acme.origin.service");
origin = String.valueOf(value == null ? "unknown" : value).getBytes(StandardCharsets.UTF_8);
}
@Override
public ProducerRecord<Object, Object> onSend(ProducerRecord<Object, Object> record) {
// Runs on the caller's thread before serialization; keep it cheap
record.headers()
.add("x-origin", origin)
.add("x-sent-at", Long.toString(System.currentTimeMillis()).getBytes(StandardCharsets.UTF_8));
return record; // same key and value, so the partition does not change
}
@Override
public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
// Runs on the producer's I/O thread: count, never block
if (exception != null) {
failures.increment();
}
}
@Override
public void close() {
System.err.printf("OriginHeaderInterceptor: %d failed sends%n", failures.sum());
}
}
Example: end-to-end latency on the consumer side
package com.acme.kafka.interceptors;
import org.apache.kafka.clients.consumer.ConsumerInterceptor;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.header.Header;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.atomic.AtomicLong;
public class LatencyInterceptor implements ConsumerInterceptor<Object, Object> {
private final AtomicLong maxLatencyMs = new AtomicLong();
@Override
public ConsumerRecords<Object, Object> onConsume(ConsumerRecords<Object, Object> records) {
long now = System.currentTimeMillis();
for (ConsumerRecord<Object, Object> r : records) {
Header sentAt = r.headers().lastHeader("x-sent-at");
if (sentAt == null) continue; // record from a producer without the interceptor
long latency = now - Long.parseLong(new String(sentAt.value(), StandardCharsets.UTF_8));
maxLatencyMs.accumulateAndGet(latency, Math::max); // in practice: record into a histogram
}
return records; // pass everything through unchanged
}
@Override
public void onCommit(Map<TopicPartition, OffsetAndMetadata> offsets) { }
@Override
public void configure(Map<String, ?> configs) { }
@Override
public void close() {
System.err.printf("LatencyInterceptor: max end-to-end latency %d ms%n", maxLatencyMs.get());
}
}
# producer.properties
interceptor.classes=com.acme.kafka.interceptors.OriginHeaderInterceptor
acme.origin.service=checkout-api
# consumer.properties
interceptor.classes=com.acme.kafka.interceptors.LatencyInterceptor
The latency is only as accurate as the clocks of the producer and consumer hosts. Keep them synchronized with NTP, and treat values within a few milliseconds as noise.
Key Takeaways
- Interceptors are platform hooks configured with
interceptor.classes. They run in order and cannot fail a send by throwing. onSendruns before serialization and partitioning on the caller's thread.onAcknowledgementruns on the I/O thread and must not block.- Use them for headers, metrics, and auditing, not for business logic, encryption, or filtering.
- Brokers have no interceptors. Use policies, authorizers, or a protocol proxy for broker-side checks.
Practice
- (Beginner) Add
OriginHeaderInterceptorto a console producer with--producer-property interceptor.classes=...(the JAR on the classpath) and confirm the headers withkafka-console-consumer.sh --property print.headers=true.- (Intermediate) Turn
LatencyInterceptorinto one that registers a p50/p99 latency metric per topic in the consumer's own metrics registry, so it appears in JMX next to the client's built-in metrics.- Interview question: "A team adds a consumer interceptor that drops records with an invalid schema header. Months later, an audit finds records missing from the warehouse. What went wrong?" Hint: what happens to offsets of records removed in
onConsume, and where invalid records should go instead.
Pluggable Authorizers and Callbacks
Theory
Every organization has its own identity provider, policy rules, and quota model, and Kafka cannot ship all of them. Instead, the server defines Java interfaces at the points where decisions are made, and loads your implementation by class name from its classpath. Earlier chapters used several of these. This topic gives the full catalog and then looks at the most central one, the Authorizer contract, in depth.
Server-side extension points
| Extension point | Interface | Configured with | Runs on | See |
|---|---|---|---|---|
| Authorization | Authorizer | authorizer.class.name | Brokers and controllers | Section 11.3, below |
| Principal mapping | KafkaPrincipalBuilder (+ KafkaPrincipalSerde) | principal.builder.class | Brokers and controllers | Section 11.3 |
| SASL credential checks | AuthenticateCallbackHandler | listener.name.<listener>.<mechanism>.sasl.server.callback.handler.class | Brokers and controllers | Section 11.2 |
| Topic creation rules | CreateTopicPolicy | create.topic.policy.class.name | Controllers | Section 15.2 |
| Config change rules | AlterConfigPolicy | alter.config.policy.class.name | Controllers | Section 11.3 |
| Quotas | ClientQuotaCallback | client.quota.callback.class | Brokers | Custom quota grouping, for example per tenant instead of per user |
| Follower fetching | ReplicaSelector | replica.selector.class | Brokers | Section 2.2 |
| Tiered storage | RemoteStorageManager, RemoteLogMetadataManager | remote.log.storage.manager.class.name, ... | Brokers | Section 7.3 |
| Metrics export | MetricsReporter | metric.reporters | Every process | Section 12.2 |
| Secrets in configs | ConfigProvider | config.providers | Every process | Section 9.4 |
Rules that apply to all of them:
- Deploy the same JAR and class on every node that runs the hook. In KRaft that usually includes controllers, because brokers forward admin requests to them.
- Many hooks run on request threads and must be fast and thread-safe.
- A failure in
configure()stops the node from starting. That is usually what you want: better a node that does not start than one with no authorization.
The Authorizer contract
new MyAuthorizer() loaded by class name
-> configure(brokerConfigs) all broker configs, including custom keys
-> start(serverInfo) returns a CompletionStage per listener endpoint;
each listener opens only when its stage completes
-> authorize(context, actions) x N the hot path: called for nearly every request
authorizeByResourceType(...) "may this principal do X on ANY resource of type T?"
createAcls / deleteAcls / acls backs the ACL admin APIs and kafka-acls.sh
-> close()
| Method | What to know |
|---|---|
start | Lets the authorizer load its rules before traffic arrives. StandardAuthorizer completes it for most listeners only after it has loaded ACLs from the metadata log, but immediately for early.start.listeners (Section 11.3). |
authorize | Takes a list of actions and returns results in the same order. Runs on I/O threads, so no blocking I/O: precompute, cache, and refresh in the background. |
Action | Operation, a literal resource name, resourceReferenceCount, and logIfAllowed / logIfDenied flags. Respect the flags in audit logs: for example, a Metadata request for all topics sets logIfDenied=false because unauthorized topics are simply filtered out. |
AuthorizableRequestContext | Principal, listener, security protocol, client address, request type and version, client ID |
authorizeByResourceType | Used where Kafka checks permission on any resource of a type, for example WRITE on any topic for an idempotent producer's InitProducerId. The default implementation evaluates ACLs from acls(), so an authorizer that does not store ACLs must override it, or idempotent producers are denied. |
| ACL methods | An authorizer that does not store ACLs should reject createAcls and deleteAcls with a clear error. |
Kafka's ACL model also has implied operations: READ, WRITE, DELETE, and ALTER imply DESCRIBE, and ALTER_CONFIGS implies DESCRIBE_CONFIGS. A custom authorizer must reproduce these, or clients break in confusing ways (a producer with WRITE but no DESCRIBE cannot fetch metadata).
Example: a read-only, file-based authorizer (Java 21)
This authorizer reads rules from a version-controlled file instead of storing ACLs in the cluster, which some teams prefer because every permission change goes through code review.
# /etc/kafka/policy.txt principal | operation | resource type | pattern | name
User:orders-svc | WRITE | TOPIC | PREFIXED | sales.orders.
User:reporting | READ | TOPIC | PREFIXED | sales.
User:reporting | READ | GROUP | LITERAL | reporting
package com.acme.kafka;
import org.apache.kafka.common.Endpoint;
import org.apache.kafka.common.acl.AclBinding;
import org.apache.kafka.common.acl.AclBindingFilter;
import org.apache.kafka.common.acl.AclOperation;
import org.apache.kafka.common.errors.InvalidRequestException;
import org.apache.kafka.common.resource.PatternType;
import org.apache.kafka.common.resource.ResourceType;
import org.apache.kafka.server.authorizer.*;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
public class PolicyFileAuthorizer implements Authorizer {
private static final Set<AclOperation> IMPLIES_DESCRIBE =
EnumSet.of(AclOperation.READ, AclOperation.WRITE, AclOperation.DELETE, AclOperation.ALTER);
record Rule(String principal, AclOperation op, ResourceType type, PatternType pattern, String name) {
boolean grants(AclOperation requested) {
return op == AclOperation.ALL || op == requested
|| (requested == AclOperation.DESCRIBE && IMPLIES_DESCRIBE.contains(op))
|| (requested == AclOperation.DESCRIBE_CONFIGS && op == AclOperation.ALTER_CONFIGS);
}
boolean matches(String who, AclOperation requested, ResourceType t, String resource) {
if (!principal.equals(who) || type != t || !grants(requested)) return false;
return switch (pattern) {
case LITERAL -> name.equals("*") || name.equals(resource);
case PREFIXED -> resource.startsWith(name);
default -> false;
};
}
}
private volatile List<Rule> rules = List.of(); // swapped as a whole, read without locks
private Set<String> superUsers = Set.of();
@Override
public void configure(Map<String, ?> configs) {
Object su = configs.get("super.users"); // e.g. "User:broker;User:admin"
superUsers = su == null ? Set.of() : Set.of(su.toString().split(";"));
rules = load(Path.of(String.valueOf(configs.get("acme.authorizer.policy.file"))));
}
@Override
public Map<Endpoint, ? extends CompletionStage<Void>> start(AuthorizerServerInfo serverInfo) {
// Rules were loaded in configure(), so every listener may open right away
Map<Endpoint, CompletableFuture<Void>> ready = new HashMap<>();
for (Endpoint endpoint : serverInfo.endpoints()) {
ready.put(endpoint, CompletableFuture.completedFuture(null));
}
return ready;
}
@Override
public List<AuthorizationResult> authorize(AuthorizableRequestContext ctx, List<Action> actions) {
String who = ctx.principal().toString(); // "User:orders-svc"
boolean superUser = superUsers.contains(who);
List<AuthorizationResult> results = new ArrayList<>(actions.size());
for (Action action : actions) { // results must keep the order of actions
var resource = action.resourcePattern(); // always a LITERAL name here
boolean allowed = superUser || rules.stream().anyMatch(
r -> r.matches(who, action.operation(), resource.resourceType(), resource.name()));
results.add(allowed ? AuthorizationResult.ALLOWED : AuthorizationResult.DENIED);
}
return results;
}
@Override
public AuthorizationResult authorizeByResourceType(AuthorizableRequestContext ctx,
AclOperation op, ResourceType type) {
// The default implementation reads acls(), which is empty here, so override it
String who = ctx.principal().toString();
boolean allowed = superUsers.contains(who) || rules.stream().anyMatch(
r -> r.principal().equals(who) && r.type() == type && r.grants(op));
return allowed ? AuthorizationResult.ALLOWED : AuthorizationResult.DENIED;
}
@Override
public List<? extends CompletionStage<AclCreateResult>> createAcls(AuthorizableRequestContext ctx,
List<AclBinding> bindings) {
return bindings.stream()
.map(b -> CompletableFuture.completedFuture(new AclCreateResult(
new InvalidRequestException("ACLs are managed in the policy file"))))
.toList();
}
@Override
public List<? extends CompletionStage<AclDeleteResult>> deleteAcls(AuthorizableRequestContext ctx,
List<AclBindingFilter> filters) {
return filters.stream()
.map(f -> CompletableFuture.completedFuture(new AclDeleteResult(
new InvalidRequestException("ACLs are managed in the policy file"))))
.toList();
}
@Override
public Iterable<AclBinding> acls(AclBindingFilter filter) {
return List.of(); // could convert rules to AclBindings so kafka-acls.sh --list shows them
}
@Override
public void close() { }
private static List<Rule> load(Path file) {
try {
return Files.readAllLines(file).stream()
.map(String::strip)
.filter(line -> !line.isEmpty() && !line.startsWith("#"))
.map(line -> line.split("\\s*\\|\\s*"))
.map(f -> new Rule(f[0], AclOperation.fromString(f[1]), ResourceType.fromString(f[2]),
PatternType.fromString(f[3]), f[4]))
.toList();
} catch (IOException e) {
throw new UncheckedIOException("Cannot read policy file " + file, e); // node refuses to start
}
}
}
# On every broker AND controller (JAR in libs/)
authorizer.class.name=com.acme.kafka.PolicyFileAuthorizer
acme.authorizer.policy.file=/etc/kafka/policy.txt
super.users=User:broker;User:controller;User:ops-admin
The fromString methods return UNKNOWN for unrecognized values, so a typo silently creates a rule that never matches. A production version should validate every rule at load time, reload the file in the background (for example on a timer), and log decisions according to logIfAllowed and logIfDenied.
Key Takeaways
- Kafka servers expose interfaces for authorization, authentication callbacks, principals, policies, quotas, replica selection, tiered storage, metrics, and config secrets.
- Deploy plug-ins on every node that runs them, including controllers in KRaft. Keep hot-path hooks fast and thread-safe.
- The
Authorizerlifecycle is configure, start (per-listener readiness), authorize (batched, ordered, non-blocking), close. - Custom authorizers must handle implied operations and override
authorizeByResourceTypeif they do not store ACLs.
Practice
- (Beginner) Deploy
PolicyFileAuthorizeron a single-node cluster. Give a principal onlyWRITEon a topic prefix and confirm that the producer can fetch metadata (impliedDESCRIBE) and send.- (Intermediate) Remove the
authorizeByResourceTypeoverride and start an idempotent producer for the same principal. What error does it get, and from which API? Restore the override and add rule validation that rejectsUNKNOWNvalues at load time.- Interview question: "Your custom authorizer calls an external policy service over HTTP for each decision. Under load, produce latency rises sharply. Explain why and redesign it." Hint: which threads run
authorize, what blocking them does to the request pipeline, and how caching with background refresh changes the failure modes.
Kafka Improvement Proposals (KIPs)
Theory
Kafka runs in thousands of companies, and changes to anything users depend on can break them. The Kafka Improvement Proposal process makes such changes deliberate and public: before code is merged, a written proposal explains the motivation, the exact public changes, and the compatibility plan, and the community discusses and votes on it. For users, KIPs are the most reliable way to understand why a feature behaves as it does, and what is coming next.
Analogy: building permits. You can repaint your interior without one, but changing anything the neighbors rely on (the facade, the street access) needs a public plan, a review, and approval.
What needs a KIP
A KIP is required for any change to a public interface:
- The binary protocol and on-disk formats
- Public Java APIs of clients, Streams, Connect, and server plug-ins
- Configuration names, defaults, and meanings
- Command-line tools and their options
- Metrics and anything else operators rely on
Internal refactoring, bug fixes, and performance improvements that change none of these go directly through a JIRA ticket and a pull request.
Lifecycle
1. Claim the next KIP number on the wiki and write the page
2. [DISCUSS] thread on dev@kafka.apache.org weeks to months; the design changes here
3. [VOTE] thread open at least 72 hours; passes with at least
3 binding +1 votes (committers) and more
binding +1 than -1
4. Accepted -> JIRA tickets and pull requests
5. Released (often in parts across several versions, sometimes early access first)
Other states: Under Discussion, Discarded/Rejected, Dormant
Anatomy of a KIP page
| Section | Why to read it |
|---|---|
| Status | Current state, discussion thread, JIRA, target release |
| Motivation | The problem being solved: tells you whether the feature applies to you |
| Public Interfaces | The exact new configs, APIs, protocol fields, and metrics |
| Proposed Changes | How it works internally |
| Compatibility, Deprecation, and Migration Plan | What breaks, what is deprecated, and how to upgrade |
| Test Plan | How it was validated |
| Rejected Alternatives | Why the obvious alternative was not chosen. Often the most educational part. |
KIPs referenced in this guide
| KIP | Subject | Release |
|---|---|---|
| KIP-35 | ApiVersions and version negotiation | 0.10.0 |
| KIP-98 | Idempotence, transactions, exactly-once | 0.11 |
| KIP-101 | Leader epochs for replication | 0.11 |
| KIP-227 | Incremental fetch sessions | 1.1 |
| KIP-392 | Fetch from closest replica | 2.4 |
| KIP-482 | Tagged fields and flexible versions | 2.4 |
| KIP-405 | Tiered storage | Early access 3.6, production-ready 3.9 |
| KIP-500 | Replace ZooKeeper with KRaft | Early access 2.8, only mode in 4.0 |
| KIP-714 | Client metrics and observability | 3.7 |
| KIP-848 | Next-generation consumer rebalance protocol | GA in 4.0 |
| KIP-853 | Dynamic KRaft controller quorum | 3.9 |
| KIP-896 | Removal of old protocol API versions | 4.0 |
| KIP-932 | Queues for Kafka (share groups) | Early access 4.0 |
| KIP-966 | Eligible Leader Replicas | 4.x |
| KIP-1150 | Diskless topics | Proposed 2025 (next topic) |
Example: tracing a KIP into the code
The wiki tells you what was planned. The source history tells you what shipped and when:
git clone https://github.com/apache/kafka.git && cd kafka
# How much work referenced the KIP, and where it started
git log --oneline --grep="KIP-848" | wc -l
git log --oneline --grep="KIP-848" --reverse | head -5
# First release containing a given commit
git tag --contains <commit-sha> | sort -V | head -1
# Where the KIP's public configuration lives
git grep -n "group.protocol" -- clients/src/main/java | head
Then check the release's upgrade notes (docs/upgrade.html in the source, "Upgrading" in the documentation) for the user-visible summary.
Example: a one-page KIP summary card
When evaluating a KIP for your team, summarize it in a fixed format:
KIP: KIP-848 Next-generation consumer rebalance protocol
Problem: Client-driven rebalances stop the whole group; slow, fragile with many members
Change: Broker-side assignment; incremental, per-member reconciliation via
ConsumerGroupHeartbeat (new API, key 68)
Public surface: group.protocol=consumer, group.remote.assignor, new group configs, new metrics
Requires: Brokers 4.0+ (GA); Java client 4.0+
Compatibility: Classic groups keep working; online migration of a group from classic to consumer
Risks for us: Custom client-side assignors must be replaced; check non-Java client support
Decision: Pilot on one non-critical consumer group after the broker upgrade
Key Takeaways
- A KIP is required for any change to public interfaces: protocol, formats, APIs, configs, tools, metrics.
- KIPs are discussed and voted on in public on
dev@kafka.apache.organd often ship in parts across several releases. - The Motivation, Public Interfaces, Compatibility, and Rejected Alternatives sections are the most useful for users.
- Confirm what actually shipped through release upgrade notes and the source history, not only the KIP page.
Practice
- (Beginner) Open the KIP index page on the Apache Kafka wiki. Find three KIPs adopted in the most recent release and write one sentence on each.
- (Intermediate) Write a summary card for KIP-932 (share groups) using the format above. Include which parts are early access in your release.
- Interview question: "Why does Kafka require a KIP to rename a command-line option, but not to rewrite the log cleaner's internals?" Hint: public interfaces versus implementation, and who could break in each case.
Diskless and Object-Storage-Native Kafka
Theory
Kafka's original design is shared-nothing: each broker stores its partitions on local disks, and durability comes from replicating each partition to other brokers. That design is excellent in a data center. In the cloud, it creates three costs:
- Cross-zone network fees. Replicas sit in different availability zones (AZs) for safety. Every byte produced is copied across zones by replication, and producers and consumers often talk to leaders in other zones. Cloud providers charge for that traffic.
- Provisioned disks. Each byte is stored three times on block storage, plus headroom.
- Stateful brokers. Adding or removing a broker means moving partitions' data, which is slow.
Object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage) is already replicated across zones, cheap per gigabyte, and in the same region usually free of transfer fees. Tiered storage (KIP-405, Section 7.3) already moves old segments there, but the hot write path is unchanged. Diskless designs go further: brokers write new data directly to object storage, and object storage provides durability instead of broker replication.
Analogy: classic Kafka is a set of shops, each keeping its own stock and copying every delivery to two other shops across town. Diskless Kafka is a set of counters in front of one shared central warehouse: any counter can serve any customer, nothing is copied between counters, but each order takes longer because the warehouse is further away.
Three designs compared
| Classic | Tiered storage (KIP-405) | Diskless | |
|---|---|---|---|
| Where new, acknowledged data lives | Local disks of RF brokers | Local disks of RF brokers | Object storage (some designs add a small write-ahead log) |
| Old data | Local disks | Object storage | Object storage |
| Durability from | Kafka replication (ISR) | Kafka replication, then the object store | The object store |
| Cross-zone traffic for writes | Producer to leader, plus RF - 1 replica copies | Same as classic | None or minimal with zone-aware routing |
| Broker state | All data | Recent data | Mostly cache: brokers are nearly stateless |
| Typical produce latency (p99) | Single-digit to tens of ms | Same as classic | Hundreds of ms |
| Scaling | Move partition data | Move only the local part | Add or remove brokers without moving data |
| Best for | Low-latency pipelines, transactions | Long retention at lower cost | High-volume, latency-tolerant data: logs, telemetry, CDC into data lakes |
How a diskless write and read work
The details differ between implementations, but most follow this pattern:
AZ-a AZ-b object storage
producer --Produce--> broker A producer --Produce--> broker B (regional, multi-AZ)
| buffers batches for |
| many partitions for |
| ~100-250 ms or a few MiB |
v v
PUT one object ------------------------------------------> [obj-17: orders-3,
| payments-0, clicks-9 ...]
v
batch coordinator "orders-3: batch in obj-17 bytes 0..48K gets offsets 9100-9163"
| (assigns offsets, keeps per-partition order, tracks producer IDs)
v
ProduceResponse(base_offset=9100) to the producer
consumer (AZ-b) --Fetch(orders-3, 9100)--> broker B --asks coordinator--> obj-17, range 0..48K
broker B --ranged GET (or cache hit)--> records
background: merge many small mixed objects into larger per-partition segments
Key consequences:
- No partition leaders for data. Any broker can accept writes for any partition, so producers and consumers can always use a broker in their own zone. The metadata response can steer each client to a local broker.
- Latency comes from the object store. An object
PUTtakes tens to hundreds of milliseconds, and brokers deliberately wait to fill objects, because each request costs money. - The batch coordinator is the new critical component. It provides ordering, offsets, idempotence, and transactions, which classic Kafka gets from the partition leader.
- The protocol does not change. Clients still speak the Kafka protocol from Section 16.1, which is why this can be adopted without changing applications.
Implementations and proposals
| Name | Approach |
|---|---|
| WarpStream (acquired by Confluent in 2024) | Stateless agents in your own cloud account write to your bucket. Metadata and coordination run in a vendor-hosted control plane. |
| Confluent Freight clusters | A Confluent Cloud cluster type that writes directly to object storage |
| AutoMQ | A fork of Apache Kafka that replaces local log storage with a write-ahead log plus shared object storage |
| Bufstream (Buf) | Kafka-compatible broker writing to object storage, with metadata in a database and Apache Iceberg integration |
| Aiven Inkless | A fork of Apache Kafka implementing the KIP-1150 design |
| Redpanda Cloud Topics | Object-storage-backed topics in Redpanda |
| KIP-1150 "Diskless Topics" (with follow-up KIPs for the core and the batch coordinator) | Proposal to add diskless topics to Apache Kafka as a per-topic choice, so classic and diskless topics share one cluster |
| Related: KIP-1176, KIP-1183 | Alternative proposals: tiering the active segment quickly, and a pluggable shared-storage layer |
These proposals were under active discussion in 2025, with competing designs. Check each KIP page for its current status and target release before planning around one.
Example: a rough cost model (Python)
The main argument for diskless is cross-zone traffic. This model compares monthly network and request costs at AWS list prices for one region with three AZs. It is an order-of-magnitude estimate, not a quote: prices change, and storage and compute are not included.
SECONDS_PER_MONTH = 30 * 24 * 3600
CROSS_AZ_PER_GB = 0.02 # $0.01 out of one AZ + $0.01 into the other
S3_PUT_PER_1000 = 0.005
S3_GET_PER_1000 = 0.0004
def classic(ingress_mb_s, rf=3, azs=3, consumer_groups=1, follower_fetching=False):
gb = ingress_mb_s * SECONDS_PER_MONTH / 1000 # data produced per month (decimal GB)
remote = (azs - 1) / azs # chance the leader is in another AZ
produce = gb * remote # producer -> leader
replication = gb * (rf - 1) # leader -> followers in the other AZs
consume = 0 if follower_fetching else gb * remote * consumer_groups
return (produce + replication + consume) * CROSS_AZ_PER_GB
def diskless(ingress_mb_s, brokers=12, flush_ms=250, consumer_groups=1, get_size_mb=4):
puts = brokers * (1000 / flush_ms) * SECONDS_PER_MONTH # each broker writes one object per flush
mb = ingress_mb_s * SECONDS_PER_MONTH
gets = consumer_groups * mb / get_size_mb # ranged GETs; caching lowers this
return puts / 1000 * S3_PUT_PER_1000 + gets / 1000 * S3_GET_PER_1000
rate = 100 # MB/s produced, read once by one consumer group
print(f"classic, leader fetching : ${classic(rate):>9,.0f} / month")
print(f"classic, follower fetching : ${classic(rate, follower_fetching=True):>9,.0f} / month")
print(f"diskless (S3 requests only): ${diskless(rate):>9,.0f} / month")
classic, leader fetching : $ 17,280 / month
classic, follower fetching : $ 13,824 / month
diskless (S3 requests only): $ 648 / month
Replication alone is 60% of the classic cost here and cannot be removed in a classic design, which is why object-storage designs are attractive at high throughput. The picture reverses at low volume: at 1 MB/s the classic design costs about $170 a month in cross-zone traffic, while the diskless design still pays about $620 for its object PUTs, because each broker writes an object every flush interval whether or not it is full. Request costs are fixed, and traffic costs scale with volume.
Example: client settings for a high-latency backend
When acknowledgments take 500 ms instead of 5 ms, producers need more data in flight to keep the same throughput (throughput x latency = bytes in flight):
# Producer against an object-storage-backed cluster, ~50 MB/s per producer, ~0.5-1 s acks
linger.ms=100 # build larger batches; the backend batches anyway
batch.size=1048576 # 1 MiB batches
compression.type=zstd # fewer bytes stored and transferred
buffer.memory=134217728 # 128 MiB: 50 MB/s x 1 s in flight, plus headroom (default 32 MiB)
max.in.flight.requests.per.connection=5 # the maximum that keeps idempotent ordering
request.timeout.ms=60000 # tolerate slow object-store PUTs
delivery.timeout.ms=300000
enable.idempotence=true
Consumers should also allow larger fetches (fetch.max.bytes, max.partition.fetch.bytes), and zone-aware routing needs each client to tell the cluster its zone. How that is done is product-specific (for example a rack setting or a zone hint in client.id), so check your vendor's documentation.
Key Takeaways
- In the cloud, classic replication creates cross-zone traffic and disk costs. Diskless designs make object storage the durable store and remove broker-to-broker replication.
- Diskless trades latency (hundreds of milliseconds) for much lower network cost and nearly stateless, fast-scaling brokers.
- A batch coordinator takes over ordering, offsets, and idempotence from partition leaders. The Kafka protocol stays the same, so clients do not change.
- Several vendors ship diskless Kafka-compatible systems. KIP-1150 and related KIPs propose it for Apache Kafka; check their current status.
Practice
- (Beginner) Use the cost model to find the ingress rate at which the classic design's cross-zone cost (with follower fetching) exceeds $1,000 per month.
- (Intermediate) Extend the model with storage: 7 days of retention, RF 3 on block storage at $0.08 per GB-month (with 30% headroom) versus object storage at $0.023 per GB-month. How does the comparison change at 10 MB/s and at 200 MB/s?
- Interview question: "Your company wants to move all Kafka topics to a diskless engine to cut costs. Which topics would you keep on classic storage, and why?" Hint: latency-sensitive request/response flows, transactions and exactly-once pipelines, compacted topics, low-volume topics where savings are small, and how mature each feature is in the chosen engine.