13. Performance and Capacity Planning
A Kafka cluster that works in testing can still fall over in production because it is too slow at the 99th percentile, saturated at peak, or out of disk three months after launch. This chapter covers two linked skills: performance tuning (measuring a cluster honestly, finding the bottleneck in the broker's request path, and tuning threads, queues, and fetches to meet a latency or throughput target) and capacity planning (turning business requirements into partition counts, disk sizes, and broker counts with enough headroom to survive failures and growth). Both rely on the same habit: estimate with arithmetic, then confirm with measurement.
<a id="131-performance-tuning"></a>
13.1 Performance Tuning
This section covers how to benchmark a cluster, then how to tune the broker-side parts of the data path (threads, request queues, fetches) and the full producer-to-consumer path for latency.
Benchmarking with Perf Tools
Theory
Published Kafka throughput figures ("2 million writes per second on three machines") are real, but they were measured with someone else's record sizes, compression ratio, replication settings, hardware, and network. Change any of those and the numbers change, sometimes by an order of magnitude. Before you tune anything or buy any hardware, you need numbers for your workload on your infrastructure. A benchmark answers two questions:
- Capacity: how much traffic can this cluster sustain before latency becomes unacceptable?
- Effect of a change: did this setting, instance type, or Kafka version make things better or worse?
Analogy: a truck's brochure lists its top speed with an empty trailer on a flat road. If you haul gravel through the mountains, you test-drive it loaded with gravel on a mountain road. A benchmark is that test drive: same cargo, same route, same driver.
The tools
| Tool | Ships with Kafka | Measures | Good for |
|---|---|---|---|
kafka-producer-perf-test.sh | Yes | Produce throughput, produce latency percentiles | Producer settings, broker write capacity |
kafka-consumer-perf-test.sh | Yes | Consume throughput (MB/s, records/s), fetch time | Consumer fetch settings, broker read capacity |
org.apache.kafka.tools.EndToEndLatency | Yes (via kafka-run-class.sh) | Time from send to consume, one record at a time | Latency floor of the full path |
| Trogdor | Yes (trogdor.sh) | Coordinated workloads and fault injection across many agents | Running repeatable tests across many machines, testing behavior under faults |
| OpenMessaging Benchmark (OMB) | No (separate project) | Many producers and consumers, fixed-rate load, latency histograms | Realistic multi-client capacity tests, comparing platforms |
The two perf-test scripts are enough for most tuning work. Move to OMB or Trogdor when one client machine cannot generate enough load or you need many coordinated clients.
What the built-in perf tools do and do not model
| Aspect | kafka-producer-perf-test.sh behavior | Consequence |
|---|---|---|
| Payload | Random uppercase letters unless you pass --payload-file | Compression ratios differ from real data. Use a file of real sample records |
| Keys | Records are sent without keys | Partition distribution is that of the sticky partitioner, not your key distribution |
| Rate | --throughput N caps records/s. -1 means unlimited | Unlimited runs measure saturation, and their latency is mostly queueing |
| Threads | One producer instance in one process | A single client may become the bottleneck before the cluster does |
A sound methodology
- Write down the question and the target in numbers ("sustain 200 MB/s with produce p99 below 30 ms").
- Mirror production: the same instance types, disks, replication factor,
min.insync.replicas, TLS, andacks. A benchmark without TLS says little about a TLS cluster. - Use realistic data: real record sizes and a
--payload-filewith sample records, so compression behaves as it will in production. - Warm up, then measure: the first seconds include JIT compilation, connection setup, and metadata fetches. Run long enough (10+ minutes) to see segment rolls, page cache pressure, and GC.
- Generate enough load: run several perf-test processes on several machines, and confirm the clients are not CPU- or network-bound.
- Step the load: increase the rate in steps and record latency at each step. The interesting result is where latency starts to climb.
- Watch the brokers during every run: CPU, disk, network,
RequestHandlerAvgIdlePercent,NetworkProcessorAvgIdlePercent, and request time phases (Section 12.3). The result is only useful if you know which resource ran out. - Change one variable per run and keep a log of every run's settings and results.
The shape you are looking for
p99 latency
^
| *
| *
| * <- saturation: queues grow without bound
| *
| * <- the "knee": plan to run well below this
| * * * * *
+--------------------------------------------------> offered load (MB/s)
50 100 150 200 250 300 350 400
Usable capacity = the load at which latency still meets the target, minus headroom
Example: a step-load producer test
#!/usr/bin/env bash
# Step-load test: run at increasing fixed rates and keep only the summary line of each run
BS=broker1:9092,broker2:9092,broker3:9092
TOPIC=perf-orders
bin/kafka-topics.sh --bootstrap-server $BS --create --topic $TOPIC \
--partitions 24 --replication-factor 3 --config min.insync.replicas=2
# samples.txt: one real (anonymized) order event per line, so compression is realistic
for RATE in 50000 100000 150000 200000 250000 300000; do
echo "=== target ${RATE} records/s ==="
bin/kafka-producer-perf-test.sh --topic $TOPIC \
--num-records $((RATE * 120)) \
--throughput $RATE \
--payload-file samples.txt --payload-delimiter $'\n' \
--producer-props bootstrap.servers=$BS acks=all linger.ms=5 batch.size=65536 \
compression.type=lz4 \
| tail -n 1 # final summary line only
done
# === target 50000 records/s ===
# 6000000 records sent, 49998.1 records/sec (23.84 MB/sec), 3.10 ms avg latency, ... 7 ms 99th ...
# ...
# === target 250000 records/s ===
# 30000000 records sent, 231877.4 records/sec (110.57 MB/sec), 412.66 ms avg latency, ... 1650 ms 99th ...
# -> achieved rate fell below the target and p99 exploded: the cluster is saturated
(The numbers are illustrative. Each run lasts about two minutes, so the first seconds of warm-up have little effect on the summary.)
Example: producing and consuming at the same time
Production clusters read while they write, and reads of old data compete with writes for disk. Test the mix:
# Terminal 1-3 (three client machines): each produces at a fixed share of the target load
bin/kafka-producer-perf-test.sh --topic perf-orders --num-records 36000000 --throughput 100000 \
--record-size 500 --producer-props bootstrap.servers=$BS acks=all linger.ms=5 compression.type=lz4
# Terminal 4: a consumer group reading the same topic in real time
bin/kafka-consumer-perf-test.sh --bootstrap-server $BS --topic perf-orders \
--group perf-live --from-latest --messages 100000000 --timeout 600000 \
--show-detailed-stats --reporting-interval 10000
# Terminal 5: a "catch-up" consumer in a new group, which starts from the earliest offset
# (the tool's default) and forces disk reads once the data no longer fits in the page cache
bin/kafka-consumer-perf-test.sh --bootstrap-server $BS --topic perf-orders \
--group perf-replay-$(date +%s) --messages 100000000
Compare the live consumer's throughput and the producers' p99 before and after the catch-up consumer starts. If producer latency jumps, cold reads are evicting hot data from the page cache or saturating the disk.
Note: the
kafka-consumer-perf-test.shoptions vary slightly between releases (for example,--from-latestand the timeout options). Run it with--helpon your version before scripting it.
Example: end-to-end latency floor
# Arguments: bootstrap servers, topic, number of messages, producer acks, message size in bytes
bin/kafka-run-class.sh org.apache.kafka.tools.EndToEndLatency \
broker1:9092 e2e-probe 10000 all 512
# Avg latency: 3.1203 ms
# Percentiles: 50th = 3, 99th = 7, 99.9th = 15
This tool sends one record, waits until it is consumed, then sends the next. It measures the best case for the full path, not behavior under load. Use it to compare configurations or detect regressions, not to size a cluster.
Key Takeaways
- Never size or tune from published benchmarks. Measure with your record sizes, compression, replication, TLS, and hardware.
- Use real sample data (
--payload-file) so compression ratios are realistic, and remember the producer perf tool sends unkeyed records. - Step the load and find the knee of the latency curve. Plan to operate well below it.
- Unlimited-rate runs measure saturation. Latency should be measured at the expected production rate.
- A benchmark result is only useful together with the broker metrics that show which resource was exhausted.
Practice
- (Beginner) Run
kafka-producer-perf-test.shonce with--record-size 1000and once with a--payload-fileof 1 KB JSON documents, both withcompression.type=zstd. Compare MB/s and the topic's size on disk, and explain the difference.- (Intermediate) Write a step-load script for a 3-broker cluster that runs at 25%, 50%, 75%, 100%, and 125% of an expected peak rate. For each step, capture produce p99 and
RequestHandlerAvgIdlePercentfrom one broker. Identify the step at which the cluster stops meeting a 50 ms p99 target and name the bottleneck.- Interview question: "A vendor benchmark shows 1 GB/s on three brokers, but your identical-looking cluster tops out at 150 MB/s. How do you explain and investigate the gap?" Hint: list every difference between the two setups (replication,
acks, TLS, record size, compression, number of clients, disks, network), and decide how to measure which one matters.
Network and I/O Threads
Theory
Section 2.2 showed the path a request takes through a broker: an acceptor accepts the connection, a network thread (also called a processor) reads the request bytes from the socket, an I/O thread (also called a request handler) does the actual work, and a network thread writes the response back. The two thread pools are sized independently, and either can become the bottleneck. When that happens, requests wait in queues even though the disks and the rest of the machine may be idle.
Analogy: a restaurant has waiters (network threads) and cooks (I/O threads). Waiters take orders from tables and carry plates back. Cooks prepare the food. If there are too few waiters, customers wait to order and finished plates go cold on the counter. If there are too few cooks, order tickets pile up on the rail. Hiring more cooks does nothing if the waiters are the problem, and neither helps if the stove (the disk) is too small.
The broker's thread pools
| Pool | Setting (default) | Work done | Scales with |
|---|---|---|---|
| Acceptor | One per listener (fixed) | Accepts new TCP connections | Connection churn |
| Network threads | num.network.threads (3) per listener | Read requests from sockets, write responses, TLS encryption and decryption | Connections, request rate, bytes on the wire, TLS |
| I/O threads | num.io.threads (8), shared by all listeners | Validate and append produce batches, read for fetches, handle metadata, offsets, group, and transaction requests | Request rate, batch validation (decompression), disk reads that miss the page cache |
| Replica fetchers | num.replica.fetchers (1) per source broker | Copy data from leaders to this broker's followers | Partitions and bytes replicated |
| Log cleaner | log.cleaner.threads (1) | Compact topics with cleanup.policy=compact | Compacted data volume |
| Background | background.threads (10) | Retention deletion, segment file deletion, other scheduled tasks | Partition count |
Where CPU goes matters for choosing which pool to grow:
- TLS runs on network threads. Enabling TLS on a busy listener often turns network threads into the bottleneck.
- Zero-copy (
sendfile) only works without TLS. With plaintext, network threads mostly shuffle file pointers, and they are rarely saturated. - Batch validation runs on I/O threads. The broker decompresses compressed batches to validate them, and it recompresses only if the topic's
compression.typediffers from the producer's codec, which is expensive and best avoided. - Disk reads that miss the page cache block an I/O thread until the disk returns data. Lagging consumers reading old data can tie up I/O threads that produce requests also need.
Diagnosing which pool is saturated
| Signal | Interpretation | Action |
|---|---|---|
NetworkProcessorAvgIdlePercent below 0.3, ResponseQueueTimeMs rising | Network threads saturated | Raise num.network.threads; look at TLS and connection counts |
RequestHandlerAvgIdlePercent below 0.3, RequestQueueTimeMs rising | I/O threads saturated | Raise num.io.threads if CPU and disk have headroom |
| I/O threads busy, disk utilization near 100% | The disk is the real bottleneck | More threads will not help; reduce cold reads, add disks or brokers |
| Both pools mostly idle, but latency is high | Not a thread problem | Look at RemoteTimeMs (followers) and LocalTimeMs (disk), Section 12.3 |
| One network processor much busier than others | Uneven connection distribution (a few heavy clients) | Usually acceptable; check kafka.network:type=Processor,name=IdlePercent,networkProcessor=* |
The idle-percent metrics are ratios between 0 and 1: 1 means always idle, 0 means always busy. Values below 0.3 for sustained periods are the usual alert threshold.
Starting points
| Machine / workload | num.network.threads | num.io.threads |
|---|---|---|
| Small (4 to 8 cores), plaintext | 3 (default) | 8 (default) |
| Medium (16 cores), TLS | 6 to 8 | 16 |
| Large (32+ cores), TLS, many connections | 8 to 12 | 16 to 32 |
These are starting points, not answers. A common rule is to keep num.io.threads at least equal to the number of data disks so each disk can have a request in flight, and to raise it further only while CPU has headroom. Too many threads add context switching and lock contention without adding throughput.
Example: changing thread counts online, one broker first
Both settings are dynamic and cluster-wide (Section 12.1). A safe pattern is to try a new value on a single broker, compare it with its peers, then roll it out:
BS=broker1:9092
# 1. Canary: raise I/O threads on broker 3 only
bin/kafka-configs.sh --bootstrap-server $BS --alter \
--entity-type brokers --entity-name 3 \
--add-config num.io.threads=16
# 2. Compare broker 3's request handler idle ratio with broker 1's for 15-30 minutes
for B in broker1 broker3; do
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
--jmx-url service:jmx:rmi:///jndi/rmi://$B:9999/jmxrmi \
--object-name kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent \
--attributes OneMinuteRate --reporting-interval 10000 --one-time true
done
# 3. Roll out as a cluster-wide default, then remove the per-broker override
bin/kafka-configs.sh --bootstrap-server $BS --alter \
--entity-type brokers --entity-default --add-config num.io.threads=16
bin/kafka-configs.sh --bootstrap-server $BS --alter \
--entity-type brokers --entity-name 3 --delete-config num.io.threads
# Network threads: a single dynamic change must stay within half to double the current value,
# so going from 3 to 12 takes two steps (3 -> 6 -> 12)
bin/kafka-configs.sh --bootstrap-server $BS --alter \
--entity-type brokers --entity-default --add-config num.network.threads=6
Example: a quick capacity check of the thread pools
# Rough check: how busy will the I/O pool be at a new request rate?
# Measured today on one broker (from JMX): request rate and average local processing time
requests_per_s = 12_000 # Produce + Fetch requests handled per second
avg_local_ms = 0.45 # LocalTimeMs averaged across request types
io_threads = 8
busy_fraction = requests_per_s * (avg_local_ms / 1000) / io_threads
print(f"I/O threads busy: {busy_fraction:.0%}") # -> 68%: idle ratio around 0.32
# Traffic is expected to double: 135% busy is impossible, so the request queue would grow without bound
print(f"After doubling: {busy_fraction * 2:.0%}")
# Option: 16 threads (if CPU and disk allow) -> about 68% busy again
Key Takeaways
- Network threads move bytes between sockets and the broker (and do TLS). I/O threads do the work of each request. Either pool can saturate on its own.
num.network.threadsapplies per listener.num.io.threadsis shared across listeners.- Use
NetworkProcessorAvgIdlePercentandRequestHandlerAvgIdlePercent(alert below 0.3) together with queue time metrics to decide which pool to grow. - Adding threads only helps when CPU and disk have headroom. A saturated disk needs fewer cold reads or more hardware, not more threads.
- Change thread counts dynamically on one broker first, compare, then apply cluster-wide.
Practice
- (Beginner) On a test broker, run
kafka-producer-perf-test.shwithout a rate limit and recordRequestHandlerAvgIdlePercentandNetworkProcessorAvgIdlePercent. Reducenum.io.threadsto 2 dynamically and run again. Which metric moved, and how did produce latency change?- (Intermediate) Enable TLS on one listener of a test cluster, repeat the same benchmark through the TLS listener and the plaintext listener, and compare broker CPU and the network idle ratio. Decide how many network threads the TLS listener needs to match the plaintext throughput.
- Interview question: "
RequestHandlerAvgIdlePercentis at 0.1 on one broker and 0.6 on the others. Would you raisenum.io.threadscluster-wide?" Hint: first find out why one broker does more work: leadership imbalance, a hot partition, or a lagging consumer reading old data from that broker's disks.
Request Queue Tuning
Theory
Between the network threads and the I/O threads sits the request queue, and between the I/O threads and the network threads sits a response queue per network thread. Queues absorb short bursts: if 2,000 requests arrive in the same millisecond, they wait their turn instead of being rejected. But a queue is only a buffer, not extra capacity. If requests arrive faster than I/O threads can process them for long enough, the queue fills, latency grows, and eventually the broker must push back on clients.
Kafka's push-back mechanism is simple and effective: when the request queue is full, network threads stop reading new requests from sockets. Unread bytes accumulate in the kernel's TCP buffers, TCP flow control slows the clients down, and their own buffers (for example, the producer's buffer.memory) start to fill. Nothing is dropped. The pressure moves back to the clients.
Analogy: a highway on-ramp with a traffic light. The light lets cars in at the rate the highway can absorb. When the highway is jammed, the queue on the ramp grows, and eventually cars back up onto the city streets. The ramp does not make the highway faster. It only decides where the waiting happens.
One request at a time per connection
The broker processes at most one request per connection at a time: after reading a request, the network thread stops reading that socket until the response has been sent. This guarantees that requests from one connection are answered in order. A producer with max.in.flight.requests.per.connection=5 still pipelines (requests sit in socket buffers while earlier ones are processed), but the broker never works on two requests from the same connection in parallel. Throughput across many partitions therefore comes from batching many partitions into each request, and from many connections.
The settings
| Setting | Default | Meaning |
|---|---|---|
queued.max.requests | 500 | Maximum requests in the request queue before network threads stop reading |
queued.max.request.bytes | -1 (no limit) | Maximum total bytes of queued requests. Bounds broker memory used by large queued produce requests |
socket.request.max.bytes | 104857600 (100 MB) | Largest single request the broker accepts |
socket.receive.buffer.bytes / socket.send.buffer.bytes | 102400 | Socket buffer sizes. -1 uses the OS default (which can auto-tune). Raise for high-latency links |
socket.listen.backlog.size | 50 | Pending TCP connections waiting for the acceptor |
max.connections, max.connections.per.ip | Unlimited | Caps on connection count (Section 11.3) |
max.connection.creation.rate | Unlimited | Caps new connections per second, which protects against reconnect storms |
Reading the queue metrics
| Metric | Meaning |
|---|---|
kafka.network:type=RequestChannel,name=RequestQueueSize | Requests waiting for an I/O thread |
kafka.network:type=RequestChannel,name=ResponseQueueSize | Responses waiting for a network thread |
kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=* | Time a request spent in the request queue |
kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=* | Time a response spent waiting for a network thread |
kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Produce | acks=all produce requests waiting for followers |
kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Fetch | Fetch requests waiting for fetch.min.bytes or new data |
Purgatory is a different kind of waiting: those requests already passed through an I/O thread and are parked until a condition is met (Section 2.2). A large fetch purgatory is normal for idle topics with long fetch.max.wait.ms. A growing produce purgatory means followers are slow.
Little's Law: how queues, rate, and latency relate
For any stable queueing system, items in the system = arrival rate x time in the system. It turns queue metrics into something you can reason about:
RequestQueueSize (avg) = request rate x RequestQueueTimeMs (avg)
Example: 20,000 requests/s and an average queue time of 5 ms
-> about 20,000 x 0.005 = 100 requests waiting on average
Raising queued.max.requests from 500 to 2000 does not change this number while the
broker keeps up. It only matters during bursts, or when the broker cannot keep up at all.
When to change the queue settings
| Situation | Change | Reason |
|---|---|---|
| Short bursts (for example, all producers flushing at the same second) hit the limit | Raise queued.max.requests moderately (for example, 1000) | Absorb the burst without stalling sockets |
| Broker memory spikes from queued large produce requests | Set queued.max.request.bytes (for example, 10% to 20% of the heap) | Bound memory instead of only request count |
| Clients reconnect in a storm after a broker restart | Set max.connection.creation.rate and max.connections.per.ip | Protect the acceptor and network threads |
| One client application floods the broker with requests | Request-rate quota (request_percentage, Section 11.3) | Isolate a noisy tenant without hurting others |
| Queue is persistently non-empty and I/O threads are busy | Do not grow the queue. Add I/O threads or capacity | A larger queue only adds latency when the broker is saturated |
Example: watching queues during a load test
BS=broker1:9092
JMX=service:jmx:rmi:///jndi/rmi://broker1:9999/jmxrmi
# Request queue depth every 5 seconds while a perf test runs elsewhere
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool --jmx-url $JMX \
--object-name kafka.network:type=RequestChannel,name=RequestQueueSize \
--reporting-interval 5000
# p99 queue time for produce requests
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool --jmx-url $JMX \
--object-name 'kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=Produce' \
--attributes 99thPercentile --reporting-interval 5000
# server.properties: queue and connection protection for a busy broker
queued.max.requests=1000 # absorb bursts (default 500)
queued.max.request.bytes=1073741824 # at most 1 GiB of queued request data (heap is 6 GiB)
socket.request.max.bytes=104857600 # keep the default unless you raise message.max.bytes a lot
socket.receive.buffer.bytes=-1 # let the OS auto-tune (pair with net.ipv4.tcp_rmem, Section 12.1)
socket.send.buffer.bytes=-1
max.connections.per.ip=500
max.connection.creation.rate=200 # new connections per second, per broker
Key Takeaways
- The request queue absorbs bursts. It does not add capacity. A persistently non-empty queue means the I/O threads (or something behind them) are saturated.
- When the queue is full, network threads stop reading sockets and TCP flow control pushes back on clients. Nothing is dropped.
- The broker handles one request per connection at a time, so ordering per connection is guaranteed and parallelism comes from batching and multiple connections.
- Use Little's Law (
queue size = rate x wait time) to reason about queue metrics. - Bound memory with
queued.max.request.bytes, protect against connection storms with connection rate limits, and isolate noisy clients with quotas.
Practice
- (Beginner) A broker handles 8,000 requests/s with an average
RequestQueueTimeMsof 2 ms. Use Little's Law to estimate the average request queue size. What does it mean if the measuredRequestQueueSizeis close toqueued.max.requests?- (Intermediate) Set
queued.max.requests=10andnum.io.threads=1on a test broker, then run an unlimited producer perf test. Observe producer latency,bufferpool-wait-ratio, and broker queue metrics. Explain the chain of back-pressure from the broker's I/O thread to the application callingsend().- Interview question: "A colleague proposes raising
queued.max.requeststo 50,000 because they saw the request queue full during peak. What do you say?" Hint: consider what is happening when the queue is full for long periods, what a bigger queue does to latency and memory, and what the real fix is.
Fetch Size Optimization
Theory
Most bytes that leave a broker are fetch responses: consumers reading data and followers replicating it. Section 5.1 explained the consumer settings (fetch.min.bytes, fetch.max.wait.ms, fetch.max.bytes, max.partition.fetch.bytes). This topic looks at fetching from the cluster's point of view: how fetch sizes drive request rates and broker CPU, how followers fetch, and how to keep fetch limits, batch sizes, and memory consistent across producers, brokers, and consumers.
The core trade-off is the same for consumers and followers. Every fetch request has a fixed cost (a network round trip, a pass through a network thread and an I/O thread, a metadata lookup for every partition in the request). Bigger, fuller fetches spread that cost over more bytes. Smaller, faster fetches reduce latency but multiply the fixed cost.
Analogy: a delivery van that leaves the warehouse with a single parcel every time one is ready gives the fastest delivery for that parcel, but the warehouse loading dock is jammed with vans. A van that waits a moment for a full load delivers more parcels per trip, and the dock stays clear.
Fetch request rate grows quickly
Each consumer instance sends one fetch to each broker that leads some of its partitions.
When there is no new data, the broker holds each fetch for fetch.max.wait.ms.
idle fetch requests/s ~ consumer instances x brokers contacted / fetch.max.wait.ms
2,000 consumer instances x 6 brokers / 0.5 s = 24,000 requests/s (defaults)
2,000 consumer instances x 6 brokers / 0.05 s = 240,000 requests/s (fetch.max.wait.ms=50)
Lowering fetch.max.wait.ms across a large fleet can multiply broker request load tenfold even when there is no data at all. Check kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer before and after such a change.
Replication fetch settings
Followers are consumers too, with their own settings on the broker:
| Setting | Default | Effect |
|---|---|---|
num.replica.fetchers | 1 | Fetcher threads per source broker. More threads replicate more partitions in parallel |
replica.fetch.max.bytes | 1048576 (1 MiB) | Bytes per partition per fetch (soft limit, like max.partition.fetch.bytes) |
replica.fetch.response.max.bytes | 10485760 (10 MiB) | Bytes per fetch response across partitions |
replica.fetch.min.bytes | 1 | Minimum bytes before the leader answers |
replica.fetch.wait.max.ms | 500 | How long the leader holds a follower fetch with no new data. Keep it well below replica.lag.time.max.ms |
replica.socket.receive.buffer.bytes | 65536 | Socket buffer for replication. Raise for cross-zone or high-latency links |
A single fetcher thread moves at most one response (up to 10 MiB by default) per round trip. On a busy broker, or when replicating across zones, that can cap replication throughput, and followers fall behind and drop out of the ISR. Raising num.replica.fetchers to 2 to 8 is the usual first fix (Section 12.3, "Under-Replicated Partitions").
Fetch sessions
A consumer or follower that reads hundreds of partitions would waste bandwidth if every fetch listed every partition. Incremental fetch sessions (KIP-227) let the client send only the partitions that changed since the last request, and the broker answers only for partitions with new data. The broker caches sessions in a limited number of slots:
| Setting / metric | Meaning |
|---|---|
max.incremental.fetch.session.cache.slots (default 1000) | Number of sessions a broker caches |
kafka.server:type=FetchSessionCache,name=NumIncrementalFetchSessions | Sessions in use |
kafka.server:type=FetchSessionCache,name=IncrementalFetchSessionEvictionsPerSec | Sessions evicted. Evicted clients fall back to full fetch requests |
If you run more consumer instances than slots and see continuous evictions, raise the slot count (each slot costs a little broker memory).
Keeping sizes aligned
producer broker / topic follower consumer
batch.size <= max.message.bytes (topic) <= replica.fetch.max.bytes ~ max.partition.fetch.bytes
max.request.size message.max.bytes (broker) fetch.max.bytes <= broker fetch.max.bytes
- A producer batch larger than the follower or consumer per-partition limit is still delivered (the limits are soft for the first batch), but each fetch then returns only that one batch for the partition, which makes fetching inefficient.
- The broker's own
fetch.max.bytes(default 57671680, 55 MiB) caps what any consumer can request in one response, whatever the consumer sets.
Estimating consumer memory for fetched data
# Worst-case fetched-but-unprocessed data held by one consumer instance (approximate)
def consumer_fetch_memory_mb(brokers_contacted: int, partitions_per_broker: int,
max_partition_fetch_mb: float = 1.0,
fetch_max_mb: float = 50.0) -> float:
# Each broker's response is bounded by fetch.max.bytes and by the per-partition limit
per_response = min(fetch_max_mb, partitions_per_broker * max_partition_fetch_mb)
return brokers_contacted * per_response
# A consumer reading 240 partitions spread over 6 brokers, with defaults
print(consumer_fetch_memory_mb(6, 40)) # -> 240.0 MB: too much for a 512 MB container
# Lower the per-partition limit and the response cap
print(consumer_fetch_memory_mb(6, 40, 0.5, 16.0)) # -> 96.0 MB
Example: tuning profiles side by side
# ---- Broker: replication for a busy, multi-AZ cluster ----
num.replica.fetchers=4
replica.fetch.max.bytes=4194304 # 4 MiB per partition (>= max.message.bytes of any topic)
replica.fetch.response.max.bytes=33554432 # 32 MiB per response
replica.socket.receive.buffer.bytes=-1 # OS auto-tuning for cross-AZ links
max.incremental.fetch.session.cache.slots=4000 # thousands of consumer instances
# ---- Consumer: bulk analytics sink ----
fetch.min.bytes=1048576 # wait for 1 MiB...
fetch.max.wait.ms=500 # ...or 500 ms, whichever comes first
max.partition.fetch.bytes=4194304
fetch.max.bytes=52428800
# ---- Consumer: low-latency service in a large fleet ----
fetch.min.bytes=1
fetch.max.wait.ms=200 # lower than default, but not so low that idle fetches flood brokers
max.partition.fetch.bytes=1048576
Key Takeaways
- Every fetch has a fixed cost. Fuller fetches mean fewer requests and less broker CPU per byte.
- Idle fetch rate is roughly
instances x brokers / fetch.max.wait.ms. Lowering the wait across a large fleet can overload brokers with empty fetches. - Followers are fetchers too. Raise
num.replica.fetchersand the replica fetch sizes when followers fall behind on busy or cross-zone clusters. - Watch fetch session evictions when running many consumer instances, and raise
max.incremental.fetch.session.cache.slotsif needed. - Keep producer batch sizes, topic message limits, and replica and consumer fetch limits aligned, and bound consumer memory explicitly.
Practice
- (Beginner) Start 20 console consumers on an idle topic with
fetch.max.wait.ms=500, then withfetch.max.wait.ms=10. Compare the broker'sRequestsPerSecforFetchConsumerand explain the result.- (Intermediate) On a 3-broker cluster, throttle the network between two brokers with
tcto add 20 ms of latency, produce at a high rate withacks=all, and observe follower lag (MaxLag) and produce latency. Then raisenum.replica.fetchersandreplica.socket.receive.buffer.bytes, and measure again.- Interview question: "After a large consumer fleet was migrated to new hosts, broker CPU rose by 40% while traffic stayed the same. What fetch-related causes would you check?" Hint: compare request rates and request sizes before and after, consider fetch session cache evictions, and look for changed client settings.
End-to-End Latency Tuning
Theory
End-to-end latency is the time from producer.send() in one application to the moment the record is handed to the processing code of another. It is the number users actually feel, and it is made of several stages owned by different components. Tuning one stage in isolation (for example, setting linger.ms=0) often has little effect because another stage dominates. The method is to break the path into stages, measure which ones dominate, and tune those.
Analogy: a parcel's delivery time is not just the truck's driving time. It includes waiting to be picked up, sorting at the hub, waiting for the next truck, and the last-mile delivery. Speeding up the truck does nothing if parcels wait a day at the hub.
The stages
producer app producer client leader broker follower(s) consumer
------------ --------------- ------------- ----------- --------
send() ----> [1] serialize, partition,
wait in batch (linger.ms)
and in buffer
[2] request on the wire ----> [3] queue + append to log
(RequestQueueTime + LocalTime)
[4] wait for ISR to fetch -----> fetch, append, ack
(RemoteTime; high watermark advances)
[5] consumer fetch held until data
is below the high watermark -----------------> [6] response on the wire
[7] poll() returns,
deserialize,
wait behind earlier
records in the batch,
process
A key point: consumers can only read records below the high watermark, that is, records replicated to every in-sync replica (Section 2.4). That is true whatever acks the producer uses. Setting acks=1 makes the producer's acknowledgment faster, but the record still becomes visible to consumers only after the followers have it. So lowering acks rarely improves end-to-end latency, and it costs durability.
Where the time usually goes
| Stage | Typical contribution | Settings that affect it |
|---|---|---|
| 1. Producer batching and buffering | 0 to linger.ms, plus queueing if the producer is saturated | linger.ms, batch.size, buffer.memory, compression codec |
| 2, 6. Network | Sub-millisecond in one zone, 1 to 2 ms across zones, tens of ms across regions | Placement, client.rack (fetch from a nearby follower) |
| 3. Leader queue and append | Sub-millisecond when healthy | I/O threads, disk, page cache |
| 4. Replication | About one follower round trip; dominated by the slowest ISR member | num.replica.fetchers, follower health, cross-zone latency |
| 5. Consumer fetch wait | Up to fetch.max.wait.ms when traffic is low and fetch.min.bytes is high | fetch.min.bytes, fetch.max.wait.ms |
| 7. Consumer processing | Often the largest: processing time times records ahead in the batch | max.poll.records, processing parallelism, partition count |
In steady state, replica.fetch.wait.max.ms does not add latency: a follower's pending fetch completes as soon as new data arrives on the leader.
Sources of tail latency (p99 and above)
| Cause | How it shows up | Mitigation |
|---|---|---|
| GC pauses (broker or client) | Spikes that line up with GC logs | Right-size heaps, G1 or ZGC settings (Section 12.1) |
| Page cache misses | Spikes when lagging consumers read old data | More RAM, isolate replay workloads, tiered storage for history |
| Slow follower in the ISR | RemoteTimeMs p99 high on produce | Fix or replace the slow broker, more fetchers |
| Consumer rebalances | Consumption stops for seconds | Cooperative or next-generation rebalance protocol, static membership (Section 5.2) |
| Leader elections and restarts | Brief retries and higher latency | Controlled shutdown, rolling restarts one broker at a time |
Forced fsync per message | Every write waits for disk | Leave log.flush.interval.messages and flush.messages at their defaults, rely on replication for durability |
| Head-of-line blocking in processing | One slow record delays the records behind it in the same partition | Timeouts on external calls, more partitions, parallel processing per key |
A low-latency profile
| Component | Setting | Value | Why |
|---|---|---|---|
| Producer | linger.ms | 0 to 1 | Send immediately |
| Producer | compression.type | lz4 or none | Lowest CPU cost per batch |
| Producer | acks, enable.idempotence | all, true | Keep durability; acks=1 barely helps end-to-end |
| Broker | num.replica.fetchers | 2 to 4 | Faster replication, so the high watermark advances sooner |
| Broker / topic | min.insync.replicas | 2 (with RF 3) | Durability; latency depends on the slowest ISR member regardless |
| Consumer | fetch.min.bytes / fetch.max.wait.ms | 1 / 100 to 200 | Return data as soon as it exists without flooding brokers when idle |
| Consumer | max.poll.records | Small enough that one batch is processed quickly | Records late in a batch wait for earlier ones |
| Consumer | Rebalance protocol | group.protocol=consumer or cooperative | Avoid stop-the-world pauses |
| Placement | Clients and brokers | Same region; client.rack for follower fetching | Fewer and shorter network hops |
Example: measuring end-to-end latency in your own consumer
Built-in tools measure the platform. Your users experience your application, so measure inside it. Every record carries a timestamp, set by the producer by default (CreateTime):
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.record.TimestampType;
import java.time.Duration;
import java.util.*;
public class LatencyProbeConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "latency-probe");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 100); // low-latency fetch profile
props.put(ConsumerConfig.GROUP_PROTOCOL_CONFIG, "consumer"); // next-generation rebalance protocol
List<Long> samples = new ArrayList<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> r : records) {
process(r); // the real work
if (r.timestampType() == TimestampType.CREATE_TIME) {
// Includes producer batching, broker, replication, fetch, and processing time.
// Assumes producer and consumer clocks are synchronized (NTP/chrony).
samples.add(System.currentTimeMillis() - r.timestamp());
}
}
if (samples.size() >= 10_000) {
Collections.sort(samples);
System.out.printf("e2e ms p50=%d p99=%d p99.9=%d max=%d%n",
samples.get(samples.size() / 2),
samples.get((int) (samples.size() * 0.99)),
samples.get((int) (samples.size() * 0.999)),
samples.get(samples.size() - 1));
samples.clear();
}
}
}
}
private static void process(ConsumerRecord<String, String> r) { /* business logic */ }
}
In production, record these samples in a histogram metric (for example, a Micrometer Timer exported to Prometheus) instead of printing them. If the topic uses message.timestamp.type=LogAppendTime, the timestamp is set by the broker, and the measurement excludes the producer's batching time.
A tuning procedure
- Set a target as a percentile ("p99 below 50 ms at 20,000 records/s"), not an average.
- Measure end-to-end latency in the application and the per-stage metrics: producer
record-queue-time-avgandrequest-latency-avg, broker produceTotalTimeMsphases, consumerfetch-latency-avg, and processing time. - Fix the largest stage first. Usually that is consumer processing or batching, not the broker.
- Re-measure at the production rate, not at maximum throughput, and look at p99 and p99.9, not only the median.
Key Takeaways
- End-to-end latency is a sum of stages: batching, network, leader append, replication, fetch wait, and processing. Measure them before tuning.
- Consumers only see records below the high watermark, so replication time is always part of end-to-end latency, and
acks=1buys little. - Tail latency usually comes from GC, page cache misses, slow ISR followers, rebalances, and head-of-line blocking in processing.
- Low-latency settings:
linger.msnear 0, light compression, lowfetch.max.wait.ms, more replica fetchers, modern rebalance protocols, and nearby placement. - Measure latency inside the consuming application from record timestamps, with synchronized clocks, as a percentile histogram.
Practice
- (Beginner) Run the
EndToEndLatencytool withacks=1and withacks=allon a 3-broker cluster. Explain why the difference in end-to-end latency is smaller than the difference in producer acknowledgment latency.- (Intermediate) Build the latency probe consumer above, produce at a steady 5,000 records/s, and record p50 and p99. Then add a simulated 2 ms
Thread.sleepinprocess()withmax.poll.records=500and measure again. Explain the new p99 in terms of head-of-line blocking within a batch, and propose two fixes.- Interview question: "An order notification must reach the user within 200 ms of the order being placed, and your p99 is 1.5 seconds. Walk me through how you find and fix the problem." Hint: decompose the path into stages with metrics for each, decide which stage dominates the tail, and consider low-traffic fetch behavior, rebalances, and processing.
<a id="132-capacity-planning"></a>
13.2 Capacity Planning
This section turns business requirements into concrete numbers: the traffic a cluster must handle, how many partitions each topic needs, how much disk the data occupies, how many brokers are required, and where partition counts hit their practical limits.
Estimating Throughput Requirements
Theory
Every capacity decision starts with one question: how much data will flow through the cluster? A "Kafka cluster for orders" is not a requirement. "12,000 order events per second at peak, 1.2 KB each, read by four applications, kept for 7 days" is. Converting the first into the second is the most valuable part of capacity planning, because every later number (partitions, disk, brokers) is derived from it, and errors multiply through the chain.
Producer traffic is only the starting point. Inside the cluster, each byte produced is written once by the leader, copied to followers (replication), and read by every consumer group (fan-out). A modest-looking 50 MB/s topic with replication factor 3 and five consumer groups moves 50 MB/s in, 100 MB/s of replication, and 250 MB/s out to consumers.
Analogy: planning a highway from the number of people who live in a town undercounts traffic. You need trips per person, when they travel (rush hour), whether each trip returns (round trips), and how the town will grow. Kafka's equivalents are record rate, peak factor, replication and fan-out, and growth.
The inputs
| Input | How to obtain it | Common mistake |
|---|---|---|
| Records per second (average) | Source system metrics, database change rates, API request logs | Using daily totals divided by 86,400 and stopping there |
| Peak-to-average ratio | Busiest hour (or minute) versus the daily average | Ignoring seasonal peaks (sales events, month-end batch jobs) |
| Record size | Sample real serialized records, including key and headers | Using the size of the business object in memory |
| Compression ratio | Compress a sample with the intended codec | Assuming 4:1 for data that is already compressed or encrypted |
| Replication factor | Usually 3 | Forgetting replication traffic on the network |
| Consumer groups per topic | Count of applications reading the topic, plus planned ones | Forgetting connectors, stream processors, and ad hoc tools |
| Catch-up traffic | How fast a lagging or new consumer must replay history | Planning only for real-time reads |
| Growth | Business forecast over the planning horizon (12 to 24 months) | Planning for today and buying hardware twice |
From inputs to cluster-wide traffic
logical write rate W = records/s x record size / compression ratio (bytes stored per second)
peak write rate Wp = W x peak factor x growth factor
cluster ingress (from producers) = Wp
replication traffic (broker to broker) = Wp x (RF - 1)
disk write rate, all brokers together = Wp x RF
cluster egress to consumers (real time) = Wp x consumer groups
extra egress while catching up = replay rate of lagging consumers
Compression usually happens in the producer, and the broker stores and serves the compressed batches, so compressed sizes apply to the network, disk, and replication on the broker side. Producers and consumers pay the CPU cost of compressing and decompressing.
Example: a capacity model for several topics
from dataclasses import dataclass
@dataclass
class Topic:
name: str
avg_records_s: float
record_bytes: int # serialized key + value + headers, before compression
compression_ratio: float # measured on real samples, e.g. 3.0 means 3:1
peak_factor: float # busiest period / average
consumer_groups: int
replication_factor: int = 3
GROWTH = 1.5 # expected growth over the planning horizon
MB = 1_000_000
topics = [
Topic("orders", avg_records_s=4_000, record_bytes=1_200, compression_ratio=4.0, peak_factor=3.0, consumer_groups=5),
Topic("clickstream", avg_records_s=60_000, record_bytes=700, compression_ratio=5.0, peak_factor=2.5, consumer_groups=3),
Topic("cdc.inventory", avg_records_s=2_500, record_bytes=900, compression_ratio=3.0, peak_factor=4.0, consumer_groups=2),
]
totals = {"ingress": 0.0, "replication": 0.0, "disk_writes": 0.0, "egress": 0.0}
for t in topics:
w = t.avg_records_s * t.record_bytes / t.compression_ratio / MB # MB/s stored, average
wp = w * t.peak_factor * GROWTH # MB/s at planned peak
totals["ingress"] += wp
totals["replication"] += wp * (t.replication_factor - 1)
totals["disk_writes"] += wp * t.replication_factor
totals["egress"] += wp * t.consumer_groups
print(f"{t.name:14s} avg {w:6.2f} MB/s planned peak {wp:6.2f} MB/s")
for k, v in totals.items():
print(f"{k:12s} {v:7.1f} MB/s")
# orders avg 1.20 MB/s planned peak 5.40 MB/s
# clickstream avg 8.40 MB/s planned peak 31.50 MB/s
# cdc.inventory avg 0.75 MB/s planned peak 4.50 MB/s
# ingress 41.4 MB/s
# replication 82.8 MB/s
# disk_writes 124.2 MB/s
# egress 130.5 MB/s
Add catch-up traffic on top: if a consumer of clickstream must replay 6 hours of history in 1 hour after an outage, it reads at 6 times the topic's write rate during that hour, mostly from disk.
Measuring instead of guessing
For an existing cluster, measure the real values instead of estimating them:
| Value | Where to read it |
|---|---|
| Bytes in and out per topic | BytesInPerSec / BytesOutPerSec with the topic= tag (Section 12.3) |
| Records per second per topic | MessagesInPerSec with the topic= tag |
| Average record size | BytesInPerSec / MessagesInPerSec (compressed bytes per record) |
| Peak factor | The ratio of the highest 5-minute average to the daily average over several weeks |
| Compression ratio | Producer metric compression-rate-avg (compressed size / uncompressed size) |
Key Takeaways
- Express requirements as numbers: records/s, record size, compression ratio, peak factor, retention, consumer groups, and growth.
- Plan for peak traffic times growth, not averages.
- Each produced byte becomes
RFdisk writes,RF - 1replication transfers, and one read per consumer group. - Include catch-up and replay traffic, which can exceed real-time traffic for hours.
- On existing clusters, measure with per-topic broker metrics and producer compression metrics.
Practice
- (Beginner) A topic receives 20,000 records/s of 500 bytes, compresses 4:1, has a peak factor of 2, RF 3, and 3 consumer groups. Compute the peak ingress, replication traffic, disk writes, and consumer egress in MB/s.
- (Intermediate) Extend the Python model to read topic definitions from a YAML file and add a "catch-up" term: one consumer group per topic must be able to replay 4 hours of data in 1 hour. Report how much this adds to egress.
- Interview question: "A product team asks for a new Kafka topic and says it will be 'about a million events a day.' What do you ask them before you agree it fits on the existing cluster?" Hint: a daily total hides size, peaks, compression, readers, retention, and growth. List the questions that turn it into a peak MB/s and a disk figure.
Sizing Partitions
Theory
Section 3.3 introduced the per-topic rule partitions >= max(T/p, T/c): enough partitions that producers (at p MB/s per partition) and consumers (at c MB/s per partition) can keep up with target throughput T. Capacity planning applies that rule across the whole cluster and adds three refinements: how to measure p and c realistically, how to size consumers whose limit is processing time rather than bytes, and how partition counts interact with batching, broker balance, and the cluster's partition budget.
Analogy: when planning a new office building, you size each department's rooms first (per-topic sizing), then check that the total fits the building's floors, elevators, and fire code (the cluster's budget). A plan where every department gets its ideal size can still be impossible as a whole.
Measuring per-partition rates
| Rate | How to measure | Typical range |
|---|---|---|
p: produce MB/s per partition | Producer perf test against a 1-partition topic with production settings (acks=all, compression, RF) | 10 to 50+ MB/s; limited by one leader's disk and one producer batch pipeline |
c: consume MB/s per partition | Run your consumer (with its real processing) on a 1-partition topic | From well under 1 MB/s (each record calls a database) to tens of MB/s (simple sinks) |
The consumer side almost always decides the count, because real processing is much slower than Kafka.
Processing-bound consumers: Little's Law again
When each record takes a known time to process, the required parallelism is:
required concurrent processors = records/s x processing time per record
3,000 records/s x 20 ms per record = 60 records being processed at any moment
-> with one record at a time per partition (a classic consumer group), you need at least 60 partitions
-> or fewer partitions with parallel processing inside each consumer (Section 5.4),
or a share group, where parallelism is not tied to partition count
Partition count and batching
Producer batches are built per partition. Spreading a fixed throughput over more partitions makes each partition's batch fill more slowly:
keyed topic, 20 MB/s, linger.ms=5
24 partitions -> 0.83 MB/s per partition -> about 4.2 KB per batch in 5 ms
480 partitions -> 0.04 MB/s per partition -> about 0.2 KB per batch in 5 ms
-> 20x more requests and partition entries for the same bytes, worse compression,
more broker CPU per MB
Unkeyed topics are less affected, because the sticky partitioner fills one partition's batch at a time (Section 4.2). Keyed, high-partition-count topics with modest traffic are where over-partitioning costs the most. Producer memory also grows with partition count: in the worst case, a producer holds up to batch.size of open batch memory for every partition it writes to.
Balancing across brokers
- Choose counts that divide evenly by the number of brokers (or a multiple of it), so every broker leads the same number of partitions of a busy topic. 24 partitions on 6 brokers puts 4 leaders on each broker. 20 partitions on 6 brokers puts 4 on some and 3 on others, so two brokers carry 33% more of that topic.
- Prefer counts with many divisors (12, 24, 48, 96) so consumer instances also get even shares (Section 3.3).
Example: planning partitions for a set of topics and checking the cluster budget
import math
BROKERS = 6
MAX_REPLICAS_PER_BROKER = 4000 # the budget you have validated on your hardware (see "Partition Count Limits")
NICE_COUNTS = [3, 6, 12, 24, 48, 96, 192] # many divisors, multiples of the broker count where possible
def plan(target_mb_s, p_mb_s, c_mb_s, max_consumers, keyed, growth=2.0):
need = max(target_mb_s / p_mb_s, target_mb_s / c_mb_s, max_consumers)
if keyed:
need *= growth # keyed topics cannot grow later without remapping keys
return next(n for n in NICE_COUNTS if n >= math.ceil(need))
topics = {
# name: (peak MB/s, p, c, max consumer instances, keyed)
"orders": (5.4, 25, 1.5, 12, True),
"clickstream": (31.5, 30, 8, 8, False),
"cdc.inventory": (4.5, 25, 2, 4, True),
"audit": (0.2, 25, 5, 2, False),
}
total_replicas = 0
for name, args in topics.items():
n = plan(*args)
total_replicas += n * 3 # RF 3
print(f"{name:14s} {n:4d} partitions")
per_broker = total_replicas / BROKERS
print(f"replicas per broker: {per_broker:.0f} of a {MAX_REPLICAS_PER_BROKER} budget")
# orders 24 partitions (12 consumer instances set the floor, x2 growth because it is keyed)
# clickstream 12 partitions (8 consumers -> next count with many divisors)
# cdc.inventory 12 partitions (4 consumers, x2 growth because it is keyed)
# audit 3 partitions
# replicas per broker: 26 of a 4000 budget
A real cluster has hundreds of topics, plus internal topics (__consumer_offsets has 50 partitions by default, and Connect and Streams create their own). Run the same check over all of them.
Key Takeaways
- Measure per-partition produce and consume rates with your real settings and processing logic. Consumers usually decide the count.
- For processing-bound consumers, required parallelism equals record rate times processing time per record.
- More partitions mean smaller producer batches for keyed topics, more requests, and more producer memory. Do not over-partition modest topics.
- Pick counts that divide evenly by the broker count and by likely consumer group sizes.
- Size each topic, then sum replicas across the cluster and compare with the per-broker budget.
Practice
- (Beginner) A consumer processes each record in 15 ms, one at a time per partition, and the topic peaks at 2,000 records/s. What is the minimum partition count for a classic consumer group? Which count would you pick for a 6-broker cluster, and why?
- (Intermediate) Produce a keyed workload at a fixed 10 MB/s into topics with 12, 96, and 768 partitions (same
linger.msandbatch.size). Compare producerbatch-size-avg,requests-per-second(request-rate), broker CPU, and the compression ratio. Explain the trend.- Interview question: "Why not give every topic 100 partitions so we never have to worry about scaling?" Hint: think about batching, producer memory, files and memory maps on brokers, controller metadata, failover time, and the cluster-wide partition budget.
Sizing Storage
Theory
Disk is the resource Kafka runs out of most often, and running out is one of the most damaging failures: a broker with a full log directory takes that directory offline, and the partitions on it lose a replica (Section 12.4). Storage sizing answers "how many bytes will the cluster hold at steady state, and how much room is needed around that?" The core calculation is simple: write rate times retention times replication factor. The details that matter are the extras: segment granularity, compacted topics, uneven distribution, and headroom for failures.
Analogy: sizing a warehouse means calculating how many pallets arrive per day and how long each stays, then adding room for aisles, for the day deliveries arrive before anything ships out, and for the neighbor's stock when their warehouse floods.
The base formula
stored bytes (cluster-wide) = average write rate (compressed) x retention period x RF
per broker = stored bytes / number of brokers (if replicas are balanced)
Use the AVERAGE write rate for time-based retention: retention holds a window of time,
so a peak hour fills only one hour of the window. Use the peak rate for throughput sizing.
What the base formula misses
| Factor | Effect | How to account for it |
|---|---|---|
| Segment granularity | Retention deletes whole closed segments only. A partition can hold up to retention plus one segment | Add segment.bytes per partition (1 GiB by default), or rely on segment.ms for slow partitions |
Slow partitions and segment.ms | A partition that writes little rolls its active segment only after segment.ms (7 days by default), so it keeps data much longer than retention.ms | Set segment.ms close to retention.ms on low-volume topics with short retention |
| Compacted topics | Size depends on distinct keys, not time. Uncompacted "dirty" data adds up to about 1 / (1 - min.cleanable.dirty.ratio) times the clean size | distinct keys x average record size x 2 (with the default ratio of 0.5) as a starting estimate |
| Index files | Offset and time indexes; small relative to data | A few percent at most |
| Uneven placement | Partitions differ in size and brokers (and JBOD directories) are not perfectly balanced | Size for the fullest broker or directory, not the average |
| Broker failure | After a reassignment, survivors absorb the failed broker's replicas | Each broker needs room for 1/(N-1) more data |
| Tiered storage | Only local.retention.* stays on broker disks; the rest moves to object storage (Section 7.3) | Local disk = write rate x local retention x RF; remote = write rate x total retention (one copy) |
Headroom targets
| Utilization | Meaning |
|---|---|
| Below 60% | Normal operating range: room for growth, failure absorption, and reassignments |
| 60% to 75% | Plan expansion now (adding brokers and rebalancing takes days to weeks) |
| Above 75% to 80% | Alert: one failure or traffic spike from trouble |
| Above 90% | Emergency (Section 12.4) |
Example: a storage calculator
GiB = 2**30
TiB = 2**40
def storage_per_broker_tib(avg_write_mb_s, retention_days, rf, brokers, partitions,
segment_bytes=1 * GiB, imbalance=1.15, target_utilization=0.6):
base = avg_write_mb_s * 1e6 * retention_days * 86_400 * rf # bytes, cluster-wide
open_segments = partitions * rf * segment_bytes # worst case: one undeletable segment per replica
per_broker = (base + open_segments) / brokers * imbalance # size for the fullest broker
after_failure = per_broker * brokers / (brokers - 1) # survivors absorb one broker's replicas
return per_broker / TiB, after_failure / TiB, after_failure / target_utilization / TiB
used, used_after_failure, disk_needed = storage_per_broker_tib(
avg_write_mb_s=14, retention_days=7, rf=3, brokers=6, partitions=300)
print(f"steady state per broker: {used:5.1f} TiB")
print(f"after losing one broker: {used_after_failure:5.1f} TiB")
print(f"disk to provision (60% target): {disk_needed:5.1f} TiB per broker")
# steady state per broker: 4.6 TiB
# after losing one broker: 5.5 TiB
# disk to provision (60% target): 9.2 TiB per broker
The open-segment term is small here (about 0.9 TiB of 24 TiB), but it dominates for clusters with many low-volume partitions: 10,000 partitions x RF 3 x 1 GiB is about 29 TiB of worst-case overhead, even if those partitions hold almost no retained data.
Example: compacted topic estimate
customer-profiles: 20 million distinct keys, average compressed record 800 bytes, RF 3
clean size = 20,000,000 x 800 B = 16 GB
with dirty data (min.cleanable.dirty.ratio = 0.5) up to about 2x = 32 GB
x RF 3 = 96 GB cluster-wide
+ growth in distinct keys over the planning horizon
Example: checking actual usage per broker
# Total bytes stored per broker, from the log directory descriptions (JSON on the last line)
bin/kafka-log-dirs.sh --bootstrap-server broker1:9092 --describe \
| grep '^{' \
| python3 -c '
import json, sys
for broker in json.load(sys.stdin)["brokers"]:
total = sum(p["size"] for d in broker["logDirs"] for p in d["partitions"])
print(f"broker {broker[\"broker\"]}: {total / 2**40:.2f} TiB")'
Compare these figures with the model regularly. A model that has drifted from reality is worse than no model, because it creates false confidence.
Disk throughput, not just capacity
Capacity is only half of storage sizing. Each broker's disks must also sustain its share of Wp x RF peak writes (from the throughput model) plus reads from consumers that miss the page cache. Cloud block volumes in particular often have throughput limits per volume and per instance that are reached long before the volume is full (Section 12.1, "Disk and Filesystem Choices").
Key Takeaways
- Base storage = average compressed write rate x retention x RF, divided by brokers.
- Add one segment per replica for undeletable active segments, and set
segment.mson low-volume topics with short retention so data is not kept far longer than intended. - Estimate compacted topics from distinct keys and record size, about doubled for uncompacted data.
- Size for the fullest broker after losing one broker, and keep normal utilization below about 60%.
- With tiered storage, broker disks only need to hold local retention. Check disk throughput limits as well as capacity.
Practice
- (Beginner) A topic writes an average of 5 MB/s (compressed), with 3 days of retention and RF 3, on a 3-broker cluster. How much disk does each broker use at steady state, and how much should you provision at a 60% utilization target?
- (Intermediate) Create a topic with
retention.ms=3600000(1 hour) and default segment settings, produce a few records per minute for several hours, and check its size and oldest offset withkafka-log-dirs.shandkafka-get-offsets.sh. Explain why data older than one hour is still present, then fix it withsegment.ms.- Interview question: "Your storage model says brokers should be 50% full, but one broker is at 85%. What could explain the difference, and what would you do?" Hint: consider partition size skew, leadership versus replica placement, JBOD directory imbalance, slow partitions not rolling segments, and compacted topics that are not being cleaned.
Sizing Brokers
Theory
Once throughput, partitions, and storage are known, the remaining question is how many brokers to run, and how big each should be. A broker has five resources that can run out: network bandwidth, disk throughput, disk capacity, memory for the page cache, and CPU. Each resource gives its own answer to "how many brokers do I need?", and the cluster needs the largest of those answers. Then come the constraints that have nothing to do with load: surviving a broker failure, spreading across availability zones, and keeping per-broker partition counts within limits.
Analogy: choosing how many trucks a moving company needs. You check the total volume of furniture, the total weight, the number of drivers, and the fuel range. Whichever limit is hit first decides the fleet size, and then you add a spare truck because one is always in the shop.
The per-resource calculation
brokers needed for resource R = cluster demand for R / (usable capacity of R per broker)
usable capacity = raw capacity x target utilization (for example 0.6 to 0.7)
brokers = max(network, disk throughput, disk capacity, memory, CPU, partition budget)
then round up for failure tolerance and zones
| Resource | Cluster demand (from earlier topics) | Per-broker capacity |
|---|---|---|
| Network in | Wp x RF (producer traffic plus replication received) | NIC bandwidth (and the cloud instance's sustained, not burst, limit) |
| Network out | Wp x (RF - 1) + Wp x consumer groups + catch-up | Same NIC, other direction |
| Disk throughput | Wp x RF writes plus cold reads | Sum of disk or volume throughput, capped by the instance's storage bandwidth |
| Disk capacity | Storage model, including headroom | Provisioned disk |
| Memory | Page cache for the "hot" read window: write rate x seconds consumers trail behind (Section 12.1) | RAM minus heap (about 6 GB) and OS |
| CPU | Hard to compute in advance; driven by request rate, TLS, compression | Measure with benchmarks; keep sustained usage below about 60% to 70% |
| Partitions | Total replicas from the partition plan | Your validated replicas-per-broker budget |
Failure headroom
With N brokers, losing one pushes its leaders and traffic onto the other N - 1. If the cluster runs at utilization u normally, it runs at about u x N / (N - 1) after a failure, and higher while the replacement re-replicates data. On a 3-broker cluster at 60%, one failure means 90%. On a 12-broker cluster at 60%, it means 65%. This is a strong argument for more, smaller brokers, as long as the cluster stays manageable.
Zones
In a cloud region with three availability zones, use a broker count that is a multiple of 3 and set broker.rack to the zone (Section 2.2), so each partition's replicas land in different zones and losing a whole zone leaves two replicas of every partition.
Scale up or scale out?
| Fewer, larger brokers | More, smaller brokers |
|---|---|
| Fewer machines to operate and upgrade | Smaller impact of losing one broker |
| More page cache per broker | Faster recovery: less data to re-replicate per failed broker |
| Large per-broker data volumes make recovery and reassignment slow | More connections and metadata overhead |
| Bigger instances often have higher network and disk limits | Small cloud instances often have burstable network that throttles under sustained load |
A common middle ground is 6 to 30 brokers with 8 to 32 cores, 32 to 128 GB of RAM, and network of 10 to 25 Gbit/s each, but your benchmark decides.
Example: computing the broker count
import math
# Planned peak demand (from the throughput and storage models)
demand = {
"net_in_mb_s": 41.4 * 3, # Wp x RF
"net_out_mb_s": 41.4 * 2 + 130.5 + 60, # replication out + consumer egress + catch-up
"disk_write_mb_s": 41.4 * 3,
"disk_tib": 25.0 * 1.0 / 0.6, # storage model, provisioned at 60% target
"page_cache_gb": 41.4 * 3 * 60 / 1000, # 60 s of all writes kept hot, cluster-wide
"replicas": 3_600,
}
# Candidate instance type (check the provider's sustained limits, not burst limits)
instance = {
"net_in_mb_s": 1_250, # 10 Gbit/s
"net_out_mb_s": 1_250,
"disk_write_mb_s": 600, # instance storage bandwidth
"disk_tib": 8,
"page_cache_gb": 64 - 6 - 4, # RAM minus heap minus OS
"replicas": 4_000,
}
UTILIZATION = 0.6 # plan to use at most 60% of each resource
needed = {k: math.ceil(demand[k] / (instance[k] * (UTILIZATION if k != "disk_tib" else 1)))
for k in demand} # disk_tib already includes its headroom
for k, n in needed.items():
print(f"{k:16s} {n} brokers")
brokers = max(needed.values())
brokers = max(brokers, 3) # minimum for RF 3
brokers = math.ceil(brokers / 3) * 3 # multiple of 3 zones
print("planned brokers:", brokers)
# net_in_mb_s 1 brokers
# net_out_mb_s 1 brokers
# disk_write_mb_s 1 brokers
# disk_tib 6 brokers
# page_cache_gb 1 brokers
# replicas 2 brokers
# planned brokers: 6
In this example, disk capacity decides the broker count: a typical result for clusters with long retention. If retention were cut to a day or moved to tiered storage, the same workload would fit on three brokers. Clusters with short retention and heavy fan-out are more often limited by network egress.
Validate with a benchmark
The model gives a candidate size. Before committing, run the step-load benchmark from Section 13.1 on that instance type at the planned peak, with one broker stopped, and confirm the latency target still holds.
Key Takeaways
- Compute the broker count separately for network in, network out, disk throughput, disk capacity, memory, CPU, and partitions. The largest answer wins.
- Plan with target utilization (about 60% to 70%), and check what happens to the remaining brokers when one fails:
u x N / (N - 1). - Use a multiple of the zone count with
broker.rackset, so a zone failure leaves every partition available. - More, smaller brokers reduce the impact of a failure. Fewer, larger ones are simpler to run. Watch for burstable network limits on small cloud instances.
- Validate the planned size with a benchmark at peak load, including a failed broker.
Practice
- (Beginner) A cluster runs 4 brokers at 65% network utilization. What utilization do the remaining brokers reach when one fails? How many brokers would keep post-failure utilization below 80%?
- (Intermediate) Using the Python model, compare two instance types for the same demand: one with 4 TiB of disk and 25 Gbit/s, and one with 16 TiB and 10 Gbit/s. Which needs fewer brokers, which resource limits each, and which would you choose given recovery time after a broker failure?
- Interview question: "Your 3-broker cluster is at 40% CPU, 50% network, and 70% disk. Leadership asks whether it can absorb a 50% traffic increase. What is your answer?" Hint: scale each resource by the growth, add the failure scenario with only two brokers left, and remember that disk usage depends on retention as well as traffic.
Partition Count Limits
Theory
Partitions are not free. Each partition replica is a directory of segment files on a broker, open file handles, memory-mapped index files, an entry in the cluster metadata, a unit of work for replica fetchers, and an object that must be loaded at startup and moved during failover. One more partition costs almost nothing. Tens of thousands per broker add up to slow restarts, slow failovers, memory pressure, and operating system limits. Knowing where those limits are, and how to stay under them, matters for any cluster that hosts many topics or many teams.
Historically the binding limit was ZooKeeper: every leader change was a ZooKeeper write, so failover time grew with partition count, and common guidance was about 4,000 partitions per broker and 200,000 per cluster. KRaft (Section 2.3) removed most of the cluster-wide limit, since metadata is a replicated log and leader changes are batched records, and clusters with millions of partitions have been demonstrated. Per-broker costs remain, so most operators still treat a few thousand replicas per broker as comfortable and validate anything above roughly 10,000 per broker with their own tests.
Analogy: a library can catalog millions of books in a computerized index (KRaft metadata), but each branch still has a fixed number of shelves, and moving a branch's whole collection to another building takes longer the more books it holds (per-broker files, recovery, and failover).
Where each partition costs
| Layer | Cost per partition replica | What breaks first |
|---|---|---|
| Files | At least one open file per segment's .log, plus a directory | File descriptor limit (nofile) |
| Memory maps | Offset index and time index per segment (mapped when needed) | vm.max_map_count (default 65,530): Map failed errors |
| Broker memory | Index buffers, per-partition state, fetch session entries | Heap pressure, longer GC |
| Replication | Each follower replica is fetched by a replica fetcher thread | Fetchers fall behind, ISR shrinks |
| Startup and shutdown | Every segment is loaded at startup and recovered after an unclean stop | Long restarts and long rolling upgrades |
| Failover | Leaders on a failed broker must be elected elsewhere, and clients must refresh metadata | Longer unavailability windows during failures and controlled shutdowns |
| Controller | Partition and replica records in the metadata log and snapshots | Larger snapshots, slower propagation to brokers |
| Clients | Producer batch memory per partition, consumer fetch state, larger metadata responses | Client memory and slower rebalance assignment |
Estimating operating system limits
segments per replica ~ retention bytes per partition / segment.bytes + 1 (active)
memory maps per broker ~ replicas per broker x segments per replica x 2 (offset + time index)
file handles per broker ~ replicas per broker x segments per replica + client connections
5,000 replicas x 6 segments x 2 = 60,000 maps -> above 90% of the 65,530 default: raise vm.max_map_count
5,000 replicas x 6 segments + 20,000 connections = 50,000 files -> raise nofile well above it
Checking current counts
BS=broker1:9092
# Replicas and leaders per broker, from the topic descriptions
bin/kafka-topics.sh --bootstrap-server $BS --describe \
| grep -o 'Replicas: [0-9,]*' | cut -d' ' -f2 | tr ',' '\n' \
| sort -n | uniq -c | awk '{print "broker " $2 ": " $1 " replicas"}'
bin/kafka-topics.sh --bootstrap-server $BS --describe \
| grep -o 'Leader: [0-9-]*' | cut -d' ' -f2 \
| sort -n | uniq -c | awk '{print "broker " $2 ": " $1 " leaders"}'
# The same numbers from JMX on each broker
# kafka.server:type=ReplicaManager,name=PartitionCount
# kafka.server:type=ReplicaManager,name=LeaderCount
# Memory maps and open files of the running broker, versus the limits
PID=$(pgrep -f 'kafka\.Kafka')
echo "maps: $(wc -l < /proc/$PID/maps) / $(cat /proc/sys/vm/max_map_count)"
echo "fds: $(ls /proc/$PID/fd | wc -l) / $(awk '/open files/ {print $4}' /proc/$PID/limits)"
Managed services
Managed Kafka services publish their own limits, usually as a maximum partition count per broker that depends on the instance size, or a maximum per cluster for serverless tiers. Exceeding them can block topic creation or void performance guarantees, so include them in the partition budget (Chapter 14).
Staying within the budget
| Situation | Remedy |
|---|---|
| Many low-volume topics with default (or large) partition counts | Lower num.partitions for new topics; create small topics with 1 to 6 partitions |
| A topic was over-partitioned at creation | Partition counts cannot be reduced: create a new topic with fewer partitions, migrate producers and consumers, then delete the old one |
| Many tiny topics that differ only by a name suffix (per customer, per tenant) | Consolidate into one topic keyed by tenant, where access control and retention allow it |
| Unused topics | Find topics with no traffic and no consumer groups over weeks, confirm ownership, delete |
| Replica count grows with the business | Add brokers and reassign (Section 12.2), keeping replicas per broker within budget |
| Many segments per partition | Larger segment.bytes for high-volume topics, or tiered storage to keep fewer segments locally |
Enforcing the budget
Automated topic creation (Section 3.3) is the best place to stop the problem before it starts. A simple pre-creation check in a topic provisioning pipeline:
def check_partition_budget(current_replicas_per_broker: dict[int, int],
new_partitions: int, rf: int,
budget_per_broker: int = 4000) -> None:
brokers = len(current_replicas_per_broker)
added_per_broker = new_partitions * rf / brokers # assumes balanced placement
worst = max(current_replicas_per_broker.values()) + added_per_broker
if worst > budget_per_broker:
raise ValueError(f"Topic would bring a broker to {worst:.0f} replicas "
f"(budget {budget_per_broker}). Use fewer partitions or add brokers.")
current = {1: 3850, 2: 3900, 3: 3800, 4: 3820, 5: 3880, 6: 3790} # busiest broker at 3,900
check_partition_budget(current, 96, 3) # +48 per broker -> 3,948: accepted
check_partition_budget(current, 384, 3) # +192 per broker -> 4,092
# ValueError: Topic would bring a broker to 4092 replicas (budget 4000). Use fewer partitions or add brokers.
Key Takeaways
- Each partition replica costs files, memory maps, memory, replication work, startup time, failover time, and metadata.
- KRaft removed most cluster-wide limits, but per-broker costs remain. Keep replicas per broker within a budget you have tested.
- Check
vm.max_map_countand file descriptor limits againstreplicas x segments, and watchPartitionCountper broker. - Partition counts cannot be reduced. Recover from over-partitioning by migrating to a new topic.
- Enforce the budget at topic creation time, and include managed-service limits when they apply.
Practice
- (Beginner) Count replicas and leaders per broker on a test cluster with the commands above. Create a topic with 60 partitions and RF 3 and count again. Are the new replicas and leaders evenly spread?
- (Intermediate) On a test broker, create enough small partitions (for example, 3,000 replicas with
segment.bytes=1048576and a little data in each) to push memory maps near the limit. Lowervm.max_map_counttemporarily to trigger the failure, read the broker log, then measure how long a controlled restart takes compared with an empty broker.- Interview question: "A platform team runs one Kafka cluster for 200 product teams, and replica counts per broker double every six months. What policies and technical changes would you introduce?" Hint: consider defaults and quotas at topic creation, consolidating per-tenant topics, deleting unused topics, scaling out brokers, tiered storage, and splitting into multiple clusters (Chapter 14).
<a id="14-multi-cluster-and-deployment-architectures"></a>