7. Storage Internals
Every guarantee Kafka offers (durability, ordering, replay, high throughput) comes down to how a broker lays bytes out on disk and moves them to the network. This chapter opens up the partition log: the segment and index files, the binary format of record batches, and the operating system features (zero-copy and the page cache) that make Kafka fast. It then covers how data leaves the log through retention, compaction, and tombstones, and how tiered storage moves older segments to object storage. Knowing these internals is what lets you size disks, explain why a record has not been deleted yet, and work out why a lagging consumer slows down a whole broker.
<a id="71-log-structure"></a>
7.1 Log Structure
This section follows a record from the moment the broker appends it to the moment a consumer reads it: the files it lands in, the indexes that find it again, the bytes it is encoded as, and the kernel mechanisms that carry it to the network.
Log Segments
Theory
Section 2.1 showed that each partition is stored as a series of segments, each a .log file with companion index files named after its base offset. This topic looks at segments as the broker manages them: when they are created, how they survive a crash, and how they are removed.
Why not one file per partition? Three operations would be expensive on a single huge file:
| Operation | Single file | Segmented log |
|---|---|---|
| Delete old data | Rewrite or punch holes in the file | Delete whole old files |
| Find an offset | Search one giant index | Pick the segment, then search a small index |
| Recover after a crash | Re-validate the entire file | Re-validate only segments written after the last checkpoint |
| Compact | Rewrite everything | Rewrite a few closed segments at a time |
Analogy: a warehouse that stores goods in sealed, dated shipping containers instead of one endless pile. New goods go into the one open container. Once a container is sealed it never changes, so it can be inspected, moved, or scrapped as a single unit.
Segment lifecycle
create roll retention / compaction
(none) ------> ACTIVE ------------------> CLOSED ------------------------> DELETED
- appended to - immutable - renamed *.deleted
- indexes grow - indexes trimmed to size - removed after
- never deleted - eligible for retention, file.delete.delay.ms
or compacted compaction, tiered upload (default 60 s)
The broker rolls the active segment (closes it and opens a new one) when any of these conditions holds for the next batch:
| Trigger | Setting (broker / topic) | Default |
|---|---|---|
| Segment would exceed the size limit | log.segment.bytes / segment.bytes | 1 GiB |
| First record in the segment is older than the age limit (minus jitter) | log.roll.hours / segment.ms | 7 days |
| Random jitter subtracted from the age limit | log.roll.jitter.hours / segment.jitter.ms | 0 |
| Offset or time index file is full | log.index.size.max.bytes / segment.index.bytes | 10 MiB |
| Batch offset no longer fits as a 4-byte offset relative to the base offset | (not configurable) | more than 2^31 - 1 offsets past the base |
segment.jitter.ms exists because many partitions created at the same moment would otherwise roll at the same moment, causing a burst of file creation and index trimming. A random jitter spreads those rolls out.
Checkpoint files and crash recovery
Kafka does not fsync every write (see Page Cache Usage below). After a crash, the broker must work out which bytes on disk can be trusted. It does this with small checkpoint files at the root of each log directory:
/var/lib/kafka/data/
|-- meta.properties <- cluster ID, node ID, directory ID (Section 2.3)
|-- recovery-point-offset-checkpoint <- per partition: offset up to which data is known flushed
|-- replication-offset-checkpoint <- per partition: high watermark (Section 2.4)
|-- log-start-offset-checkpoint <- per partition: first offset still readable
|-- cleaner-offset-checkpoint <- per partition: how far compaction has progressed
|-- .kafka_cleanshutdown <- marker written on clean shutdown
|-- orders-0/
`-- orders-1/
On startup:
- If the clean shutdown marker exists, the broker trusts all segments and only loads them.
- Otherwise, for each partition it re-reads every segment after the recovery point, validates each batch's CRC, rebuilds the
.indexand.timeindexfiles, and truncates the log at the first corrupt or partial batch. - Data lost by truncation is restored from the partition leader through normal replication.
Recovery of a broker with thousands of partitions can take a long time, which is why num.recovery.threads.per.data.dir exists: it runs recovery in parallel per log directory.
Operating system limits
Every segment keeps file descriptors open, and its .index and .timeindex are memory-mapped. A broker with 4,000 partition replicas and 50 segments each has 200,000 segments, meaning roughly 400,000 memory maps and more open files than Linux defaults allow:
| Limit | Linux default | Typical Kafka setting |
|---|---|---|
Open files (ulimit -n) | 1,024 | 100,000 or more |
Memory maps (vm.max_map_count) | 65,530 | 262,144 or more |
When the map limit is exceeded, the broker fails with java.io.IOException: Map failed / OutOfMemoryError: Map failed, even though plenty of heap is free.
Example: inspecting segments on a running broker
# Size of every partition replica on broker 1, as JSON
bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 \
--describe --broker-list 1 --topic-list orders
# The checkpointed recovery points: version, entry count, then "topic partition offset"
cat /var/lib/kafka/data/recovery-point-offset-checkpoint
# 0
# 3
# orders 0 368769
# orders 1 351200
# orders 2 360004
# Segment count per partition directory: watch for directories with thousands of segments
for d in /var/lib/kafka/data/*-[0-9]*; do
printf '%6d %s\n' "$(ls "$d"/*.log 2>/dev/null | wc -l)" "$(basename "$d")"
done | sort -rn | head
A Python model of the roll decision:
from dataclasses import dataclass
@dataclass
class Segment:
base_offset: int
size_bytes: int
first_record_ts: int | None # None while the segment is still empty
index_full: bool = False
def should_roll(seg: Segment, batch_size: int, batch_last_offset: int, now_ms: int,
segment_bytes=1 << 30, segment_ms=7 * 24 * 3600 * 1000, jitter_ms=0) -> bool:
if seg.size_bytes > segment_bytes - batch_size: # would overflow the size limit
return True
if seg.size_bytes > 0 and seg.first_record_ts is not None \
and now_ms - seg.first_record_ts > segment_ms - jitter_ms: # too old
return True
if seg.index_full: # .index or .timeindex is full
return True
if batch_last_offset - seg.base_offset > 2**31 - 1: # relative offset no longer fits in 4 bytes
return True
return False
seg = Segment(base_offset=368_769, size_bytes=1_073_730_000, first_record_ts=0)
print(should_roll(seg, batch_size=16_384, batch_last_offset=400_000, now_ms=1_000)) # True: size limit
Key Takeaways
- Segments make deletion, lookup, recovery, and compaction cheap, because they act on small immutable files instead of one large one.
- The active segment rolls on size, age (measured from its first record, minus jitter), a full index, or relative offset overflow.
- Checkpoint files record recovery points, high watermarks, and log start offsets. After an unclean shutdown, only segments past the recovery point are re-validated, and a corrupt tail is truncated and refetched from the leader.
- Deleted segments are first renamed with a
.deletedsuffix and removed asynchronously. - Segment count drives file descriptor and memory map usage. Raise
ulimit -nandvm.max_map_counton production brokers.
Practice
- (Beginner) Create a topic with
segment.bytes=1048576(1 MiB), produce about 10 MB withkafka-producer-perf-test.sh, and list the partition directory. How many segments do you see, and which one is active?- (Intermediate) A broker with 6,000 partition replicas crashed and takes 40 minutes to start. Explain what it is doing during that time, which files decide how much work there is, and two configuration changes that would shorten it.
- Interview question: "Why does Kafka make closed segments immutable, and what would break if it allowed in-place updates?" Hint: consider page cache coherence, replication by byte copying, index validity, and what compaction does instead.
Offset and Time Indexes
Theory
Consumers ask for data in two ways: "give me everything from offset N" (the normal fetch) and "give me everything since 09:00" (offsetsForTimes, used to rewind a consumer group to a point in time). A .log file is just concatenated batches with no structure for jumping to a position, so each segment carries two indexes, one for each kind of question.
Why are they sparse? A dense index (one entry per record) for a 1 GiB segment of 100-byte records would hold about 10 million entries, 80 MB per segment, mostly unused, because consumers read sequentially. A sparse index gets within a few KB of the target, and a short sequential scan of a few KB costs almost nothing.
Analogy: the thumb tabs on a dictionary. They do not list every word, only where each letter starts. You jump to the right tab, then flip a few pages.
Entry formats
| Index | Entry | Size | Maps |
|---|---|---|---|
.index | relative offset (int32), physical position (int32) | 8 bytes | offset -> byte position in .log |
.timeindex | timestamp (int64), relative offset (int32) | 12 bytes | timestamp -> offset |
.txnindex | version, producer ID, first offset, last offset, last stable offset | 34 bytes | aborted transaction ranges (Chapter 8) |
Rules the broker follows when writing them:
- An entry is added after roughly every
index.interval.bytes(default 4,096) bytes appended to the.log. - Entries in
.timeindexstore the largest timestamp seen so far in the segment, so the index is monotonic even if producers send records with out-of-order timestamps. An entry is only written when that maximum increases. - While a segment is active, its index files are preallocated to
segment.index.bytes(10 MiB) and memory-mapped. On roll they are trimmed to their real size. - Positions are 4-byte integers, which is one reason a segment cannot exceed 2 GiB.
Lookup by offset
Section 2.1 showed the three steps: binary search the in-memory list of segment base offsets, binary search the segment's .index for the largest entry at or below the target, then scan the .log forward. One refinement matters in production: the broker first searches only the last 8 KB of the index (the "warm" section). Nearly all fetches are for recent offsets, so this keeps lookups within a few pages that are always in memory, instead of touching pages spread across the whole mapped file.
Lookup by time
offsetsForTimes(orders-0, 09:00:00)
1. Find the first segment whose largest timestamp >= 09:00:00
segment 0 max ts 08:12 -> too old
segment 368769 max ts 09:47 -> this one
2. In its .timeindex, find the greatest entry with timestamp <= 09:00:00
(08:58:31 -> rel 1200) (09:00:04 -> rel 1450)
^ use relative offset 1200 -> absolute offset 369969
3. Translate offset 369969 to a byte position with the .index (as above)
4. Scan forward to the first record whose timestamp >= 09:00:00 and return its offset
The result is "the earliest offset whose timestamp is at or after the target". If no such record exists, the answer is null in Java (or offset -1 on the wire).
Index corruption and rebuild
Indexes can be derived entirely from the .log file. If one is missing or fails a sanity check at startup (for example, entries not increasing), the broker deletes and rebuilds it. This is why deleting a .index file by hand is safe, and why a broker with many corrupt indexes starts slowly.
Example: rewinding by time from code and CLI
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
TopicPartition tp = new TopicPartition("orders", 0);
consumer.assign(List.of(tp));
long twoHoursAgo = Instant.now().minus(Duration.ofHours(2)).toEpochMilli();
Map<TopicPartition, OffsetAndTimestamp> found =
consumer.offsetsForTimes(Map.of(tp, twoHoursAgo)); // uses .timeindex on the leader
OffsetAndTimestamp oat = found.get(tp);
if (oat != null) {
consumer.seek(tp, oat.offset()); // first record at or after the target time
} else {
consumer.seekToEnd(List.of(tp)); // nothing that recent: start at the end
}
}
# Offset per partition for a timestamp (ms since epoch); -1 = latest, -2 = earliest
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 \
--topic orders --time 1761210942120
# Decode an index file and check it against the log
bin/kafka-dump-log.sh --files /var/lib/kafka/data/orders-0/00000000000000368769.timeindex
# timestamp: 1761210942120 offset: 368769
# timestamp: 1761210943502 offset: 368801
# ...
bin/kafka-dump-log.sh --index-sanity-check \
--files /var/lib/kafka/data/orders-0/00000000000000368769.index
Reading an offset index directly with Python shows how simple the format is:
import struct
import sys
def read_offset_index(path: str, base_offset: int):
with open(path, "rb") as f:
data = f.read()
entries = []
for rel, pos in struct.iter_unpack(">ii", data[: len(data) // 8 * 8]): # big-endian int32 pairs
if entries and rel == 0 and pos == 0: # preallocated, unused space of an active index
break
entries.append((base_offset + rel, pos))
return entries
path = sys.argv[1] # e.g. .../00000000000000368769.index
base = int(path.rsplit("/", 1)[-1].split(".")[0]) # base offset is the file name
for offset, position in read_offset_index(path, base)[:5]:
print(f"offset {offset:>10} -> byte {position}")
Key Takeaways
.indexmaps relative offset to byte position (8 bytes per entry)..timeindexmaps maximum timestamp so far to relative offset (12 bytes per entry).- Both are sparse, with an entry about every 4 KB of log, and memory-mapped. Lookups are binary searches followed by a short scan.
- The time index is monotonic by design, so time lookups return the earliest offset at or after the target even with out-of-order timestamps.
- Indexes can always be rebuilt from the
.log, which happens automatically after a crash or corruption. offsetsForTimes,kafka-get-offsets.sh --time, andkafka-consumer-groups.sh --reset-offsets --to-datetimeall depend on the time index.
Practice
- (Beginner) Run
kafka-dump-log.shon a closed segment's.indexand.log. Pick an index entry and confirm that the batch at that byte position in the log starts at the indexed offset.- (Intermediate) A producer has a misconfigured clock and sends records stamped one day in the future into a segment, then the clock is fixed. Describe what the
.timeindexof that segment looks like afterwards, and whatoffsetsForTimesreturns for a target time of "one hour ago".- Interview question: "Why is Kafka's offset index sparse, and how would you pick
index.interval.bytes?" Hint: compare the memory cost of a dense index with the cost of scanning a few KB, and think about which reads actually use the index.
Record Batches Format
Theory
The .log file stores record batches exactly as the producer built them (Section 4.4). This one decision explains a lot of Kafka's design: the same bytes travel from producer to broker, to disk, to followers, and to consumers, so the broker never has to re-encode data on the read path. That is what makes zero-copy transfer possible (next topic).
The current format is magic v2, introduced in Kafka 0.11. Earlier formats (v0, v1) wrapped every message in its own header and are no longer supported for writes in Kafka 4.0. v2 has two main ideas:
- Put shared metadata in the batch header once: base offset, producer ID, epoch, base sequence, timestamps, compression codec.
- Store per-record fields as small deltas using variable-length integers (varints), so a record's own overhead is often under 10 bytes.
Analogy: a shipping pallet with one label (sender, date, tracking number) and boxes marked only "item 1, item 2, item 3", instead of every box carrying a full shipping label.
Batch header (61 bytes)
RecordBatch
+------------------------+--------+------------------------------------------------------+
| baseOffset | int64 | offset of the first record (assigned by the leader) |
| batchLength | int32 | bytes after this field |
| partitionLeaderEpoch | int32 | leader epoch at append time (set by the broker) |
| magic | int8 | 2 |
| crc | uint32 | CRC-32C of everything from attributes to the end |
+------------------------+--------+---------------------- CRC-covered from here ---------+
| attributes | int16 | compression, timestamp type, txn, control flags |
| lastOffsetDelta | int32 | lastOffset - baseOffset |
| baseTimestamp | int64 | timestamp of the first record |
| maxTimestamp | int64 | largest timestamp in the batch |
| producerId | int64 | idempotent / transactional producer ID, or -1 |
| producerEpoch | int16 | |
| baseSequence | int32 | sequence number of the first record, or -1 |
| recordsCount | int32 | number of records |
+------------------------+--------+------------------------------------------------------+
| records[] | | compressed as one block if a codec is set |
+------------------------+--------+------------------------------------------------------+
The fields before the CRC (baseOffset, batchLength, partitionLeaderEpoch, magic) are deliberately excluded from it. The leader overwrites baseOffset and partitionLeaderEpoch when it appends the batch, and excluding them means it can do so without recomputing the checksum or touching the (possibly compressed) records.
Attributes bits
| Bits | Meaning |
|---|---|
| 0-2 | Compression: 0 none, 1 gzip, 2 snappy, 3 lz4, 4 zstd |
| 3 | Timestamp type: 0 CreateTime, 1 LogAppendTime |
| 4 | Transactional batch |
| 5 | Control batch (transaction commit/abort marker, Chapter 8) |
| 6 | Has delete horizon (set by the log cleaner on tombstones, see Tombstone Records) |
Record layout
Record
+-------------------+---------+------------------------------------------+
| length | varint | size of the rest of the record |
| attributes | int8 | unused, always 0 |
| timestampDelta | varlong | timestamp - baseTimestamp |
| offsetDelta | varint | offset - baseOffset |
| keyLength | varint | -1 for a null key |
| key | bytes | |
| valueLength | varint | -1 for a null value (tombstone) |
| value | bytes | |
| headersCount | varint | |
| headers[] | | each: keyLength varint, key (UTF-8), |
| | | valueLength varint, value |
+-------------------+---------+------------------------------------------+
Varints use zigzag encoding: small numbers, positive or negative, take one byte (0 -> 00, -1 -> 01, 1 -> 02, ...). Values up to 63 fit in one byte, and up to 8,191 in two.
What the broker does with a batch
On the leader, before appending, the broker:
- Checks the CRC and the magic byte, and validates the records (for example, that every record on a compacted topic has a key). For compressed batches this means decompressing to iterate over them.
- Assigns offsets by writing
baseOffset. Records already carry theiroffsetDelta, so nothing inside them changes. - Sets
maxTimestampto the current time if the topic usesmessage.timestamp.type=LogAppendTime. - Recompresses only if the topic's
compression.typenames a different codec than the batch uses. The default,producer, keeps the producer's codec and avoids this cost.
Followers and consumers then receive the stored bytes unchanged.
Example: encoding a record and parsing batch headers
import struct
def zigzag_varint(n: int) -> bytes:
n = (n << 1) ^ (n >> 63) # zigzag: map signed to unsigned so small negatives stay small
out = bytearray()
while True:
byte, n = n & 0x7F, n >> 7
out.append(byte | (0x80 if n else 0)) # high bit set = more bytes follow
if not n:
return bytes(out)
def encode_record(key: bytes | None, value: bytes | None, ts_delta=0, offset_delta=0) -> bytes:
body = b"\x00" # attributes
body += zigzag_varint(ts_delta) + zigzag_varint(offset_delta)
body += zigzag_varint(-1) if key is None else zigzag_varint(len(key)) + key
body += zigzag_varint(-1) if value is None else zigzag_varint(len(value)) + value
body += zigzag_varint(0) # no headers
return zigzag_varint(len(body)) + body
rec = encode_record(b"CUST-77", b'{"orderId":"ORD-1042","total":39.98}')
print(len(rec), "bytes for a 7-byte key and 36-byte value") # 50 -> only 7 bytes of overhead
import struct
import sys
HEADER = struct.Struct(">qiibIhiqqqhii") # the 61-byte v2 batch header, big-endian
CODECS = {0: "none", 1: "gzip", 2: "snappy", 3: "lz4", 4: "zstd"}
def iter_batches(path: str):
with open(path, "rb") as f:
while True:
raw = f.read(HEADER.size)
if len(raw) < HEADER.size: # end of file (or a partial batch after a crash)
return
(base_offset, batch_len, leader_epoch, magic, crc, attrs, last_delta,
base_ts, max_ts, pid, epoch, base_seq, count) = HEADER.unpack(raw)
yield {
"offsets": (base_offset, base_offset + last_delta),
"records": count,
"codec": CODECS[attrs & 0x07],
"transactional": bool(attrs & 0x10),
"control": bool(attrs & 0x20),
"producer_id": pid,
"bytes": 12 + batch_len, # batchLength excludes baseOffset and itself
}
f.seek(batch_len - (HEADER.size - 12), 1) # skip the records section
for batch in iter_batches(sys.argv[1]):
print(batch)
Key Takeaways
- Kafka stores, replicates, and serves record batches in the producer's format, so the read path never re-encodes data.
- The v2 batch header (61 bytes) holds shared metadata once. Records store varint deltas, often with less than 10 bytes of overhead each.
baseOffsetandpartitionLeaderEpochare outside the CRC so the leader can set them without touching the records.- Compression applies to the records section of a batch as one block. Larger batches compress better (Section 4.4).
- Keep the topic
compression.type=producerunless you have a reason to change it, to avoid recompression on the broker.
Practice
- (Beginner) Using the record layout, compute the encoded size of a record with a null key, a 100-byte value, one header
trace-idwith a 16-byte value,timestampDelta=5, andoffsetDelta=3.- (Intermediate) Run the batch parser above on a segment from a topic written by an idempotent producer with
linger.ms=0, then again withlinger.ms=20. Compare records per batch and bytes per record, and explain the difference.- Interview question: "Why did Kafka move from per-message headers (v0/v1) to record batches (v2)?" Hint: think about per-record overhead, where producer ID and sequence numbers must live for idempotence, and how compression worked with wrapper messages.
Zero-Copy Transfer
Theory
A busy broker sends far more data than it receives: each record is written once but read by followers and by every consumer group. The read path has to be cheap, and the cheapest work is work the broker never does. For a fetch, the broker does not need to look inside the data at all: it already has the exact bytes on disk in the format the client expects. It only has to move them from a file to a socket.
A naive server would do that by reading the file into its own memory and then writing it to the socket. That copies every byte through user space twice and costs four context switches per chunk:
Traditional read() + write()
disk --DMA--> [page cache] --CPU copy--> [JVM buffer] --CPU copy--> [socket buffer] --DMA--> NIC
kernel space user space kernel space
4 copies (2 by CPU), 4 user/kernel context switches, data passes through the JVM heap
Zero-copy sendfile() / FileChannel.transferTo()
disk --DMA--> [page cache] ------------- descriptors only -------------------------> NIC
kernel space (with scatter-gather DMA, the NIC reads directly
from page cache pages)
2 copies (both DMA, 0 by CPU), 2 context switches, data never enters the JVM
Kafka uses Java's FileChannel.transferTo(), which on Linux becomes the sendfile(2) system call. When a consumer or follower fetches, the broker sends a response header it built itself, then hands the kernel a file region ("bytes 81,920 to 1,130,496 of this segment") and lets the kernel move the data.
Analogy: instead of a courier carrying a parcel from the warehouse to the office and then back out to the delivery truck, the warehouse loads it straight onto the truck. The office only signs the paperwork.
Why it works for Kafka specifically
Zero-copy is only possible because of choices covered earlier in this chapter:
| Design choice | Why it enables zero-copy |
|---|---|
| Batches are stored in the wire format | No re-encoding is needed between disk and network |
| Compression is end to end | The broker does not decompress on the read path |
| Offsets are assigned in the batch header on append | Nothing must be rewritten per reader |
| Logs are immutable files | A file region can be handed to the kernel safely |
When zero-copy does not apply
| Situation | Why |
|---|---|
TLS listeners (SSL, SASL_SSL) | Data must be encrypted in user space by the JVM before it is sent, so it is read into memory first |
| Message format down-conversion (old clients, before Kafka 4.0) | The broker had to rewrite batches into an older format for each fetch |
| Reads from tiered storage (Section 7.3) | Data arrives from the remote store as a stream, not as a local file |
TLS is the most common case in production. Expect noticeably higher broker CPU usage on encrypted listeners, and plan capacity with it in mind (Chapter 13).
Example: the same mechanism outside Kafka
A minimal Java file server using transferTo, the call Kafka's FileRecords.writeTo relies on:
import java.net.InetSocketAddress;
import java.nio.channels.FileChannel;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
public class ZeroCopyServer {
public static void main(String[] args) throws Exception {
Path segment = Path.of(args[0]);
try (ServerSocketChannel server = ServerSocketChannel.open()) {
server.bind(new InetSocketAddress(9999));
while (true) {
try (SocketChannel client = server.accept();
FileChannel file = FileChannel.open(segment, StandardOpenOption.READ)) {
long position = 0, size = file.size();
while (position < size) {
// The kernel moves bytes from page cache to the socket; no byte[] is allocated
position += file.transferTo(position, size - position, client);
}
}
}
}
}
}
The Python equivalent uses os.sendfile directly:
import os
import socket
def serve_file(path: str, port: int = 9999) -> None:
with socket.create_server(("0.0.0.0", port)) as server:
while True:
conn, _ = server.accept()
with conn, open(path, "rb") as f:
size = os.fstat(f.fileno()).st_size
offset = 0
while offset < size:
# sendfile(out_fd, in_fd, offset, count): kernel-to-kernel copy
offset += os.sendfile(conn.fileno(), f.fileno(), offset, size - offset)
To see the effect, serve a 1 GB file with both this program and a version using f.read() + conn.sendall(), and compare CPU time with /usr/bin/time -v. The sendfile version uses a fraction of the user CPU time.
Key Takeaways
- Zero-copy (
sendfile,FileChannel.transferTo) moves data from page cache to the socket without copying it through the JVM. - It saves two CPU copies and two context switches per transfer, and keeps fetched data out of the Java heap, which reduces garbage collection pressure.
- It works because Kafka stores batches in the wire format and never re-encodes them on the read path.
- TLS disables it, because encryption happens in user space. Older down-conversion and tiered storage reads also bypass it.
Practice
- (Beginner) Draw the copy path of a fetch for a consumer on a
PLAINTEXTlistener and for one on anSSLlistener. Mark where the data enters user space in the second case.- (Intermediate) A cluster's broker CPU rose from 25% to 60% after enabling TLS, with the same traffic. Explain why, and list two ways to restore capacity without disabling encryption.
- Interview question: "What is zero-copy, and why is it effective for Kafka but less useful for a typical REST API server?" Hint: compare what each server does to the bytes between storage and the network.
Page Cache Usage
Theory
Most databases manage their own cache of disk blocks in process memory. Kafka deliberately does not. It writes to and reads from files through normal system calls and lets the operating system's page cache (free RAM that the kernel uses to cache file contents) do the caching.
Why give up control?
| Reason | Explanation |
|---|---|
| No double caching | An in-process cache would duplicate data already cached by the kernel, wasting half the memory |
| No garbage collection cost | Gigabytes of cached data on the Java heap would cause long GC pauses. The page cache lives outside the heap |
| Survives broker restarts | After a process restart, the kernel cache is still warm. A heap cache would start empty |
| Fits the access pattern | Producers write at the tail and most consumers read near the tail, which is exactly what an LRU-like cache keeps |
| Free read-ahead | The kernel detects sequential reads and prefetches upcoming pages |
Analogy: a librarian who keeps the newest magazines on the front desk. Most visitors ask for this week's issue and get it immediately. Someone asking for an issue from three years ago has to wait while it is fetched from the archive in the basement.
Write path: durability without fsync
When a producer's batch is appended, the broker calls write(), which copies the bytes into the page cache and returns. The pages are marked dirty and the kernel writes them to disk in the background. Kafka does not call fsync per batch by default:
| Setting | Default | Meaning |
|---|---|---|
log.flush.interval.messages / flush.messages | Long.MAX_VALUE | Force a flush after this many messages |
log.flush.interval.ms / flush.ms | not set (effectively never) | Force a flush after this much time |
Durability comes from replication instead: with acks=all and min.insync.replicas=2 (Section 4.3), a write is acknowledged only once it is in the page cache of at least two brokers. Losing the data requires those brokers to lose power at the same time, before the kernel flushed it. If one broker crashes, the recovery process from Log Segments truncates its unflushed tail and refetches it from the leader. Forcing fsync per message trades a large part of throughput for protection against correlated power loss, which is better addressed by spreading replicas across racks or zones (Section 2.2).
Read path: hot tail, cold history
page cache (RAM)
+----------------------------------+
offsets 0 ....... 800k ........ 960k ........ 1,000k (log end)
[ on disk only ][ evicted ][ cached: recent writes ]
^ ^
| |
lagging consumer reads here tail consumers and followers
-> real disk I/O read here -> served from RAM
-> pulls cold pages INTO the cache -> no disk reads at all
-> can EVICT the hot tail
A healthy broker serves almost all reads from memory, and iostat shows nearly only writes. The classic failure mode is a consumer that falls far behind or replays from the beginning: its reads force disk I/O and fill the cache with old pages, evicting pages that tail consumers need. Those consumers then hit disk too, and if producers share the same disks, produce latency rises. One slow consumer can degrade the whole broker. Tiered storage (Section 7.3) and quotas (Chapter 12) are the usual mitigations.
Sizing memory
Give the JVM a modest heap (commonly 4 to 8 GB) and leave the rest of the RAM to the page cache. A useful target is to cache the amount of data your consumers might need to reread when they are briefly behind:
cache needed ~= write throughput into this broker (including replica traffic) x seconds of lag to tolerate
~= 60 MB/s x 600 s = 36 GB
Linux settings that matter
| Setting | Typical value | Why |
|---|---|---|
vm.swappiness | 1 | Prefer dropping page cache over swapping the JVM out |
vm.dirty_background_ratio | 5 | Start background writeback earlier, in smaller bursts |
vm.dirty_ratio | 60 to 80 | Allow many dirty pages before writes block |
| Filesystem | XFS or ext4, mounted noatime | No metadata write on every read |
| Transparent huge pages | madvise or never | Avoid latency spikes from page compaction |
Example: observing the page cache
# /etc/sysctl.d/90-kafka.conf
vm.swappiness = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 60
vm.max_map_count = 262144
# Apply with: sudo sysctl --system
# How much RAM is page cache ("buff/cache") versus used by processes
free -h
# Which part of a partition's segments is resident in memory (vmtouch is a small utility)
vmtouch -v /var/lib/kafka/data/orders-0/*.log
# Disk reads per second: should be near zero on a broker with only tail consumers
iostat -x 5
# Page cache hit ratio over time (bcc-tools)
sudo cachestat 5
A Python model shows why one replaying consumer hurts everyone. It uses a simple LRU cache of pages (the real Linux cache is more elaborate, with active and inactive lists, but the effect is the same):
from collections import OrderedDict
class PageCache:
def __init__(self, capacity_pages: int):
self.pages, self.capacity = OrderedDict(), capacity_pages
def read(self, page: int) -> bool:
if page in self.pages:
self.pages.move_to_end(page)
return True # hit: served from RAM
self.pages[page] = True # miss: disk read, page now cached
if len(self.pages) > self.capacity:
self.pages.popitem(last=False) # evict least recently used page
return False
def tail_hit_ratio(with_replay: bool, lag: int = 400, steps: int = 10_000) -> float:
cache, hits = PageCache(capacity_pages=1_000), 0
for page in range(steps):
cache.read(page) # producer write lands in the cache
if page >= lag:
hits += cache.read(page - lag) # tail consumer, 400 pages behind the producer
if with_replay:
cache.read(1_000_000 + page) # replaying consumer: old pages never read before
return hits / (steps - lag)
print(f"tail consumer alone: {tail_hit_ratio(False):.0%} hits") # 100% hits
print(f"tail consumer with replay: {tail_hit_ratio(True):.0%} hits") # 2% hits
The tail consumer did not change at all. The replaying consumer brought in a stream of cold pages that pushed the tail out of the cache before the tail consumer reached it.
Key Takeaways
- Kafka relies on the OS page cache instead of an in-process cache: no double caching, no GC pressure, and the cache stays warm across broker restarts.
- Writes return once data is in the page cache. Durability comes from replication with
acks=allandmin.insync.replicas, not from per-writefsync. - Tail reads are served from memory. Lagging or replaying consumers cause disk reads and can evict the hot tail, hurting other clients.
- Keep the JVM heap modest, leave most RAM to the page cache, set
vm.swappiness=1, and watch disk read rates as an early warning.
Practice
- (Beginner) On a test broker, run a consumer at the tail while watching
iostat -x 5. Then start a second consumer group from--from-beginningon a large topic. Describe how disk reads change.- (Intermediate) A broker has 64 GB of RAM and a 32 GB heap "to be safe". It receives 40 MB/s including replication, and consumers are sometimes 10 minutes behind. Evaluate the configuration and propose a better memory split, with calculations.
- Interview question: "Kafka does not fsync each write by default. How can it still claim durability?" Hint: discuss what
acks=allactually waits for, what failure would lose data, and how log recovery handles a single broker crash.
<a id="72-retention-and-cleanup"></a>
7.2 Retention and Cleanup
A queue deletes a message once it has been consumed. Kafka keeps data based on a policy, regardless of who has read it. This section covers the policies that decide when data leaves a partition: deleting old segments by age or size, compacting a log down to the latest value per key, and the tombstones that delete keys from a compacted log.
Time-Based Retention
Theory
Keeping data after it has been consumed is what lets new consumer groups join late, lets a bug fix be followed by a replay, and lets several applications read the same events at their own pace. But disks are finite, so most topics need a limit. The most common one is age: "keep seven days of orders".
Analogy: a security camera that keeps the last 30 days of footage. Nobody needs to have watched a recording for it to be kept, and every recording older than 30 days is overwritten, whether anyone watched it or not.
Configuration
| Broker setting | Topic setting | Default |
|---|---|---|
log.retention.ms (highest priority), log.retention.minutes, log.retention.hours | retention.ms | 168 hours (7 days) |
log.retention.check.interval.ms | (broker only) | 300,000 (5 minutes) |
file.delete.delay.ms | file.delete.delay.ms | 60,000 (1 minute) |
Set retention.ms=-1 for unlimited retention.
How it is enforced
Retention never deletes individual records. Every log.retention.check.interval.ms, a background task looks at each partition's closed segments from oldest to newest and deletes a segment when:
now - largestTimestamp(segment) > retention.ms
largestTimestamp is the largest record timestamp in the segment (read from the last .timeindex entry). It stops at the first segment that is too young, so retention removes a contiguous prefix of the log. The active segment is never deleted by time retention.
Deletion happens in two phases: the segment's files are renamed with a .deleted suffix and removed from the in-memory segment list, and they are physically deleted file.delete.delay.ms later, so readers still holding the file can finish. The partition's log start offset then moves to the base offset of the oldest remaining segment.
retention.ms = 7 days, now = Day 10 00:00, segment.ms = 1 day
segment largest timestamp age of newest record action
00000000 Day 1 00:00 9 days delete
00041000 Day 2 00:00 8 days delete
00082000 Day 3 00:00 exactly 7 days keep (not older than retention.ms)
...
00369000 (active) - never deleted
log start offset: 0 -> 82000
Consequences worth knowing
| Effect | Cause | What to do |
|---|---|---|
Data lives longer than retention.ms | A segment is deleted only when its newest record expires, and the active segment must roll first | Lower segment.ms on low-traffic topics with short retention |
| Data deleted earlier than expected, or never | Retention uses record timestamps. Producers with wrong clocks write timestamps in the past or the future | Use message.timestamp.type=LogAppendTime, or reject outliers with message.timestamp.before.max.ms / message.timestamp.after.max.ms (Kafka 3.6+) |
Consumer gets OffsetOutOfRange | Its committed offset is below the new log start offset | auto.offset.reset decides where it restarts (Section 5.3) |
| Disk usage drops in steps, not smoothly | Whole segments are deleted | Expected. Smaller segments make it smoother |
Example: setting and observing retention
# Keep 3 days on an existing topic; roll segments daily so retention is precise to about a day
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
--entity-type topics --entity-name orders \
--add-config retention.ms=259200000,segment.ms=86400000
# Check the effective config (shows defaults and their source)
bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --all \
--entity-type topics --entity-name orders | grep -E 'retention.ms|segment.ms'
# Earliest offset still available per partition: rises after retention runs
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders --time -2
A Python model of the retention pass:
from dataclasses import dataclass
DAY = 24 * 3600 * 1000
@dataclass
class Segment:
base_offset: int
largest_ts: int
def time_retention(segments: list[Segment], now: int, retention_ms: int) -> tuple[list[Segment], int]:
"""Return (segments to delete, new log start offset). The last segment is the active one."""
closed, deletable = segments[:-1], []
for seg in closed: # oldest first
if now - seg.largest_ts > retention_ms:
deletable.append(seg)
else:
break # stop at the first segment that is too young
remaining = segments[len(deletable):]
return deletable, remaining[0].base_offset
segments = [Segment(0, 1 * DAY), Segment(41_000, 2 * DAY), Segment(82_000, 3 * DAY),
Segment(123_000, 9 * DAY)] # last one is active
gone, log_start = time_retention(segments, now=10 * DAY, retention_ms=7 * DAY)
print([s.base_offset for s in gone], log_start) # [0, 41000] 82000
Key Takeaways
- Time retention deletes whole closed segments once their largest record timestamp is older than
retention.ms. Records are never deleted individually. - The check runs every 5 minutes by default, and files are physically removed a minute after being marked
.deleted. - Real retention is between
retention.msand roughlyretention.ms + segment.ms(plus the check interval). - Retention depends on record timestamps, so wrong producer clocks affect it.
LogAppendTimeor timestamp bounds protect against that. - Deleting segments advances the log start offset. Consumers committed below it fall back to
auto.offset.reset.
Practice
- (Beginner) Create a topic with
retention.ms=60000andsegment.ms=10000. Produce records for one minute, stop, and pollkafka-get-offsets.sh --time -2every 30 seconds. Explain when and why the earliest offset changes.- (Intermediate) A producer with a broken clock writes one record dated the year 2030 into an otherwise normal segment of a topic with 7-day retention. What happens to that segment and to all later segments? How would you detect and fix it?
- Interview question: "A user set
retention.msto one hour but can still read records from yesterday. What is happening?" Hint: think about which segment can never be deleted and what determines when it rolls.
Size-Based Retention
Theory
Age-based limits do not protect disks from traffic spikes: a producer that suddenly writes ten times its usual volume fills disks ten times faster, and seven days of that may not fit. Size-based retention caps how much data is kept regardless of age.
Analogy: a DVR with a fixed-size hard drive. When it is full, the oldest recordings are erased to make room, however recent they are.
Configuration and the per-partition trap
| Broker setting | Topic setting | Default |
|---|---|---|
log.retention.bytes | retention.bytes | -1 (no limit) |
The limit applies per partition, not per topic and not per broker. A topic with retention.bytes=10 GiB, 12 partitions, and replication factor 3 can use up to 12 x 10 x 3 = 360 GiB across the cluster.
How it is enforced
In the same periodic task as time retention, the broker computes how far the partition is over the limit and deletes the oldest closed segments while removing a segment still leaves the partition at or above the limit:
retention.bytes = 3 GiB, segment.bytes = 1 GiB
segments (oldest -> newest): [1 GiB] [1 GiB] [1 GiB] [1 GiB] [0.4 GiB active] total 4.4 GiB
over the limit by: 4.4 - 3 = 1.4 GiB
delete segment 1 (1 GiB)? 1.4 - 1.0 = 0.4 >= 0 -> yes, still over by 0.4
delete segment 2 (1 GiB)? 0.4 - 1.0 < 0 -> no, would drop below the limit
result: 3.4 GiB kept
The partition can exceed retention.bytes by up to one segment, and the active segment is never deleted. So a useful rule is:
max partition size ~= retention.bytes + segment.bytes
Combining time and size
When both are set, a segment is deleted when either condition says so. Time is the business rule ("keep seven days"). Size is a safety net ("but never more than 500 GiB per partition"). If size retention fires first during a traffic spike, you keep less history than the time rule promises, and consumers that are behind may lose data. Alert when that happens.
Disk sizing
disk per broker ~= (sum over partitions hosted of (retention.bytes + segment.bytes)) + headroom (30 to 40%)
Example: 2,000 partition replicas on a broker, retention.bytes = 20 GiB, segment.bytes = 1 GiB
2,000 x 21 GiB = 42,000 GiB ~= 41 TiB (+ headroom) -> far beyond a typical broker
-> lower retention.bytes, use fewer partitions, or move history to tiered storage (7.3)
Example: size-based retention in practice
# Cap each partition of "clickstream" at 50 GiB as a safety net alongside 7-day retention
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
--entity-type topics --entity-name clickstream \
--add-config retention.bytes=53687091200
# Current size of each partition replica, to compare against the limit
bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --topic-list clickstream \
| grep '^{' \
| python3 -c '
import json, sys
for broker in json.load(sys.stdin)["brokers"]:
for d in broker["logDirs"]:
for p in d["partitions"]:
print(broker["broker"], p["partition"], round(p["size"] / 2**30, 2), "GiB")'
def size_retention(segment_sizes: list[int], retention_bytes: int) -> int:
"""Return how many of the oldest segments to delete. The last entry is the active segment."""
if retention_bytes < 0:
return 0
diff = sum(segment_sizes) - retention_bytes # how far over the limit
deleted = 0
for size in segment_sizes[:-1]: # never the active segment
if diff - size < 0:
break # deleting this one would go below the limit
diff -= size
deleted += 1
return deleted
GiB = 2**30
print(size_retention([GiB, GiB, GiB, GiB, int(0.4 * GiB)], retention_bytes=3 * GiB)) # 1
Key Takeaways
retention.byteslimits each partition, so multiply by partition count and replication factor to get total usage.- The oldest closed segments are deleted while the partition stays at or above the limit. The real maximum is about
retention.bytes + segment.bytes. - With both limits set, whichever triggers first wins. Use size as a safety net and time as the business rule.
- Size retention during a spike can delete data that lagging consumers have not read yet. Monitor log start offsets and consumer lag together.
Practice
- (Beginner) A topic has 6 partitions, replication factor 3,
retention.bytes=5 GiB, andsegment.bytes=512 MiB. What is the maximum disk space it can use across the cluster?- (Intermediate) Write a script that, for each partition of a topic, compares the size reported by
kafka-log-dirs.shwithretention.bytesand prints partitions above 90% of the limit. Explain why some partitions may show more than 100%.- Interview question: "How would you set retention for a topic that must keep 14 days of data but must never fill the broker disks?" Hint: combine both settings, size the disks from the expected peak rate, and decide what alert fires when size retention starts deleting data younger than 14 days.
Log Compaction
Theory
Some topics do not hold a history of events but the current state of entities: the latest address of each customer, the latest configuration of each device, the latest offset committed by each consumer group (__consumer_offsets, Section 5.3). For these, old values of a key are useless once a newer one exists, but the newest value must be kept forever, however old it is. Time and size retention cannot express that. Log compaction can: it removes records that have been superseded by a later record with the same key.
Analogy: a phone's contact list synced from a log of changes. You do not need every number Ada ever had, only her current one. But you must never forget Ada just because her number has not changed in five years.
Before compaction After compaction
offset: 0 1 2 3 4 5 6 7 offset: 2 5 6 7
key: K1 K2 K1 K3 K2 K2 K3 K4 key: K1 K2 K3 K4
value: a b c d e f g h value: c f g h
- Offsets never change; compaction leaves gaps (consumers just skip them).
- Order is preserved: surviving records stay in their original order.
- The latest value of every key survives.
Head and tail
The cleaner divides a compacted partition at the cleaner point (stored in cleaner-offset-checkpoint):
cleaner point
|
[ tail: already compacted ][ head: dirty, may contain duplicates ][ active segment ]
each key appears once written since the last cleaning never cleaned
A partition is eligible for cleaning when its dirty ratio (head bytes / (head + tail bytes)) exceeds min.cleanable.dirty.ratio, or when a record has waited longer than max.compaction.lag.ms.
How the cleaner works
- Pick the partition with the highest dirty ratio.
- Scan the head and build an offset map: key hash to the latest offset of that key. Each entry costs 24 bytes (16-byte MD5 hash plus an 8-byte offset).
- Recopy segments from the start of the log into new
.cleanedfiles, keeping a record only if its offset is the latest one for its key in the map (or the key is not in the map). - Swap the cleaned files in (
.cleaned->.swap-> final name), merging small segments together, and advance the cleaner point.
The offset map's size limits how many distinct keys one pass can handle: the default 128 MiB buffer at a 0.9 load factor holds about 5 million keys, shared across all cleaner threads. If the head has more distinct keys, the cleaner compacts only part of it per pass.
Settings
| Setting | Default | Purpose |
|---|---|---|
cleanup.policy | delete | Set to compact to enable compaction |
min.cleanable.dirty.ratio | 0.5 | Clean when at least half the log is dirty. Lower means more frequent cleaning, more I/O |
min.compaction.lag.ms | 0 | A record stays uncompacted for at least this long, so consumers can see every change for a while |
max.compaction.lag.ms | Long.MAX_VALUE | Force cleaning once a record has been dirty this long (for example, for GDPR deletion deadlines) |
segment.ms, segment.bytes | 7 days, 1 GiB | The active segment is never compacted, so these bound how long duplicates linger |
log.cleaner.threads | 1 | Cleaner threads per broker |
log.cleaner.dedupe.buffer.size | 128 MiB | Total offset map memory across cleaner threads |
log.cleaner.io.max.bytes.per.second | unlimited | Throttle cleaner I/O |
Guarantees
- A consumer that stays within the head (keeps up with the log) sees every record written.
- A consumer reading from the beginning sees at least the latest value of each key, in offset order.
- Every record on a compacted topic must have a non-null key. The broker rejects keyless records.
- Compaction is not a deduplication guarantee: until the cleaner runs, duplicates are present.
Example: a compacted "customer profiles" topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic customer-profiles \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.1 \
--config segment.ms=3600000 \
--config min.compaction.lag.ms=600000
# roll hourly so data reaches the cleaner; keep every change for 10 minutes
# Cleaner health: these MBeans are worth alerting on
# kafka.log:type=LogCleanerManager,name=max-dirty-percent
# kafka.log:type=LogCleanerManager,name=uncleanable-partitions-count (> 0 means a stuck partition)
# kafka.log:type=LogCleaner,name=time-since-last-run-ms
// Rebuilding the current state of every customer from a compacted topic
Map<String, String> profiles = new HashMap<>();
consumer.assign(partitions);
consumer.seekToBeginning(partitions);
Map<TopicPartition, Long> end = consumer.endOffsets(partitions);
while (partitions.stream().anyMatch(tp -> consumer.position(tp) < end.get(tp))) {
for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
if (r.value() == null) profiles.remove(r.key()); // tombstone: the key was deleted
else profiles.put(r.key(), r.value()); // later records overwrite earlier ones
}
}
A Python model of a cleaning pass:
def compact(log: list[tuple[int, str, str | None]], dirty_from: int) -> list[tuple[int, str, str | None]]:
"""log: (offset, key, value) in offset order. dirty_from: the cleaner point."""
offset_map = {}
for offset, key, _ in log:
if offset >= dirty_from:
offset_map[key] = offset # latest offset per key in the head
kept = []
for offset, key, value in log:
latest = offset_map.get(key)
if latest is None or offset >= latest: # not superseded by a newer record
kept.append((offset, key, value))
return kept
log = [(0, "K1", "a"), (1, "K2", "b"), (2, "K1", "c"), (3, "K3", "d"),
(4, "K2", "e"), (5, "K2", "f"), (6, "K3", "g"), (7, "K4", "h")]
print(compact(log, dirty_from=0))
# [(2, 'K1', 'c'), (5, 'K2', 'f'), (6, 'K3', 'g'), (7, 'K4', 'h')]
Key Takeaways
- Compaction keeps at least the latest record for every key and removes older ones. It turns a topic into a durable, replayable table.
- Offsets and order are preserved. Compaction only removes records, leaving gaps in offsets.
- The cleaner works on closed segments past the cleaner point, triggered by
min.cleanable.dirty.ratioormax.compaction.lag.ms. The active segment is never compacted. - The offset map (24 bytes per key) limits how many keys one pass can process. Watch
max-dirty-percentanduncleanable-partitions-count. - Records on compacted topics need keys, and compaction does not guarantee there are no duplicates at any given moment.
Practice
- (Beginner) Create a compacted topic with
segment.ms=10000andmin.cleanable.dirty.ratio=0.01. Produce 1,000 records spread over 10 keys, wait a minute, and consume from the beginning. How many records do you get, and why might it be more than 10?- (Intermediate) A compacted topic has 50 million distinct keys per partition, and
max-dirty-percentkeeps rising. Explain the likely cause with the offset map numbers above, and propose two fixes.- Interview question: "When would you use a compacted topic instead of a database table, and what guarantees does compaction give a consumer that replays from the start?" Hint: think about event-carried state, Kafka Streams changelogs, and the difference between "latest value" and "every value".
Tombstone Records
Theory
Compaction keeps the latest record for every key, forever. So how do you delete a key, for example a customer who closes their account or exercises a right to erasure? You write a record with that key and a null value. This is a tombstone: it supersedes all earlier values of the key and tells consumers "this key no longer exists".
If tombstones were kept forever too, a topic with high key churn would fill up with markers for keys that no longer exist. If they were removed as soon as the cleaner saw them, a consumer that had read the old value but not yet the tombstone would never learn about the deletion. Kafka resolves this by keeping tombstones for a grace period.
Analogy: a "moved away, no forwarding address" notice taped to a mailbox. It stays up long enough for regular visitors to see it, then it is taken down.
Tombstone lifecycle
offset: 10 25 40 (tombstone)
key: C-42 C-42 C-42
value: {...} {...} null
cleaner pass 1: offsets 10 and 25 removed (superseded); tombstone kept,
its batch gets a "delete horizon" = now + delete.retention.ms
...
cleaner pass N (after the delete horizon): tombstone removed; key C-42 is gone entirely
| Setting | Default | Meaning |
|---|---|---|
delete.retention.ms | 86,400,000 (24 hours) | How long a tombstone is kept after the cleaner first processes it |
The delete horizon is written into the batch header (attribute bit 6, see Record Batches Format), so it survives broker restarts and leader changes. The same horizon also controls when the cleaner removes transaction markers (Chapter 8).
The consumer contract
A consumer that rebuilds state from a compacted topic must finish reading from the beginning to the end within delete.retention.ms. Otherwise it can read an old value of a key, miss the tombstone that was removed meanwhile, and keep a deleted key in its state.
Where tombstones come from
| Producer | How it emits tombstones |
|---|---|
| Application code | ProducerRecord(topic, key, null) |
| Kafka Streams | Deleting from a KTable writes a tombstone to its changelog (Chapter 10) |
| Debezium / CDC connectors | A delete event followed by a tombstone, so compaction can remove the row (Chapter 9) |
| Console producer | --property null.marker=NULL makes the literal NULL a null value |
On a topic with cleanup.policy=delete, a tombstone is just an ordinary record with a null value. Only compaction gives it special meaning.
Example: producing and handling tombstones
// Delete customer C-42 from a compacted topic
producer.send(new ProducerRecord<String, String>("customer-profiles", "C-42", null));
from confluent_kafka import Consumer, Producer
producer = Producer({"bootstrap.servers": "localhost:9092"})
producer.produce("customer-profiles", key="C-42", value=None) # value=None -> tombstone
producer.flush()
consumer = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "profile-cache",
"auto.offset.reset": "earliest"})
consumer.subscribe(["customer-profiles"])
cache: dict[bytes, bytes] = {}
while True:
msg = consumer.poll(1.0)
if msg is None or msg.error():
continue
if msg.value() is None:
cache.pop(msg.key(), None) # tombstone: forget the key
else:
cache[msg.key()] = msg.value()
# Tombstones from the console producer
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic customer-profiles \
--property parse.key=true --property key.separator=: --property null.marker=NULL
# > C-42:NULL
# Tombstones show as "null" values in the console consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic customer-profiles \
--from-beginning --property print.key=true
Key Takeaways
- A tombstone is a record with a key and a null value. On a compacted topic, it deletes the key.
- Compaction removes older values immediately but keeps the tombstone for
delete.retention.ms(default 24 hours) after the cleaner first sees it. - Consumers rebuilding state from the beginning must reach the end within
delete.retention.ms, or deleted keys can reappear in their state. - Serializers and deserializers must keep
nullasnull(Section 6.1). A serializer that turnsnullinto"null"or{}breaks deletion silently. - On delete-policy topics, a null value has no special meaning.
Practice
- (Beginner) On a compacted topic with
delete.retention.ms=60000and short segments, write three values and then a tombstone for one key. Consume from the beginning right away and again after two minutes. Explain the difference.- (Intermediate) A cache service rebuilds itself from a compacted topic on startup, which takes 30 hours because of throttling. Users report that deleted accounts reappear after restarts. Explain the mechanism and give two fixes.
- Interview question: "How do you implement a right-to-be-forgotten request for a key stored in Kafka?" Hint: consider compacted versus delete-policy topics,
max.compaction.lag.ms,delete.retention.ms, and copies in other topics, backups, or tiered storage.
Cleanup Policies
Theory
The previous topics described two mechanisms: delete (drop whole old segments by time or size) and compact (drop superseded records by key). cleanup.policy chooses which of them runs on a topic. The choice follows from what the topic represents: a history of events, the current state of entities, or recent state.
Analogy: a bank keeps a statement archive (delete: everything for seven years, then shred) and a list of current balances (compact: one line per account, updated forever). Some data, like this week's balances, is useful only for a limited time (compact and delete).
The three policies
| Policy | What is removed | Use for | Examples |
|---|---|---|---|
delete (default) | Whole segments older than retention.ms or beyond retention.bytes | Event streams, logs, metrics | orders, clickstream, audit-log |
compact | Records superseded by a newer record with the same key, and expired tombstones | Current state per key, kept indefinitely | customer-profiles, __consumer_offsets, Kafka Connect config topics |
compact,delete | Both: superseded records, and also any segment past the retention limits | State that is only relevant for a limited time | Kafka Streams windowed changelogs, session caches |
Which policy?
Is each record a fact that happened (an event)?
yes -> delete
no, it is the latest state of a key
-> must old keys disappear on their own after some time?
no -> compact
yes -> compact,delete
Internal topics use these policies too
| Topic | Policy | Why |
|---|---|---|
__consumer_offsets | compact | Only the latest committed offset per group and partition matters |
__transaction_state | compact | Latest state per transactional ID (Chapter 8) |
Kafka Streams *-changelog | compact (or compact,delete for windows) | Restore state stores (Chapter 10) |
Connect connect-configs, connect-offsets, connect-status | compact | Latest configuration and source offsets (Chapter 9) |
Deleting records on demand
Separate from any policy, the DeleteRecords API moves a partition's log start offset forward, making everything before it unreadable. Whole segments below that offset are deleted. This is useful when a topic is used as a work queue, or to remove bad data that was already consumed:
cat > delete-records.json <<'EOF'
{ "version": 1,
"partitions": [ { "topic": "orders", "partition": 0, "offset": 500000 } ] }
EOF
bin/kafka-delete-records.sh --bootstrap-server localhost:9092 \
--offset-json-file delete-records.json
try (Admin admin = Admin.create(props)) {
admin.deleteRecords(Map.of(
new TopicPartition("orders", 0), RecordsToDelete.beforeOffset(500_000L) // -1 = up to the high watermark
)).all().get();
}
Changing a policy
cleanup.policy can be changed on a live topic. Switching from delete to compact makes the cleaner start compacting existing segments, and time and size retention stop. Switching from compact to delete makes old segments subject to retention, which may delete the latest value of keys that have not changed recently. Plan this carefully. Also note that compaction is not supported on topics with tiered storage enabled (Section 7.3).
Example: choosing policies for a small system
# Event history: 30 days
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic payments \
--partitions 12 --replication-factor 3 \
--config cleanup.policy=delete --config retention.ms=2592000000
# Current state per account, forever
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic account-balances \
--partitions 12 --replication-factor 3 \
--config cleanup.policy=compact
# Latest session per user, but forget sessions after 2 days of inactivity
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user-sessions \
--partitions 12 --replication-factor 3 \
--config cleanup.policy=compact,delete --config retention.ms=172800000
Key Takeaways
deleteis for event histories,compactfor current state per key, andcompact,deletefor state with a limited lifetime.- With
compact,delete, a key disappears once its latest record is in a segment past the retention limit, even without a tombstone. - Kafka's internal topics and most framework topics (offsets, transactions, Streams changelogs, Connect) are compacted.
- The DeleteRecords API advances the log start offset on demand, independent of the policy.
- Changing
compacttodeleteon a live topic can remove the latest state of keys that have not changed recently.
Practice
- (Beginner) For each of these topics choose a cleanup policy and justify it: website page views, current stock level per product, a feature flag per service, IoT readings kept for 90 days.
- (Intermediate) A team wants to change a 2 TB
deletetopic tocompactto save space. Describe what happens to existing data, the I/O load this causes, and what consumers must change in how they read the topic.- Interview question: "Why does Kafka Streams use
compact,deletefor windowed state changelogs instead of plaincompact?" Hint: think about what happens to keys of windows that have already closed if nothing ever writes a tombstone for them.
<a id="73-tiered-storage"></a>
7.3 Tiered Storage
Kafka's original design keeps every byte of a partition on the local disks of every replica. This section covers tiered storage (KIP-405, generally available since Kafka 3.9), which keeps recent data on brokers and moves older segments to cheaper remote storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage, while clients keep reading the topic as one continuous log.
Tiered Storage Architecture
Theory
Without tiering, retention is limited by local disk. Keeping 90 days of a busy topic means 90 days x replication factor of data on expensive block storage. Operations get slower too: adding a broker, replacing a failed one, or reassigning partitions means copying all of that history over the network before the new replica is in sync.
Most of that data is cold: consumers read the last few minutes, and the history is only for replays and late joiners. Tiered storage separates the two:
- Local tier: broker disks hold recent segments (hours to days). All writes and tail reads happen here, exactly as before.
- Remote tier: closed segments are uploaded to an object store, which is cheaper per GB, durable without Kafka replication, and effectively unlimited in size. Older data is read from there on demand.
Analogy: a restaurant kitchen keeps today's ingredients in the fridge next to the stove and the rest in a large warehouse across town. Cooks never notice the warehouse unless they ask for something unusual, and then it takes a little longer to arrive.
Components
Broker (partition leader)
+--------------------------------------------------------------------------------+
| ReplicaManager / UnifiedLog |
| local segments: [ ... ][ ... ][ active ] |
| | |
| RemoteLogManager (RLM) |
| - per leader partition: copy task (upload rolled segments) |
| - expiration task (apply retention to remote segments) |
| - remote reader thread pool (serve fetches below the local log) |
| | | |
| RemoteStorageManager (plugin) RemoteLogMetadataManager (plugin) |
| copy / fetch / delete segments which segments exist remotely, |
| and their indexes their offsets and state |
+------------|------------------------------------|------------------------------+
v v
Object store (S3, GCS, Azure) __remote_log_metadata topic (default)
| Component | Provided by | Responsibility |
|---|---|---|
RemoteLogManager | Kafka | Schedules uploads, remote retention, and remote reads for partitions this broker leads |
RemoteStorageManager (RSM) | Plugin | Moves segment and index bytes to and from the remote store |
RemoteLogMetadataManager (RLMM) | Kafka default (TopicBasedRemoteLogMetadataManager) or plugin | Records which segments are remote and their lifecycle state, in the __remote_log_metadata topic by default |
Segment upload lifecycle
Only the leader uploads. A segment is uploaded once it has been rolled and all its offsets are below the high watermark and the last stable offset, so only committed, non-pending data reaches the remote tier.
COPY_SEGMENT_STARTED -> COPY_SEGMENT_FINISHED -> DELETE_SEGMENT_STARTED -> DELETE_SEGMENT_FINISHED
(metadata event) upload complete, remote retention removed from
local copy may now expired the segment object store
be deleted
Each upload sends the .log file together with its offset index, time index, transaction index, producer state snapshot, and leader epoch checkpoint, so a broker can later serve or rebuild state for that range without the local files.
Enabling it
# server.properties (every broker)
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=<RSM plugin class> # see Remote Storage Manager
remote.log.storage.manager.class.path=/opt/kafka-tiered/* # plugin jars, isolated class loader
remote.log.metadata.manager.listener.name=PLAINTEXT # listener the RLMM's clients use
# rsm.config.* -> passed to the RSM plugin; rlmm.config.* -> passed to the RLMM
# Per topic: opt in and choose how much stays local
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic clickstream \
--partitions 12 --replication-factor 3 \
--config remote.storage.enable=true \
--config local.retention.ms=86400000 \
--config retention.ms=7776000000
# 1 day on broker disks, 90 days in total
Limitations to know
- Compacted topics are not supported. Tiering applies to
cleanup.policy=deleteonly. - Only Kafka's own storage layer is involved. Clients need no changes and no new configuration.
- Turning tiering off for a topic needs care: since Kafka 3.9 you can stop copying (
remote.log.copy.disable=true) or disable it and delete the remote data (remote.storage.enable=falsewithremote.log.delete.on.disable=true).
Key Takeaways
- Tiered storage keeps the active and recent segments on broker disks and uploads closed segments to an object store.
- The
RemoteLogManageron each leader handles uploads, remote retention, and remote reads. PluggableRemoteStorageManagerandRemoteLogMetadataManagercomponents talk to the store and track metadata. - Only committed, rolled segments are uploaded, together with their indexes and state snapshots.
- It is enabled per cluster (
remote.log.storage.system.enable) and per topic (remote.storage.enable). Clients are unaware of it. - Compacted topics cannot use tiered storage.
Practice
- (Beginner) Draw the path of a segment from the moment it rolls to the moment it is deleted from remote storage, naming each metadata state.
- (Intermediate) Explain why only leaders upload segments, and what happens to an upload in progress if leadership moves to another broker.
- Interview question: "What problems does tiered storage solve for Kafka operators besides cost?" Hint: think about how much data must be copied when a broker is replaced or partitions are reassigned, and about how retention length relates to cluster size.
Local vs Remote Log
Theory
With tiering enabled, a partition's log is the union of two overlapping ranges: remote segments, and local segments. Retention is configured separately for each, and Kafka tracks extra offsets to know where each range starts.
log start offset local log start offset log end offset
| | |
remote tier: [ seg ][ seg ][ seg ][ seg ][ seg ][ seg ][ seg ] |
local tier: [ seg ][ seg ][ seg ][ active ]------|
<------ retention.ms / retention.bytes (whole log) ---------------------->
<-- local.retention.ms / .bytes ----->
- Recently uploaded segments exist in both tiers until local retention removes the local copy.
- The active segment and not-yet-uploaded segments exist only locally.
| Offset | Meaning |
|---|---|
| Log start offset | Oldest offset readable anywhere (remote or local). Controlled by retention.ms / retention.bytes |
| Local log start offset | Oldest offset still on the broker's disks. Controlled by local.retention.ms / local.retention.bytes |
| Highest offset in remote storage | End of the last uploaded segment |
| Log end offset | Next offset to be written, as before |
Retention settings
| Setting | Default | Applies to |
|---|---|---|
retention.ms, retention.bytes | 7 days, -1 | The whole log, local and remote together |
local.retention.ms, local.retention.bytes | -2 | Local copies only. -2 means "same as the total retention" |
The local limits must not exceed the total ones. A local segment is deleted only when both its local retention has expired and it has been uploaded successfully. If uploads fail (for example, the object store is unreachable), local data accumulates and disks can fill, so monitor upload lag.
How reads are routed
| Fetch offset | Served from | Path |
|---|---|---|
| At or above the local log start offset | Local segments | Normal path: page cache, zero-copy |
| Between log start offset and local log start offset | Remote tier | Remote reader thread pool: find the segment in RLMM, fetch indexes (cached locally), fetch the byte range from the store |
| Below log start offset | Nowhere | OFFSET_OUT_OF_RANGE, then auto.offset.reset |
Followers never read data from the remote tier. When a new follower asks for an offset below the leader's local log start offset, the leader answers with OFFSET_MOVED_TO_TIERED_STORAGE. The follower then fetches the remote metadata, rebuilds its leader epoch cache and producer state from the uploaded snapshots, and starts replicating from the leader's local log start offset. A new replica therefore copies only the local tier, often a small fraction of the full history.
Example: checking both tiers
# Special timestamps for kafka-get-offsets.sh / ListOffsets
# -2 earliest (log start offset, possibly remote)
# -4 earliest-local (local log start offset)
# -5 latest-tiered (highest offset in remote storage)
for t in -2 -4 -5 -1; do
echo "--time $t"
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic clickstream --partitions 0 --time $t
done
# --time -2 clickstream:0:0
# --time -4 clickstream:0:48210000
# --time -5 clickstream:0:51003999
# --time -1 clickstream:0:51210000
Reading old data needs no client changes. This consumer reads 90 days back, and the first records simply arrive from the remote tier:
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("clickstream"));
while (true) {
// Same API as always; the broker decides whether bytes come from disk or the object store
for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofSeconds(1))) {
process(r);
}
}
}
Key Takeaways
- A tiered partition has a log start offset (oldest data anywhere) and a local log start offset (oldest data on disk). The newest uploaded segments exist in both tiers.
retention.*governs the whole log.local.retention.*governs only the broker copies and defaults to the total retention.- Local segments are deleted only after a successful upload, so a broken remote store fills local disks.
- Fetches below the local log start offset are served from the remote tier by a separate thread pool. Clients do not change.
- New followers copy only the local tier and rebuild state from uploaded snapshots, which makes reassignment and broker replacement much faster.
Practice
- (Beginner) For a topic with
retention.msof 30 days andlocal.retention.msof 12 hours, draw the two ranges and mark where a consumer with 3 hours of lag and one replaying 10 days of data would read from.- (Intermediate) Uploads to the object store have failed for 6 hours because of an expired credential. Describe what happens to local disk usage, which offsets stop moving, and which metrics would show the problem.
- Interview question: "When you add a broker to a cluster with tiered storage, how much data must the new replicas copy, and how do they get the state they need for older offsets?" Hint: think about the local log start offset,
OFFSET_MOVED_TO_TIERED_STORAGE, and what is uploaded along with each segment.
Remote Storage Manager
Theory
Kafka does not ship a production connector for any object store. It defines a narrow plugin interface, RemoteStorageManager, and leaves the storage details (authentication, multipart uploads, encryption, chunking, caching) to implementations. This keeps the broker independent of cloud SDKs and lets each vendor or community project tune for its store.
Analogy: a power socket. The house defines the shape of the plug and the voltage. Whatever appliance you plug in deals with its own internals.
The interface
package org.apache.kafka.server.log.remote.storage;
public interface RemoteStorageManager extends Configurable, Closeable {
// Upload one rolled segment and its companion files; may return small custom metadata
// that Kafka stores in the RLMM and passes back on later calls
Optional<CustomMetadata> copyLogSegmentData(RemoteLogSegmentMetadata metadata,
LogSegmentData segmentData)
throws RemoteStorageException;
// Read the segment's .log bytes from startPosition (to the end, or to endPosition inclusive)
InputStream fetchLogSegment(RemoteLogSegmentMetadata metadata, int startPosition)
throws RemoteStorageException;
InputStream fetchLogSegment(RemoteLogSegmentMetadata metadata, int startPosition, int endPosition)
throws RemoteStorageException;
// Read one index: OFFSET, TIMESTAMP, PRODUCER_SNAPSHOT, TRANSACTION, LEADER_EPOCH
InputStream fetchIndex(RemoteLogSegmentMetadata metadata, IndexType indexType)
throws RemoteStorageException;
// Delete everything stored for the segment; must succeed if it is already gone
void deleteLogSegmentData(RemoteLogSegmentMetadata metadata) throws RemoteStorageException;
}
| Type | Contents |
|---|---|
RemoteLogSegmentMetadata | Unique segment ID (topic ID, partition, UUID), start and end offsets, max timestamp, leader epochs, size, broker ID, custom metadata |
LogSegmentData | Local paths of the .log, .index, .timeindex, optional .txnindex, producer snapshot, plus the leader epoch entries as a buffer |
Contract points an implementation must respect:
- Unique IDs: every upload attempt gets a new UUID, so a retried copy never overwrites a segment another broker may be reading.
- Idempotent delete: deleting a missing segment is not an error.
- Missing transaction index: if a segment had no
.txnindex,fetchIndex(TRANSACTION)must throwRemoteResourceNotFoundException. - Range reads:
fetchLogSegmentmust support efficient byte-range reads, because fetches start in the middle of segments.
Available implementations
| Implementation | Stores | Notes |
|---|---|---|
Aiven tiered-storage-for-apache-kafka | S3, GCS, Azure Blob, filesystem | Open source (Apache 2.0). Chunking, compression, encryption, chunk cache |
| Vendor platforms (Confluent, cloud providers) | Their own object stores | Built into managed offerings, usually not configured by users |
LocalTieredStorage (Kafka test code) | Local directory | For tests and learning only |
Example: configuring a plugin and a toy implementation
# server.properties with the Aiven plugin and S3 (values illustrative)
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/tiered-storage/core/*:/opt/tiered-storage/s3/*
rsm.config.chunk.size=4194304 # upload/fetch unit: 4 MiB
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=kafka-tiered-prod
rsm.config.storage.s3.region=eu-west-1
A minimal filesystem RSM shows what the broker expects (no error handling, caching, or chunking):
public class DirectoryRemoteStorageManager implements RemoteStorageManager {
private Path root;
@Override
public void configure(Map<String, ?> configs) {
root = Path.of((String) configs.get("dir")); // from rsm.config.dir=/mnt/remote
}
private Path dirFor(RemoteLogSegmentMetadata m) {
RemoteLogSegmentId id = m.remoteLogSegmentId();
return root.resolve(id.topicIdPartition().toString()).resolve(id.id().toString());
}
@Override
public Optional<CustomMetadata> copyLogSegmentData(RemoteLogSegmentMetadata m, LogSegmentData d)
throws RemoteStorageException {
try {
Path dir = Files.createDirectories(dirFor(m));
Files.copy(d.logSegment(), dir.resolve("segment.log"));
Files.copy(d.offsetIndex(), dir.resolve("OFFSET"));
Files.copy(d.timeIndex(), dir.resolve("TIMESTAMP"));
Files.copy(d.producerSnapshotIndex(), dir.resolve("PRODUCER_SNAPSHOT"));
if (d.transactionIndex().isPresent()) {
Files.copy(d.transactionIndex().get(), dir.resolve("TRANSACTION"));
}
ByteBuffer epochs = d.leaderEpochIndex().duplicate(); // don't move the caller's position
byte[] epochBytes = new byte[epochs.remaining()];
epochs.get(epochBytes);
Files.write(dir.resolve("LEADER_EPOCH"), epochBytes);
return Optional.empty(); // no custom metadata
} catch (IOException e) {
throw new RemoteStorageException("upload failed for " + m.remoteLogSegmentId(), e);
}
}
@Override
public InputStream fetchLogSegment(RemoteLogSegmentMetadata m, int start) throws RemoteStorageException {
return fetchLogSegment(m, start, Integer.MAX_VALUE - 1);
}
@Override
public InputStream fetchLogSegment(RemoteLogSegmentMetadata m, int start, int end)
throws RemoteStorageException {
try {
InputStream in = Files.newInputStream(dirFor(m).resolve("segment.log"));
in.skipNBytes(start); // an object store would use a ranged GET
return new BoundedInputStream(in, (long) end - start + 1); // end is inclusive
} catch (IOException e) {
throw new RemoteStorageException("fetch failed", e);
}
}
@Override
public InputStream fetchIndex(RemoteLogSegmentMetadata m, IndexType type) throws RemoteStorageException {
Path file = dirFor(m).resolve(type.name());
if (!Files.exists(file)) {
throw new RemoteResourceNotFoundException(type + " not found for " + m.remoteLogSegmentId());
}
try {
return Files.newInputStream(file);
} catch (IOException e) {
throw new RemoteStorageException("index fetch failed", e);
}
}
@Override
public void deleteLogSegmentData(RemoteLogSegmentMetadata m) throws RemoteStorageException {
try (Stream<Path> files = Files.walk(dirFor(m))) {
files.sorted(Comparator.reverseOrder()).forEach(p -> p.toFile().delete());
} catch (NoSuchFileException e) {
// already deleted: deletion must be idempotent
} catch (IOException e) {
throw new RemoteStorageException("delete failed", e);
}
}
@Override
public void close() {}
}
(BoundedInputStream is any stream wrapper that stops after a given number of bytes, for example the one in Apache Commons IO.)
Key Takeaways
RemoteStorageManageris a small plugin interface: copy a segment with its indexes, fetch byte ranges and indexes, and delete.- Kafka handles scheduling, metadata, retention, and read routing. The plugin handles only bytes and the remote store.
- Implementations must use the unique segment ID for object names, support range reads, delete idempotently, and report a missing transaction index with
RemoteResourceNotFoundException. - Plugins are loaded from
remote.log.storage.manager.class.pathand configured through properties prefixed withrsm.config..
Practice
- (Beginner) List the files a broker passes to
copyLogSegmentDataand say, for each, which kind of read or recovery needs it later.- (Intermediate) Extend the toy implementation to compress the
.logfile with zstd on upload. Explain why this makesfetchLogSegment(start, end)harder, and how chunking (compressing fixed-size chunks independently) solves it.- Interview question: "Why does every upload attempt get a new UUID instead of using the segment's base offset as the object name?" Hint: think about two brokers that both believed they were leader, retried uploads, and readers holding metadata for an older copy.
Cost and Performance Considerations
Theory
Tiered storage changes the economics and the performance profile of a cluster. It is not automatically cheaper or faster: it trades expensive, replicated block storage for cheap object storage plus request charges, and trades local disk reads of old data for higher-latency remote reads. Whether it pays off depends on how much history you keep and how often you read it.
Analogy: moving old boxes from your apartment to a storage unit. Rent drops and the apartment feels bigger, but every trip to fetch a box takes time and petrol.
Where the savings come from
Prices below are illustrative, rounded list prices, and vary by region and provider:
Workload: 1 TB/day written, replication factor 3, 30-day retention
Without tiering (block storage at ~$0.08 / GB-month)
1 TB x 30 days x 3 replicas = 90 TB on broker disks -> ~$7,200 / month
(+ ~30% headroom) -> ~$9,400 / month
With tiering: 1 day local, 29 days remote
local: 1 TB x 1 day x 3 replicas = 3 TB -> ~$240 (+ headroom ~$310)
remote: 1 TB x 30 days x 1 copy = 30 TB at ~$0.023 / GB-month -> ~$690
(object stores replicate internally, so Kafka keeps one copy)
requests: PUT/GET per chunk or segment -> usually tens of dollars at this scale
-> ~$1,050 / month
Beyond disk:
| Factor | Effect of tiering |
|---|---|
| Broker count | Can be sized for throughput rather than for disk capacity, often fewer or smaller brokers |
| Reassignment and broker replacement | Copies only the local tier: minutes instead of hours |
| Retention length | Becomes a business decision instead of a disk constraint |
| Object store requests | Small chunk sizes and many partitions increase PUT/GET charges |
| Network | Cross-AZ or egress charges if brokers and the bucket are in different zones or regions |
Performance profile
| Read type | Without tiering | With tiering |
|---|---|---|
| Tail reads (normal consumers) | Page cache | Unchanged |
| Moderately old reads within local retention | Local disk | Unchanged |
| Historical reads | Local disk, evicting the page cache | Remote store: higher time to first byte (tens to hundreds of ms), but good sustained throughput, and no local page cache pollution |
| Produce latency | Affected by disk contention from historical reads | More stable, because history no longer competes for local disk |
The isolation in the last two rows is often as valuable as the cost savings: a replay of last month's data no longer evicts the hot tail from the page cache (Page Cache Usage, Section 7.1).
Tuning and monitoring
| Setting | Purpose |
|---|---|
local.retention.ms / local.retention.bytes | Keep enough locally for normal consumer lag and fast recovery, typically hours to a day |
remote.log.reader.threads | Threads serving remote fetches. Raise when many consumers replay history at once |
remote.log.index.file.cache.total.size.bytes | Local disk cache for remote indexes, avoiding a remote GET per lookup |
remote.log.manager.copy.max.bytes.per.second, remote.log.manager.fetch.max.bytes.per.second | Quotas so uploads and remote reads do not saturate the network |
Plugin chunk size and cache (for example rsm.config.chunk.size) | Larger chunks mean fewer requests but more read amplification |
Key metrics (kafka.server:type=BrokerTopicMetrics, per topic):
RemoteCopyBytesPerSecandRemoteCopyErrorsPerSec: upload throughput and failures.RemoteCopyLagBytes/RemoteCopyLagSegments: data waiting to be uploaded. A steady rise means uploads cannot keep up and local disks will fill.RemoteFetchBytesPerSec,RemoteFetchRequestsPerSec,RemoteFetchErrorsPerSec: historical read load.RemoteLogSizeBytes: data held in the remote tier.
When it is a poor fit
- Short retention (hours) where everything fits on local disk anyway.
- Compacted topics (not supported).
- Workloads that constantly reread old data with tight latency requirements.
- Clusters where the object store is far away (different region) or expensive to reach.
Example: estimating the break-even point
def monthly_cost(tb_per_day: float, retention_days: int, local_days: float, rf: int = 3,
block_gb_month=0.08, object_gb_month=0.023, headroom=0.3,
requests_per_month=0.0) -> tuple[float, float]:
gb_day = tb_per_day * 1000
without = gb_day * retention_days * rf * block_gb_month * (1 + headroom)
local = gb_day * local_days * rf * block_gb_month * (1 + headroom)
remote = gb_day * retention_days * object_gb_month # one copy in the object store
return round(without), round(local + remote + requests_per_month)
for days in (1, 3, 7, 30, 90):
without, tiered = monthly_cost(tb_per_day=1, retention_days=days, local_days=1)
print(f"{days:>3} days retention: without ${without:>7,} tiered ${tiered:>7,}")
# 1 days retention: without $ 312 tiered $ 335 <- no benefit
# 3 days retention: without $ 936 tiered $ 381
# 7 days retention: without $ 2,184 tiered $ 473
# 30 days retention: without $ 9,360 tiered $ 1,002
# 90 days retention: without $ 28,080 tiered $ 2,382
Key Takeaways
- Tiering replaces "history x replication factor" on block storage with one copy on object storage, which is much cheaper for long retention.
- Savings grow with retention length. With retention close to the local window, tiering adds complexity for no benefit.
- Tail reads are unchanged. Historical reads have higher latency but no longer pollute the page cache or compete with producers for disk.
- Reassignment, broker replacement, and scaling get much faster because only the local tier is copied.
- Monitor upload lag and errors. A failing remote store turns into full local disks.
Practice
- (Beginner) Using the cost function above, find the retention at which tiering becomes cheaper for 5 TB/day with 12 hours of local retention.
- (Intermediate) A team enables tiering with
local.retention.msof 10 minutes to minimize disk cost. Consumers are occasionally 30 minutes behind during deployments. Describe the performance and cost effects, and propose a better value with reasoning.- Interview question: "Your CFO asks whether tiered storage will cut the Kafka bill in half. What do you need to know before answering?" Hint: retention versus local window, replication factor, read patterns over old data, request and network charges, and whether broker count is bound by disk or by throughput.
<a id="8-delivery-semantics-and-transactions"></a>