5. Consumers
Consumers turn stored records into work: updating a database, sending an email, feeding a model, or producing new events. Kafka's broker does not push records or track which ones were delivered. The consumer pulls them, decides how fast to read, and records its own progress as offsets, and a group of consumers shares a topic's partitions among themselves. This chapter covers the consumer API and poll loop, how consumer groups divide and rebalance work, how offsets are committed and moved, and the patterns that decide whether a record is processed at most once, at least once, or effectively once. Most consumer incidents in production, such as lag that keeps growing, rebalance storms, duplicate processing, and silently skipped data, come from one of these topics.
<a id="51-consumer-fundamentals"></a>
5.1 Consumer Fundamentals
This section explains what the consumer client does internally, how the poll loop drives everything, how fetch settings shape throughput and latency, how bytes become objects, and how to start and stop a consumer cleanly.
Consumer API Overview
Theory
Message brokers usually push messages to consumers and track each message's delivery. Kafka deliberately does the opposite: consumers pull. A consumer asks a broker "give me records from partition 3, starting at offset 18234", processes what comes back, and asks again. This has three consequences that shape the whole API:
- The consumer controls its pace. A slow consumer simply falls behind (it has lag) instead of being flooded. The broker never has to buffer per-consumer queues.
- The consumer owns its position. The broker stores no per-record delivery state, only a committed offset per group and partition (Section 2.1). Rewinding is just reading from a smaller offset.
- Batching is natural. One fetch returns many records from many partitions, which is why a single consumer can read hundreds of thousands of records per second.
Like the producer, the consumer is a smart client. It discovers the cluster, finds the leader of each partition it reads, joins a consumer group (Section 5.2), fetches from several brokers in parallel, deserializes records, and commits offsets.
Analogy: a consumer is like a reader in a library that never removes books. The reader picks up where their bookmark is, reads at their own speed, and moves the bookmark when done. The librarian (the broker) does not care who has read what; it only keeps the books on the shelf and stores each reader's bookmark if asked.
Two ways to get partitions
| Method | Who decides which partitions this consumer reads | Group coordination | Typical use |
|---|---|---|---|
subscribe(topics) / subscribe(Pattern) | The consumer group (Section 5.2) | Yes: membership, rebalancing, committed offsets | Almost all applications |
assign(partitions) | You, explicitly | No membership or rebalancing. Commits still work if group.id is set. | Tools, replay jobs, frameworks that manage assignment themselves |
You use one or the other on a given consumer instance, never both.
Inside the Java consumer
application thread brokers
-------------------------------------------------- ------------------------------
consumer.poll(timeout)
|
|-- group coordination: heartbeats, rebalances <------> group coordinator (Section 5.2)
| (classic protocol: heartbeats run on a
| background thread; the work happens in poll)
|
|-- fetcher: one FetchRequest per leader broker <------> broker 1 (leads orders-0, orders-3)
| for all assigned partitions led by it broker 2 (leads orders-1)
| results buffered in memory, per partition broker 3 (leads orders-2)
|
|-- [Key/Value Deserializer] byte[] -> object
|-- [ConsumerInterceptors] optional hooks (Section 16.3)
|-- auto commit, if enabled and due (Section 5.3)
v
returns ConsumerRecords (up to max.poll.records)
Since Kafka 3.7, the Java client has a second internal implementation, selected with group.protocol=consumer (Section 5.2), which moves network I/O and group coordination to a dedicated background thread. The public API is the same in both.
Essential configuration
| Setting | Default | Purpose |
|---|---|---|
bootstrap.servers | none (required) | Initial brokers used to discover the cluster (Section 2.2) |
key.deserializer, value.deserializer | none (required in Java) | Turn bytes into keys and values (topic "Deserialization") |
group.id | none | The consumer group. Required for subscribe() and for committing offsets. |
client.id | generated (consumer-<group>-1, ...) | Name shown in broker logs, quotas, and metrics |
enable.auto.commit | true | Commit offsets automatically every auto.commit.interval.ms (Section 5.3) |
auto.offset.reset | latest | Where to start when there is no valid committed offset (Section 5.3) |
max.poll.records | 500 | Maximum records returned by one poll() |
max.poll.interval.ms | 300000 | Maximum time between two poll() calls before the consumer is removed from the group |
isolation.level | read_uncommitted | read_committed hides aborted transactional records (Chapter 8) |
The Java client exposes these names as constants in ConsumerConfig (for example ConsumerConfig.GROUP_ID_CONFIG).
The ConsumerRecord
| Accessor | Returns |
|---|---|
topic(), partition(), offset() | Where the record is stored |
key(), value() | The deserialized key and value (either can be null) |
timestamp(), timestampType() | Record time and whether it is CreateTime or LogAppendTime (Section 2.1) |
headers() | Record headers |
leaderEpoch() | Leader epoch at the time of the write, used to detect log truncation |
Example: a minimal consumer in Java
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.List;
import java.util.Properties;
public class OrderConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // new group: read history
try (Consumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("orders")); // the group decides the partitions
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> rec : records) {
System.out.printf("%s-%d@%d key=%s value=%s%n",
rec.topic(), rec.partition(), rec.offset(), rec.key(), rec.value());
}
// enable.auto.commit=true (default): offsets are committed inside later poll() calls
}
}
}
}
orders-1@0 key=CUST-77 value={"orderId":"ORD-1"}
orders-1@1 key=CUST-77 value={"orderId":"ORD-2"}
orders-1@2 key=CUST-77 value={"orderId":"ORD-3"}
This loop runs forever and cannot be stopped cleanly. The topic "Consumer Lifecycle and Shutdown" shows the correct structure.
Example: the same consumer in Python
from confluent_kafka import Consumer
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "order-service",
"auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])
try:
while True:
msg = consumer.poll(0.5) # one message (or None) per call
if msg is None:
continue
if msg.error(): # errors arrive as messages, not exceptions
print(f"consumer error: {msg.error()}")
continue
print(f"{msg.topic()}-{msg.partition()}@{msg.offset()} "
f"key={msg.key()} value={msg.value()}") # key and value are bytes
finally:
consumer.close() # leave the group and commit final offsets
Other useful consumer methods
| Method | Purpose |
|---|---|
commitSync(), commitAsync() | Commit offsets manually (Section 5.3) |
seek(), seekToBeginning(), seekToEnd() | Move the position (Section 5.3) |
position(tp), committed(Set<TopicPartition>) | Current position and committed offset |
assignment(), subscription() | Partitions currently owned, topics subscribed |
pause(), resume() | Stop and restart fetching from partitions without leaving the group (Section 5.4) |
beginningOffsets(), endOffsets(), offsetsForTimes() | Look up offsets by position or timestamp |
wakeup() | Interrupt a blocking poll() from another thread (topic "Consumer Lifecycle and Shutdown") |
metrics() | Client metrics such as fetch rate and lag |
Key Takeaways
- Kafka consumers pull records and own their position, so the broker stores no per-record delivery state and slow consumers only accumulate lag.
subscribe()lets the consumer group assign partitions.assign()takes manual control and disables group membership.bootstrap.servers, the deserializers, and (forsubscribe())group.idare the essential settings. The defaultsenable.auto.commit=trueandauto.offset.reset=latestare convenient but rarely what a careful application wants (Section 5.3).librdkafka-based clients return one message perpoll()and report errors as messages instead of exceptions.
Practice
- (Beginner) Run the Java or Python example against the topic you produced to in Chapter 4. Run it a second time with the same
group.idand explain why nothing is printed. Run it again with a newgroup.id.- (Intermediate) Rewrite the example to use
assign()on partitions 0 and 1 oforderswith nogroup.id. Which methods now fail, and why?- Interview question: "Why does Kafka use a pull model for consumers instead of pushing records to them?" Hint: think about flow control for slow consumers, batching efficiency, what state the broker would need to keep per consumer, and how replay would work.
The Poll Loop
Theory
A Kafka consumer does almost nothing on its own. Fetching, deserializing, heartbeating (in the classic protocol, the heartbeat thread only keeps the session alive), joining rebalances, running rebalance callbacks, and auto-committing all happen inside poll() or are triggered by it. So the structure of every consumer application is a loop: poll, process, (commit), repeat. Getting this loop right matters more than any configuration setting.
poll(timeout) returns as soon as records are available, or after timeout if none arrive. It never returns more than max.poll.records records, even if more are buffered; the rest are kept for the next call.
Analogy: the poll loop is like a waiter's rounds in a restaurant. Each round, the waiter collects orders from the kitchen window (poll), serves them (process), and reports back to the manager (commit). If the waiter disappears into one table for too long, the manager assumes they have quit and gives their tables to someone else.
The two timers that remove a consumer from its group
classic protocol
heartbeat thread: hb hb hb hb hb hb every heartbeat.interval.ms (3 s)
|----|----|----|----|----|
no heartbeat for session.timeout.ms (45 s) -> consumer is dead
(crash, network partition)
application thread:
poll() [ process 500 records ............................. ] poll()
|<----------------- must be < max.poll.interval.ms (5 min) ------------->|
exceeded -> the consumer proactively leaves the group, partitions move,
and the next commit fails with CommitFailedException
| Timer | Detects | Default | Triggered by |
|---|---|---|---|
session.timeout.ms | A dead or unreachable process | 45000 | No heartbeat reaches the coordinator |
max.poll.interval.ms | A live process whose loop is stuck or too slow | 300000 | Too long between two poll() calls |
Separating the two means a consumer can spend minutes processing a batch without being declared dead, while a crashed process is still detected in seconds. With the new consumer protocol (Section 5.2), the session timeout and heartbeat interval are configured on the broker instead (group.consumer.session.timeout.ms, group.consumer.heartbeat.interval.ms), but max.poll.interval.ms still applies on the client.
Sizing the loop
The rule is: max.poll.records x worst-case time per record < max.poll.interval.ms, with a safety margin. If records take 50 ms each in the worst case, 500 records take 25 seconds, comfortably inside 5 minutes. If they can take 2 seconds each (a slow external API), 500 records take over 16 minutes: lower max.poll.records to about 100, raise max.poll.interval.ms, or move the work off the poll thread (Section 5.4).
Example: a well-structured poll loop
consumer.subscribe(List.of("orders"));
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
if (records.isEmpty()) continue;
for (TopicPartition tp : records.partitions()) { // per-partition iteration
List<ConsumerRecord<String, String>> batch = records.records(tp);
for (ConsumerRecord<String, String> rec : batch) {
process(rec); // ordering preserved within tp
}
long next = batch.get(batch.size() - 1).offset() + 1; // commit "next to read"
consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(next)), null);
}
}
Common poll loop mistakes
| Mistake | Consequence |
|---|---|
Calling poll() with a very long timeout and doing periodic work in the loop | Periodic work (flushes, metrics) is delayed until records arrive |
| Blocking calls in processing with no time limit (HTTP with no timeout, locks) | max.poll.interval.ms exceeded, rebalance, duplicates |
| Catching and ignoring every exception inside the loop | Poison records are skipped silently, or the loop spins on them forever |
Calling poll() from several threads on one KafkaConsumer | ConcurrentModificationException: the consumer is not thread-safe |
Thread.sleep() for backoff inside the loop, without pausing partitions | The group may consider the consumer stuck. Use pause() instead (Section 5.4). |
Key Takeaways
- Almost all consumer work happens inside
poll(). The poll loop is the backbone of every consumer application. session.timeout.msdetects dead processes;max.poll.interval.msdetects live but stuck ones. Exceeding either triggers a rebalance.- Keep
max.poll.records x worst-case processing timewell belowmax.poll.interval.ms. - A
KafkaConsumermust be used from one thread only (exceptwakeup()).
Practice
- (Beginner) Add
Thread.sleep(1000)per record to a consumer withmax.poll.records=500andmax.poll.interval.ms=60000. Watch the consumer log and describe the group withkafka-consumer-groups.sh. What happens and when?- (Intermediate) Each record calls a payment API with a p99 latency of 800 ms and a timeout of 5 s. Choose
max.poll.recordsandmax.poll.interval.ms, and justify the numbers with a worst-case calculation.- Interview question: "What is the difference between
session.timeout.msandmax.poll.interval.ms, and why does Kafka need both?" Hint: separate "the process is gone" from "the process is alive but not making progress", and think about which thread detects each.
Fetch Configuration
Theory
Each fetch is a trade-off between the producer-like choices of Chapter 4: wait for more data and get bigger, more efficient responses, or return quickly with whatever is there. The broker holds a fetch request open until enough data has accumulated or a deadline passes, a technique called long polling. This avoids both busy-waiting on empty partitions and returning tiny responses one record at a time.
Analogy: a fetch is like asking a shop to call you when your order is ready. You can say "call me as soon as anything arrives" (low latency) or "call me when there is at least a full box, but no later than in half a second" (fewer, fuller trips).
How the settings interact
consumer --FetchRequest-------------------------------------------> broker (leader)
min bytes = fetch.min.bytes (1) |
max wait = fetch.max.wait.ms (500) | holds the request until
max bytes = fetch.max.bytes (50 MB) per response | min bytes are available
per partition: max.partition.fetch.bytes (1 MB) | or max wait has passed
consumer <-FetchResponse (records for many partitions)--------------+
|
v buffered in the consumer
poll() hands out at most max.poll.records (500) at a time
| Setting | Default | Raise it to... | Lower it to... |
|---|---|---|---|
fetch.min.bytes | 1 | Get fuller responses, fewer requests, less broker CPU | Return as soon as any data exists |
fetch.max.wait.ms | 500 | Allow the broker to wait longer for fetch.min.bytes | Cap the added latency |
fetch.max.bytes | 52428800 (50 MB) | Allow bigger responses across partitions | Limit consumer memory |
max.partition.fetch.bytes | 1048576 (1 MB) | Read more per partition per request | Spread a response more fairly across partitions |
max.poll.records | 500 | Process more per loop iteration | Keep each iteration short (topic "The Poll Loop") |
receive.buffer.bytes | 65536 | Improve throughput on high-latency links (-1 uses the OS default) | Rarely needed |
max.poll.records does not affect what is fetched over the network, only how much poll() hands to your code at once. The byte limits are soft: if the first record batch in a partition is larger than the limit, it is still returned so the consumer can make progress.
Fetching from the closest replica
By default, consumers fetch from the partition leader. In multi-zone or multi-region clusters, that can mean paying for cross-zone traffic on every read. Since Kafka 2.4 (KIP-392), consumers can fetch from a follower in the same rack:
# broker
broker.rack=eu-west-1a
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
# consumer
client.rack=eu-west-1a
Followers only serve data up to the high watermark they know about, so latency can rise slightly. The trade is usually worth it for high-volume consumers in the cloud (Section 2.2).
Starting profiles
| Profile | Settings | Typical use |
|---|---|---|
| Low latency | fetch.min.bytes=1, fetch.max.wait.ms=100 | User-facing notifications, request/response flows |
| Balanced (defaults) | fetch.min.bytes=1, fetch.max.wait.ms=500 | Most microservices |
| High throughput | fetch.min.bytes=1048576, fetch.max.wait.ms=500, max.partition.fetch.bytes=4194304, max.poll.records=2000 | Bulk loaders, analytics sinks, log shippers |
Example: measuring the effect with the perf tool
BS=localhost:9092
# Default fetch settings
bin/kafka-consumer-perf-test.sh --bootstrap-server $BS --topic perf --messages 5000000 \
--group perf-default --show-detailed-stats --reporting-interval 5000
# Throughput profile: write the settings to a file and pass it in
cat > /tmp/fetch.properties <<'EOF'
fetch.min.bytes=1048576
max.partition.fetch.bytes=4194304
max.poll.records=2000
EOF
bin/kafka-consumer-perf-test.sh --bootstrap-server $BS --topic perf --messages 5000000 \
--group perf-tuned --consumer.config /tmp/fetch.properties
# Compare MB.sec and fetch.nMsg.sec between the two runs
Watch the consumer metrics fetch-size-avg, records-per-request-avg, fetch-latency-avg, and fetch-rate to confirm the change does what you intended.
Key Takeaways
- The broker holds fetch requests until
fetch.min.bytesare available orfetch.max.wait.mspasses (long polling). fetch.max.bytesandmax.partition.fetch.bytesbound memory per response.max.poll.recordsonly controls how muchpoll()returns, not what is fetched.- Raising
fetch.min.bytesimproves throughput and broker efficiency at the cost of up tofetch.max.wait.msof latency. - With
client.rackand a rack-aware replica selector, consumers can read from a nearby follower and cut cross-zone traffic.
Practice
- (Beginner) Run
kafka-consumer-perf-test.shwith the default settings and with the throughput profile above. Record MB/s and explain the difference.- (Intermediate) A consumer reads 200 partitions, and its JVM runs out of memory during catch-up after an outage. Estimate the worst-case memory used by fetched data with the default settings, and propose changes.
- Interview question: "Your consumer's end-to-end latency is about 500 ms even though the topic has very little traffic. Why might that be, and what would you change?" Hint: look at how
fetch.min.bytesandfetch.max.wait.msinteract when data arrives slowly, and at the producer'slinger.ms.
Deserialization
Theory
Kafka stores and transfers only bytes. The producer turned objects into bytes with a serializer (Section 4.1); the consumer must turn them back with a matching deserializer. The consumer has no way to know what format the bytes are in, so producer and consumer must agree on it. That agreement, and how it evolves over time, is the subject of Chapter 6. This topic covers the mechanics on the consumer side and, just as important, what happens when bytes cannot be deserialized.
Analogy: a deserializer is a translator who only speaks one language. If a letter arrives in a different language, the translator cannot read it, and someone must decide whether to set the letter aside, ask for help, or stop the mail entirely.
Built-in deserializers (Java)
| Class | Produces | Matching serializer |
|---|---|---|
StringDeserializer | String (UTF-8 by default) | StringSerializer |
ByteArrayDeserializer | byte[], unchanged | ByteArraySerializer |
IntegerDeserializer, LongDeserializer, DoubleDeserializer, ... | Numbers | Matching numeric serializer |
UUIDDeserializer | UUID | UUIDSerializer |
VoidDeserializer | Always null | VoidSerializer |
For structured data, use a schema-aware deserializer such as KafkaAvroDeserializer, KafkaProtobufDeserializer, or KafkaJsonSchemaDeserializer with a schema registry (Chapter 6).
Example: a custom JSON deserializer
public class OrderDeserializer implements Deserializer<Order> {
private final ObjectMapper mapper = new ObjectMapper();
@Override
public Order deserialize(String topic, byte[] data) {
if (data == null) return null; // tombstones and null values
try {
return mapper.readValue(data, Order.class);
} catch (IOException e) {
// Wrap in SerializationException so the consumer reports topic, partition, offset
throw new SerializationException("cannot parse order from " + topic, e);
}
}
}
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, OrderDeserializer.class.getName());
Deserializer also has a deserialize(String topic, Headers headers, byte[] data) variant, useful when the format or schema version is carried in a header.
Poison pills
A poison pill is a record that can never be deserialized: a producer bug, a wrong topic, or an incompatible schema change. Deserialization happens inside poll(), so the exception comes out of poll(), and the consumer's position does not move past the bad record. Calling poll() again throws the same exception, forever.
partition orders-2: ... | 811 ok | 812 ok | 813 BAD | 814 ok | 815 ok | ...
^
every poll() throws RecordDeserializationException here
Since Kafka 2.8 (KIP-334), the Java client throws RecordDeserializationException, which tells you exactly where the bad record is, so you can skip it:
while (running) {
try {
ConsumerRecords<String, Order> records = consumer.poll(Duration.ofMillis(500));
records.forEach(this::process);
} catch (RecordDeserializationException e) {
log.error("poison pill at {}@{}", e.topicPartition(), e.offset(), e);
deadLetter.send(e.topicPartition(), e.offset()); // record it somewhere
consumer.seek(e.topicPartition(), e.offset() + 1); // move past it
}
}
An alternative is to deserialize to byte[] and parse in your own code, so a bad record is just one failed process() call:
| Approach | Pros | Cons |
|---|---|---|
Typed deserializer, catch RecordDeserializationException | Clean types in the processing code | Must handle the exception in the poll loop |
ByteArrayDeserializer + parse in process() | Full control; the raw bytes are available for a dead letter topic | Parsing logic moves into the application |
Framework error handlers (Spring Kafka ErrorHandlingDeserializer, Kafka Streams DeserializationExceptionHandler) | Configurable policies (log and continue, fail, dead letter) | Framework-specific |
Python note
confluent-kafka returns raw bytes from msg.value() unless you use DeserializingConsumer or call a deserializer yourself, so poison pills surface in your own code rather than inside poll():
import json
msg = consumer.poll(1.0)
if msg is not None and not msg.error():
try:
order = json.loads(msg.value()) # bytes -> dict
except (ValueError, TypeError):
dead_letter.produce("orders.dlt", key=msg.key(), value=msg.value())
Key Takeaways
- Consumers must use deserializers that match the producer's serializers. Kafka itself does not know the format.
- Handle
nullin custom deserializers: tombstones and null values are normal. - A record that cannot be deserialized blocks its partition, because
poll()keeps throwing at the same offset. CatchRecordDeserializationExceptionandseek()past it, ideally after copying it to a dead letter topic. - Deserializing to bytes and parsing in application code is a simple, robust alternative.
Practice
- (Beginner) Produce a mix of valid JSON and the plain string
not-jsonto a topic. Consume with a JSON deserializer and observe the behavior. Then add theRecordDeserializationExceptionhandler and confirm the consumer continues.- (Intermediate) Extend the handler to copy the raw bytes of the bad record to
orders.dlt, including headers with the original topic, partition, offset, and exception message. Which deserializer setup makes the raw bytes available?- Interview question: "A consumer is stuck at the same offset and its logs repeat the same exception every few hundred milliseconds. What is happening and how do you fix it without losing data?" Hint: think about where deserialization happens, why the position does not advance, and how a dead letter topic preserves the record.
Consumer Lifecycle and Shutdown
Theory
A KafkaConsumer is a stateful member of a group. While it runs, it owns partitions, holds uncommitted progress, and keeps a session with the group coordinator. How it stops matters: a consumer that leaves cleanly hands its partitions over in milliseconds and commits its final offsets, while one that is killed leaves its partitions unowned until session.timeout.ms expires (45 seconds by default), and anything processed after the last commit is processed again by its successor.
The difficulty is that poll() blocks the consumer's only thread, and the consumer is not thread-safe. The one method you may call from another thread is wakeup(): it makes a blocked (or the next) poll() throw WakeupException, which the poll thread catches to exit the loop and close the consumer itself.
Analogy: closing a consumer is like a shift change at a help desk. A good employee finishes the current call, writes up notes (commit), and tells the supervisor they are leaving (leave group) so someone takes over at once. An employee who just walks out leaves callers waiting until the supervisor notices the empty chair.
Lifecycle phases
new KafkaConsumer(props) validate config, create deserializers; no network yet
|
v
subscribe(topics, listener) records intent only; nothing happens until poll()
|
v
first poll() find coordinator, join group, get assignment (Section 5.2)
| fetch committed offsets (or apply auto.offset.reset)
v
steady state -------------------------------------------------------------+
| poll / process / commit |
| rebalances: onPartitionsRevoked / onPartitionsAssigned callbacks |
| fatal error (authentication, authorization, fenced instance) --------+--> close
v
wakeup() from another thread -> poll() throws WakeupException
|
v
close() / close(timeout) commit (if auto commit), leave group, release resources
default timeout: 30 s
What close() does
| Step | Detail |
|---|---|
| Final auto commit | Only if enable.auto.commit=true. With manual commits, commit yourself before closing. |
| Revoke callbacks | onPartitionsRevoked runs, so you can flush and commit per partition |
| Leave the group | Sends a leave request so partitions are reassigned immediately (static members, Section 5.2, do not leave) |
| Release resources | Sockets, buffers, metrics, and background threads |
Example: graceful shutdown in Java
public final class OrderWorker implements Runnable {
private final KafkaConsumer<String, String> consumer;
private final CountDownLatch stopped = new CountDownLatch(1);
public OrderWorker(Properties props) {
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
this.consumer = new KafkaConsumer<>(props);
}
@Override
public void run() {
try {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
records.forEach(this::process);
consumer.commitAsync(); // fast path while running
}
} catch (WakeupException e) {
// expected on shutdown: fall through to the cleanup below
} finally {
try {
consumer.commitSync(); // last commit must not be lost
} finally {
consumer.close(Duration.ofSeconds(10)); // leave the group, bounded wait
stopped.countDown();
}
}
}
public void shutdown() throws InterruptedException {
consumer.wakeup(); // the only thread-safe method
stopped.await(); // wait for the poll thread to finish
}
}
// Wiring: run on its own thread, stop on SIGTERM
OrderWorker worker = new OrderWorker(props);
Thread t = new Thread(worker, "order-worker");
t.start();
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
try { worker.shutdown(); } catch (InterruptedException ignored) { }
}));
Example: graceful shutdown in Python
import signal
from confluent_kafka import Consumer
running = True
def stop(*_):
global running
running = False
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
consumer = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "order-service",
"enable.auto.commit": False})
consumer.subscribe(["orders"])
try:
while running: # short poll timeout so the flag is checked
msg = consumer.poll(1.0)
if msg is None or msg.error():
continue
process(msg)
consumer.commit(message=msg, asynchronous=True)
finally:
consumer.commit(asynchronous=False) # synchronous final commit
consumer.close() # leave the group
(In Python the final commit() raises an exception if there is nothing to commit; production code should catch KafkaException with _NO_OFFSET.)
Shutdown in containers
Kubernetes sends SIGTERM and then SIGKILL after terminationGracePeriodSeconds (30 seconds by default). The consumer must finish its current batch, commit, and close within that window, so keep the batch processing time and the close() timeout well below it. Rolling restarts of many instances are where clean shutdowns, cooperative rebalancing, and static membership (Section 5.2) pay off most.
Key Takeaways
KafkaConsumeris not thread-safe. Onlywakeup()may be called from another thread; it makespoll()throwWakeupException.- Exit the poll loop on
WakeupException, commit synchronously, thenclose()with a bounded timeout, all on the poll thread. - A clean
close()leaves the group immediately, so partitions move in milliseconds. A killed consumer holds them untilsession.timeout.msexpires. - Fit the whole shutdown into the platform's grace period (for example, Kubernetes' 30 seconds).
Practice
- (Beginner) Run two instances of a consumer in the same group. Stop one with
Ctrl+C(clean close) and the other withkill -9. Compare how long it takes before the remaining instance owns all partitions.- (Intermediate) Add a
ConsumerRebalanceListenerto the Java example and log inonPartitionsRevoked. Verify that it runs duringclose(), and move the final commit into it. What are the advantages?- Interview question: "How do you shut down a Kafka consumer that is blocked in
poll(), given that the consumer is not thread-safe?" Hint: there is exactly one method designed for this, and the cleanup must still happen on the polling thread.
<a id="52-consumer-groups"></a>
5.2 Consumer Groups
Consumer groups are how Kafka scales consumption and survives failures: the partitions of a topic are divided among the members of a group, and redistributed whenever members join, leave, or fail. This section covers how membership works, which broker coordinates it, how partitions are assigned, and the three generations of rebalancing: eager, cooperative, and the broker-driven protocol introduced in Kafka 4.0.
Group Membership
Theory
A single consumer can only process so many records per second. To scale, you run several instances of the same application, but then they must not all process the same records. Kafka solves this with consumer groups: every consumer configured with the same group.id is a member of the same group, and each partition of the subscribed topics is assigned to exactly one member of the group at a time.
Two rules follow, and they explain most of Kafka's consumption behavior:
- Within a group, partitions are the unit of parallelism. A group can have at most one active consumer per partition. Extra members sit idle.
- Across groups, consumption is independent. Each group has its own committed offsets and receives every record. This is how Kafka offers queue semantics (within a group) and publish-subscribe semantics (across groups) with the same topic (Section 1.1).
Analogy: a topic is a stack of mail sorted into numbered trays (partitions). A department (a group) assigns each tray to one clerk. Add clerks and the trays are spread more thinly; add more clerks than trays and some clerks have nothing to do. Another department gets its own copy of every letter and organizes its clerks independently.
How partitions map to members
topic "orders", 6 partitions
group "billing" (3 members) group "analytics" (1 member)
+-----------+ +-----------+ +-----------+ +-------------------------------+
| billing-1 | | billing-2 | | billing-3 | | analytics-1 |
| P0 P1 | | P2 P3 | | P4 P5 | | P0 P1 P2 P3 P4 P5 |
+-----------+ +-----------+ +-----------+ +-------------------------------+
group "fraud" (8 members): 6 members own one partition each, 2 members are idle
| Members vs partitions | Result |
|---|---|
| Fewer members than partitions | Some members own several partitions |
| Equal | One partition per member: maximum parallelism |
| More members than partitions | Extra members are idle hot standbys (share groups, Section 5.4, lift this limit) |
What membership gives you
- Load balancing: partitions are spread over members by an assignment strategy (topic "Partition Assignment Strategies").
- Fault tolerance: when a member fails or leaves, its partitions are reassigned to the others (topic "Rebalancing").
- Shared progress: committed offsets belong to the group, not to an individual member, so a new owner continues where the previous owner stopped (Section 5.3).
- Ordering per key: since a partition has one owner, all records with the same key are processed in order by one member.
Example: watching membership change
BS=localhost:9092
bin/kafka-topics.sh --bootstrap-server $BS --create --topic orders --partitions 6 --replication-factor 1
# Terminal 1, 2, 3: start three members of the same group
bin/kafka-console-consumer.sh --bootstrap-server $BS --topic orders --group billing
# Terminal 4: see who owns what
bin/kafka-consumer-groups.sh --bootstrap-server $BS --describe --group billing --members --verbose
GROUP CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT
billing console-consumer-1a2b... /10.0.0.5 console-1 2 orders(0,1)
billing console-consumer-3c4d... /10.0.0.6 console-2 2 orders(2,3)
billing console-consumer-5e6f... /10.0.0.7 console-3 2 orders(4,5)
Stop one member and describe again: its two partitions now belong to the remaining members.
Key Takeaways
- Consumers with the same
group.idform a group, and each partition is owned by exactly one member at a time. - Partitions cap a classic group's parallelism. Members beyond the partition count are idle.
- Different groups read independently, each with its own offsets: queue semantics within a group, publish-subscribe across groups.
- Committed offsets belong to the group, so ownership can move between members without losing progress.
Practice
- (Beginner) Create a topic with 4 partitions. Start 1, then 2, then 5 consumers in one group, describing the group after each step. Record how many partitions each member owns.
- (Intermediate) Two teams both want every record of
orders, and one of them needs four times the throughput of the other. Describe the group layout and the minimum partition count you would use.- Interview question: "How does Kafka provide both queue and publish-subscribe semantics with a single abstraction?" Hint: compare what happens to a record within one group and across several groups.
Group Coordinator
Theory
Someone has to know who is in a group, decide when the membership has changed, and store the group's committed offsets. In Kafka this job belongs to a broker called the group coordinator. There is no central coordinator for all groups: each group is mapped to one partition of __consumer_offsets, and the broker that leads that partition coordinates the group (Section 3.3). Groups are therefore spread evenly across brokers, and coordination fails over automatically when partition leadership moves.
Analogy: a large school assigns each class to one homeroom teacher, chosen by the class name. The teacher keeps the attendance list and each student's progress notes. If that teacher is ill, the substitute who takes over the room also takes over the notes, because they are kept in the room, not in the teacher's head.
Finding the coordinator
consumer any broker
|--- FindCoordinator(key="billing", type=GROUP) ------->|
|<-- coordinator = broker 2 ----------------------------| hash("billing") % 50 = 9
| leader of __consumer_offsets-9 = broker 2
|
|--- JoinGroup / Heartbeat / OffsetCommit / ... ------> broker 2 (group coordinator)
Responsibilities
| Responsibility | Detail |
|---|---|
| Membership | Tracks members, their subscriptions, and liveness (heartbeats) |
| Rebalance orchestration | Starts a rebalance when membership or subscriptions change, and distributes the result (topic "Rebalancing") |
| Generations / epochs | Increments a generation (classic) or group epoch (new protocol) on every change, and rejects requests from stale members |
| Offset storage | Writes commits to __consumer_offsets and serves OffsetFetch requests |
| Expiry | Deletes offsets of empty groups after offsets.retention.minutes (7 days by default) |
The generation fence
Each completed rebalance gets a new generation ID (in the new protocol, an epoch). Every commit and heartbeat carries it. If a member was removed while it was stuck (for example, it exceeded max.poll.interval.ms) and then tries to commit, its generation is stale and the coordinator rejects the request:
gen 7: member A owns P0
A stalls for 6 minutes ...
gen 8: coordinator removes A, gives P0 to B; B starts processing P0
A wakes up and calls commitSync() with generation 7
-> CommitFailedException: the group has rebalanced and assigned partitions to other members
Without this fence, a zombie member could overwrite the offsets of the new owner.
When the coordinator moves
If the broker leading the group's __consumer_offsets partition fails, a new leader is elected (Section 2.4). The new leader reads the partition to rebuild the group's state, then serves requests. Clients see NOT_COORDINATOR or COORDINATOR_NOT_AVAILABLE, look the coordinator up again, and retry. This is transparent to your code but adds a short pause.
Example: finding a group's coordinator
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing --state
GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
billing broker-2:9092 (2) cooperative-sticky Stable 3
// Programmatic equivalent with the Admin client
try (Admin admin = Admin.create(Map.of("bootstrap.servers", "localhost:9092"))) {
ConsumerGroupDescription d = admin.describeConsumerGroups(List.of("billing"))
.describedGroups().get("billing").get();
System.out.println(d.coordinator() + " " + d.state() + " " + d.members().size());
}
Key Takeaways
- Each group has one coordinator: the leader of the
__consumer_offsetspartition chosen by hashing the group ID. - The coordinator tracks membership, runs rebalances, and stores committed offsets.
- Generations (or epochs) fence stale members, which is why a consumer that stalled too long gets
CommitFailedException. - Coordinator failover follows partition leadership. Clients rediscover it and retry automatically.
Practice
- (Beginner) Find the coordinator of a group with
--describe --state. Then describe__consumer_offsetsand find the partition whose leader matches.- (Intermediate) On a 3-broker cluster, stop the coordinator broker of an active group while it consumes. Record what the consumer logs and how long consumption pauses.
- Interview question: "Why is group coordination distributed across brokers instead of handled by one central component?" Hint: think about load from heartbeats and commits for thousands of groups, and how failover reuses partition leadership.
Partition Assignment Strategies
Theory
When a group changes, someone must decide which member gets which partition. A good assignment is balanced (similar load per member), stable (partitions stay where they were when possible, so caches and local state remain valid), and sometimes co-partitioned (partition N of two topics goes to the same member, which joins need). These goals conflict, so Kafka offers several strategies.
In the classic protocol, the strategy is a client-side class (partition.assignment.strategy) run by one member, the group leader. In the new consumer protocol (topic "Next-Generation Consumer Rebalance Protocol"), the strategy runs on the broker (group.remote.assignor).
Analogy: dividing tables among waiters. "Range" gives each waiter a consecutive block of tables in each room. "Round robin" deals tables one by one like playing cards. "Sticky" tries to leave every waiter with the tables they already had, only moving the minimum needed.
Classic assignors
| Assignor | partition.assignment.strategy value | How it assigns | Rebalance type |
|---|---|---|---|
RangeAssignor | org.apache.kafka.clients.consumer.RangeAssignor | Per topic, consecutive ranges of partitions to members sorted by ID | Eager |
RoundRobinAssignor | ...RoundRobinAssignor | All partitions of all topics dealt one by one | Eager |
StickyAssignor | ...StickyAssignor | Balanced, and keeps previous ownership as much as possible | Eager |
CooperativeStickyAssignor | ...CooperativeStickyAssignor | Same as sticky, but only revokes what must move | Cooperative (topic "Cooperative Incremental Rebalancing") |
The Java default is [RangeAssignor, CooperativeStickyAssignor]: range is used, but the list allows a rolling upgrade to cooperative sticky by removing RangeAssignor later.
Range vs round robin with two topics
topics A and B, 3 partitions each; members C1, C2
RangeAssignor (per topic: 3 partitions / 2 members -> first member gets the extra one)
C1: A0 A1 B0 B1 4 partitions
C2: A2 B2 2 partitions unbalanced, but A-n and B-n stay together (co-partitioning)
RoundRobinAssignor (A0 A1 A2 B0 B1 B2 dealt alternately)
C1: A0 A2 B1 3 partitions
C2: A1 B0 B2 3 partitions balanced, co-partitioning lost
With many topics that have few partitions each, the range assignor's imbalance compounds: the first member gets the extra partition of every topic.
Stickiness after a member leaves
before: C1: P0 P1 C2: P2 P3 C3: P4 P5 C3 leaves
Range / round robin (recomputed from scratch):
C1: P0 P1 P2 C2: P3 P4 P5 P2 and P3 moved needlessly
Sticky / cooperative sticky:
C1: P0 P1 P4 C2: P2 P3 P5 only C3's partitions moved
Rack-aware assignment
Since Kafka 3.5 (KIP-881), the range and sticky assignors can take racks into account: if consumers set client.rack and brokers set broker.rack, partitions are preferably given to members in the same rack as a replica, which cuts cross-zone traffic when combined with follower fetching (Section 5.1).
Example: choosing an assignor
// Classic protocol: cooperative sticky only (after a rolling upgrade, see the next topics)
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
CooperativeStickyAssignor.class.getName());
// New protocol (Kafka 4.0+): server-side assignor
props.put(ConsumerConfig.GROUP_PROTOCOL_CONFIG, "consumer");
props.put(ConsumerConfig.GROUP_REMOTE_ASSIGNOR_CONFIG, "uniform"); // or "range"
Custom assignors are possible (implement ConsumerPartitionAssignor), for example to assign partitions by data center or by known per-partition cost, but they are rarely needed and are not supported by the new protocol's client configuration.
Key Takeaways
- Assignment strategies trade balance, stability, and co-partitioning against each other.
- Range keeps partition N of different topics together but can be unbalanced. Round robin balances but moves many partitions on every change.
- Sticky assignors minimize movement.
CooperativeStickyAssignoradds incremental rebalancing and is the best classic choice for most applications. - With the new consumer protocol, assignment runs on the broker (
uniformorrange).
Practice
- (Beginner) Assign topics A (4 partitions) and B (2 partitions) to members C1, C2, C3 by hand with range and with round robin. Count partitions per member.
- (Intermediate) Run a group of three consumers with
RoundRobinAssignorand three withCooperativeStickyAssignor, each on a 12-partition topic. Stop one member in each group and compare which partitions moved (use--describe --members --verbosebefore and after).- Interview question: "When would you prefer the range assignor over a sticky assignor?" Hint: think about joining two topics by key in a plain consumer, and what co-partitioning requires.
Rebalancing
Theory
A rebalance is the process of redistributing partitions among group members. It is necessary for elasticity and fault tolerance, but it is also the most common source of consumer pain, because in the classic eager protocol the whole group stops consuming while it happens.
A rebalance is triggered when:
| Trigger | Example |
|---|---|
| A member joins | A new instance starts, or a pod is rescheduled |
| A member leaves cleanly | close() sends a leave request |
| A member is declared dead | No heartbeat within session.timeout.ms |
| A member is too slow | No poll() within max.poll.interval.ms |
| The subscription changes | A new topic matches a subscription pattern, or a member changes its topics |
| Partitions are added | kafka-topics.sh --alter --partitions on a subscribed topic (Section 3.3) |
Analogy: a rebalance in the eager protocol is like a teacher reshuffling groups by asking every student to put down their work and stand up, then handing out new seats. Even students who end up in the same seat lost time standing.
The classic (eager) protocol
member C1 (leader) member C2 coordinator
| | |
| (C3 joins -> coordinator starts a rebalance; state PreparingRebalance)
|<-- heartbeat response: REBALANCE_IN_PROGRESS ---|
| onPartitionsRevoked(ALL owned partitions) | <- everyone stops consuming
|--- JoinGroup(subscriptions) ------------------->|
| |--- JoinGroup --------->|
| | | waits for all members (bounded by
|<-- JoinGroup resp: you are leader + member list-| max.poll.interval.ms)
| run assignor locally |
|--- SyncGroup(full assignment) ----------------->| state CompletingRebalance
| |--- SyncGroup --------->|
|<-- SyncGroup resp: your partitions -------------|
| |<-- your partitions ----|
| onPartitionsAssigned(new partitions) | state Stable, generation + 1
| resume consuming |
The pause is called a stop-the-world rebalance. Its length is roughly the time for the slowest member to notice and rejoin, which can be up to a full poll() iteration of processing. With large groups and slow processing, rebalances can take minutes, and if members keep timing out during a rebalance, the group can enter a rebalance storm where it never stabilizes.
Reacting to rebalances: ConsumerRebalanceListener
| Callback | Called when | Typical work |
|---|---|---|
onPartitionsRevoked(partitions) | Before partitions are taken away (clean revocation) | Finish in-flight work, commit offsets synchronously, flush local state |
onPartitionsAssigned(partitions) | After new partitions are received | Load state, seek() to externally stored offsets, warm caches |
onPartitionsLost(partitions) | Partitions were already reassigned (for example, the session expired) | Clean up local state only. Do not commit: another member already owns them. |
Example: commit on revocation
Map<TopicPartition, OffsetAndMetadata> pending = new ConcurrentHashMap<>();
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Commit what we finished before ownership moves, so the next owner does not redo it
Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();
for (TopicPartition tp : partitions) {
OffsetAndMetadata o = pending.remove(tp);
if (o != null) toCommit.put(tp, o);
}
if (!toCommit.isEmpty()) consumer.commitSync(toCommit); // allowed inside the callback
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
log.info("assigned {}", partitions);
}
@Override
public void onPartitionsLost(Collection<TopicPartition> partitions) {
partitions.forEach(pending::remove); // too late to commit
}
});
while (running) {
for (ConsumerRecord<String, String> rec : consumer.poll(Duration.ofMillis(500))) {
process(rec);
pending.put(new TopicPartition(rec.topic(), rec.partition()),
new OffsetAndMetadata(rec.offset() + 1));
}
consumer.commitAsync(Map.copyOf(pending), null);
}
Callbacks run on the poll thread, inside poll(), so they are subject to the same time limits as normal processing.
Reducing rebalance cost
| Technique | Effect | Topic |
|---|---|---|
| Cooperative sticky assignor | Only moved partitions stop | "Cooperative Incremental Rebalancing" |
| Static membership | Restarts within the session timeout cause no rebalance | "Static Group Membership" |
| New consumer protocol | Broker-driven, incremental, no group-wide barrier | "Next-Generation Consumer Rebalance Protocol" |
| Short, bounded processing per poll | Members rejoin quickly | Section 5.1 |
Clean shutdown with close() | Immediate, controlled reassignment | Section 5.1 |
| Avoid broad regex subscriptions on topic-churning clusters | Fewer subscription-change rebalances | This topic |
Key Takeaways
- Rebalances redistribute partitions when members join, leave, fail, stall, or when subscriptions and partition counts change.
- The classic eager protocol revokes all partitions from all members, so the whole group pauses.
- Use
ConsumerRebalanceListenerto commit before revocation and to initialize after assignment. Never commit inonPartitionsLost. - Rebalance storms usually come from slow processing relative to
max.poll.interval.ms, frequent restarts, or flapping members.
Practice
- (Beginner) Add logging in all three listener callbacks. Start two members, then a third, then stop one. Record the callbacks each member sees and in which order.
- (Intermediate) A group of 40 members takes several minutes to rebalance during every deployment and sometimes never settles. List the configuration and code changes you would make, in the order you would try them.
- Interview question: "What is the difference between
onPartitionsRevokedandonPartitionsLost, and why does it matter for offset commits?" Hint: ask whether the member still owns the partitions at the moment the callback runs.
Cooperative Incremental Rebalancing
Theory
In an eager rebalance, every member gives up every partition, even though usually only a few partitions need to move. Cooperative incremental rebalancing (KIP-429, Kafka 2.4) fixes this: members keep the partitions they will still own and keep consuming them during the rebalance. Only the partitions that must move are revoked, and they are handed out in a second, short rebalance.
Analogy: instead of asking all students to stand up, the teacher says "only Alice and Bob, please hand your worksheets to the new student". Everyone else keeps working.
Eager vs cooperative when C3 joins
before: C1: P0 P1 P2 C2: P3 P4 P5 C3 joins
EAGER (one round)
revoke ALL: C1 -> {} C2 -> {} whole group stops
assign: C1: P0 P1 C2: P3 P4 C3: P2 P5 everyone resumes
COOPERATIVE (two rounds)
round 1: assignor computes target; members keep what they will still own
C1: P0 P1 (keeps consuming) revokes P2
C2: P3 P4 (keeps consuming) revokes P5
C3: {}
round 2: (triggered automatically because partitions were revoked)
C3: P2 P5 only P2 and P5 paused briefly
| Aspect | Eager | Cooperative |
|---|---|---|
| Partitions revoked | All, from every member | Only those that move |
| Consumption during rebalance | Stopped for the whole group | Continues for unaffected partitions |
| Rounds | One | Two when partitions move |
onPartitionsRevoked receives | Every owned partition | Only the moved partitions (may be empty, and is then not called) |
| Assignors | Range, RoundRobin, Sticky | CooperativeStickyAssignor (or a custom cooperative assignor) |
Migrating a live group from eager to cooperative
All members of a group must agree on a protocol. To switch without downtime, use two rolling restarts:
step 1: rolling restart with partition.assignment.strategy = [CooperativeStickyAssignor, RangeAssignor]
(still eager, because some members only support Range until all are restarted)
step 2: rolling restart with partition.assignment.strategy = [CooperativeStickyAssignor]
(now every member supports cooperative, so the group uses it)
The default [RangeAssignor, CooperativeStickyAssignor] is designed for this: it already advertises cooperative support, so with the default, only the second step (removing RangeAssignor, or reordering) is needed.
Example: configuring a cooperative consumer
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
CooperativeStickyAssignor.class.getName());
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
// Only the partitions that move. Partitions we keep are not mentioned at all.
commitFor(revoked);
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> added) {
// Only newly added partitions, not the full assignment. Use consumer.assignment() for that.
log.info("added {}, now own {}", added, consumer.assignment());
}
});
Code written for eager rebalancing that assumes onPartitionsAssigned receives the complete assignment (for example, to rebuild all local state) must be changed: under cooperative rebalancing, it only receives the delta.
In Python (librdkafka), set "partition.assignment.strategy": "cooperative-sticky". Inside custom on_assign / on_revoke callbacks, use consumer.incremental_assign() and incremental_unassign() instead of assign() and unassign().
Key Takeaways
- Cooperative rebalancing revokes only the partitions that move. Everything else keeps flowing during the rebalance.
- It may take two rounds, but each round is short and most of the group never pauses.
- Migrate a live group with rolling restarts so that all members support cooperative before it is used.
- Rebalance callbacks receive deltas, not the full assignment. Use
consumer.assignment()for the full set.
Practice
- (Beginner) Run the same three-member experiment with
RangeAssignorand withCooperativeStickyAssignor, logging callbacks. Compare which partitions each member was asked to revoke.- (Intermediate) Plan the migration of a 20-instance production service from
RangeAssignor(explicitly configured) to cooperative sticky. Write down each deployment step and what you would verify between steps.- Interview question: "Why does cooperative rebalancing need two rounds, and why is that still faster in practice than one eager round?" Hint: a partition must be revoked by its old owner before it can safely be given to a new one, and think about how many partitions pause in each approach.
Static Group Membership
Theory
Every time a consumer restarts, it normally gets a new member ID. To the coordinator, a restart looks like "member A left, a new member B joined": two membership changes, and potentially two rebalances, even though the same process came back seconds later with the same work to do. In rolling deployments of large groups, this causes a cascade of rebalances.
Static membership (KIP-345, Kafka 2.3) gives each consumer instance a persistent identity with group.instance.id. A static member:
- does not send a leave request when it closes, and
- when it rejoins with the same
group.instance.idbeforesession.timeout.msexpires, gets its previous assignment back without a rebalance.
If it does not return in time, the coordinator removes it and rebalances as usual.
Analogy: a reserved seat with your name on it. If you step out briefly and come back, nobody reshuffles the room; your seat is waiting. Only if you are gone longer than the reservation lasts is your seat given to someone else.
Dynamic vs static restart
dynamic member, restart takes 20 s
t=0 close() -> leave group -> rebalance #1 (partitions spread to others)
t=20 new member joins -> rebalance #2 (partitions spread back)
static member (group.instance.id=orders-2, session.timeout.ms=120000), restart takes 20 s
t=0 close() -> no leave request; coordinator keeps the slot
t=20 rejoins as orders-2 -> same partitions, no rebalance
(partitions were unread for 20 s: lag grows a little)
Configuration
group.id=order-service
group.instance.id=order-service-2 # unique per instance, stable across restarts
session.timeout.ms=120000 # long enough to cover a restart (classic protocol)
Stable IDs come naturally from the platform: a Kubernetes StatefulSet pod name (order-service-2), a host name, or an instance index from the deployment tool.
| Aspect | Dynamic membership | Static membership |
|---|---|---|
| Identity | New member ID on every start | group.instance.id, stable |
| Clean shutdown | Leave request, immediate rebalance | No leave request; slot kept until session timeout |
| Restart within the session timeout | Up to two rebalances | No rebalance |
| Crash detection | session.timeout.ms (45 s default) | Same mechanism, usually with a longer timeout |
| Duplicate IDs | Not possible | Second instance fences the first (FencedInstanceIdException) |
Trade-offs
- Slower failure handling: a long session timeout also delays reassignment when an instance really dies. Choose a timeout that covers your normal restart time but not much more.
- Unique IDs are mandatory: if two running instances share a
group.instance.id, the newer one fences the older, which fails withFencedInstanceIdException. - Scaling down: a removed instance keeps its partitions unread until the session times out. Remove it explicitly with the Admin API (
removeMembersFromConsumerGroup) to reassign immediately.
// Removing a static member that will not come back (e.g., after scaling down)
admin.removeMembersFromConsumerGroup("order-service",
new RemoveMembersFromConsumerGroupOptions(
List.of(new MemberToRemove("order-service-5")))).all().get();
Static membership is also supported by the new consumer protocol (Kafka 4.0+), where the session timeout is set on the broker (group.consumer.session.timeout.ms) or per group (Section 3.2).
Key Takeaways
group.instance.idgives a consumer a stable identity. Restarts within the session timeout cause no rebalance.- Static members do not leave the group on
close(), so their partitions wait for them rather than moving. - A longer session timeout trades faster failure detection for fewer rebalances. Size it to your restart time.
- IDs must be unique per running instance. Remove permanently retired members explicitly.
Practice
- (Beginner) Run three consumers with
group.instance.idset andsession.timeout.ms=60000. Restart one and confirm with--describe --membersthat the assignment did not change. Repeat withoutgroup.instance.id.- (Intermediate) Start two instances with the same
group.instance.id. Describe what each instance logs, and explain why Kafka fences rather than allowing both.- Interview question: "Static membership reduces rebalances. Why not always set a very long session timeout?" Hint: consider what happens to a partition's records when its owner crashes and never returns.
Next-Generation Consumer Rebalance Protocol
Theory
The classic protocol (and its cooperative extension) has structural limits. Assignment runs on one consumer, the group leader, which must receive every member's subscription. Every rebalance is a group-wide barrier: all members must rejoin before anyone gets a new assignment, so one slow member delays everyone. And the logic is complex and duplicated in every client library.
KIP-848, generally available in Kafka 4.0, replaces it with a broker-driven, fully incremental protocol:
- The coordinator computes assignments (server-side assignors
uniformandrange). - Members talk to the coordinator with a single, periodic
ConsumerGroupHeartbeatrequest, which carries their subscription and current partitions and receives their target assignment. - There is no global barrier. Each member reconciles toward its target on its own: it releases partitions it must give up, and receives new partitions as soon as their previous owners have released them.
Analogy: under the classic protocol, a team lead (one of the workers) must call a meeting with everyone before redistributing tasks. Under the new protocol, a manager in the office keeps a target plan and tells each worker individually, at their next check-in, what to hand over and what to pick up. Nobody waits for a meeting.
Epochs instead of generations
| Concept | Meaning |
|---|---|
| Group epoch | Incremented when membership or subscriptions change |
| Target assignment epoch | The group epoch for which the coordinator has computed a target assignment |
| Member epoch | The epoch a member has reconciled up to. Commits carry it; stale members are fenced. |
C3 joins: group epoch 11 -> 12, coordinator computes target assignment for epoch 12
C1 (owns P0 P1 P2, target P0 P1) heartbeat -> "revoke P2" -> releases P2, acks
C2 (owns P3 P4 P5, target P3 P4) heartbeat -> "revoke P5" -> releases P5, acks
C3 (owns nothing, target P2 P5) heartbeat -> "P2 is yours" (once C1 released it)
heartbeat -> "P5 is yours" (once C2 released it)
C1 and C2 never stop consuming P0 P1 P3 P4; C3 receives each partition independently.
Group states: Empty / Assigning / Reconciling / Stable / Dead
Classic vs new protocol
| Aspect | Classic (eager or cooperative) | New consumer protocol (KIP-848) |
|---|---|---|
| Who assigns | Group leader (a consumer) | Group coordinator (broker) |
| RPCs | JoinGroup, SyncGroup, Heartbeat, LeaveGroup | ConsumerGroupHeartbeat |
| Barrier | All members must rejoin | None: per-member reconciliation |
| Assignor config | Client: partition.assignment.strategy | Client: group.remote.assignor; broker: group.consumer.assignors |
| Session timeout, heartbeat interval | Client: session.timeout.ms, heartbeat.interval.ms | Broker or group config: group.consumer.session.timeout.ms (45 s), group.consumer.heartbeat.interval.ms (5 s) |
| Effect of one slow member | Delays the whole group | Delays only partitions it must release |
| Client-side regex subscription | Client resolves the pattern | Broker resolves the pattern (RE2J syntax) |
Enabling it
On Kafka 4.0+ brokers the protocol is enabled by default. Clients opt in:
group.id=order-service
group.protocol=consumer # default is "classic"
group.remote.assignor=uniform # optional; "range" is the other built-in choice
# session.timeout.ms, heartbeat.interval.ms and partition.assignment.strategy are not used
# Recent librdkafka-based clients (confluent-kafka-python 2.x) also support the new protocol
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "order-service",
"group.protocol": "consumer",
})
Migrating an existing group
A classic group can be upgraded online. When the first member joins with group.protocol=consumer, the coordinator converts the group, and remaining classic members keep working through a compatibility layer until they are restarted. The broker setting group.consumer.migration.policy (default bidirectional) controls whether conversion is allowed in each direction. In practice: do a rolling restart that sets group.protocol=consumer on every instance, then remove client settings the new protocol ignores.
# The group type and protocol show up in the describe output
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service --state
# GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
# order-service broker-2:9092 (2) uniform Stable 3
# Per-group overrides of broker defaults
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups \
--entity-name order-service --add-config consumer.session.timeout.ms=60000
Key Takeaways
- KIP-848 (GA in Kafka 4.0) moves assignment to the broker and replaces JoinGroup/SyncGroup with one heartbeat RPC.
- Rebalances are fully incremental with no group-wide barrier, so one slow member no longer stalls the group.
- Clients opt in with
group.protocol=consumer. Session timeout, heartbeat interval, and assignor are now broker or group settings. - Existing classic groups can be migrated online with a rolling restart.
Practice
- (Beginner) On a Kafka 4.x cluster, run three consumers with
group.protocol=consumer. Describe the group and note its state and assignment strategy. Add a fourth member and watch the states it passes through.- (Intermediate) Migrate a running three-member classic group to the new protocol by restarting one member at a time. After each restart, describe the group and explain what you see.
- Interview question: "What problems in the classic rebalance protocol does KIP-848 solve, and what moves from the client to the broker?" Hint: think about the group-wide barrier, where the assignor runs, and which settings a client no longer controls.
<a id="53-offset-management"></a>
5.3 Offset Management
A consumer group's committed offsets are its memory: they decide where every partition resumes after a restart or rebalance. This section covers where offsets are stored, the commit styles and their failure modes, where a group starts when it has no valid offset, and how to move offsets on purpose from code and from operations tooling.
The __consumer_offsets Topic
Theory
Committing an offset means saving "group G has processed partition P up to offset N" somewhere durable. Early Kafka versions stored this in ZooKeeper, which could not cope with thousands of consumers committing every few seconds. Since Kafka 0.9, offsets are stored in Kafka itself, in the internal, compacted topic __consumer_offsets (Section 3.3). A commit is simply a record appended to that topic by the group coordinator. Kafka's own replication makes it durable, and compaction keeps only the latest offset per group and partition.
Analogy:
__consumer_offsetsis a sign-in sheet that only grows. Each line says "group G, partition P, read up to N". Once in a while, the office photocopies only the most recent line per group and partition and throws away the old sheets. That is log compaction.
What is stored
| Record type | Key | Value |
|---|---|---|
| Offset commit | (group, topic, partition) | offset (next to read), leaderEpoch, metadata string, commitTimestamp |
| Group metadata | group | Protocol type, generation or epoch, leader, members and their assignments |
| Tombstone | Same key, null value | Written when offsets expire or a group is deleted; compaction then removes the key |
The commit path
consumer group coordinator (leader of __consumer_offsets-9)
| |
|--- OffsetCommit(group=billing, -----> | 1. validate generation / member epoch
| orders-0 -> 18234, | 2. append records to __consumer_offsets-9
| orders-1 -> 17990) | 3. wait for replication to in-sync followers
| | 4. update the in-memory offset cache
|<-- OK (or error per partition) -------|
|
later, after a restart or rebalance:
|--- OffsetFetch(group=billing) ------> | served from the in-memory cache
|<-- orders-0: 18234, orders-1: 17990 --|
When the coordinator moves to another broker, the new leader of the partition replays it to rebuild the cache before serving the group (Section 5.2).
Retention of committed offsets
Offsets do not live forever. Since Kafka 2.1 (KIP-211), the offsets of a group are kept for offsets.retention.minutes (default 10080, 7 days) after the group becomes empty. While the group has active members, its offsets do not expire.
| Situation | Outcome |
|---|---|
| Group active, commits regularly | Offsets kept indefinitely |
| Group stopped for 3 days | Offsets kept; consumers resume where they stopped |
| Group stopped for 8 days | Offsets deleted; on restart auto.offset.reset decides where to start (topic "auto.offset.reset") |
| Committed offset older than the topic's log start offset (data deleted by retention) | Offset is invalid; auto.offset.reset decides |
A seasonal batch job that runs once a month, with the default 7-day offset retention and auto.offset.reset=latest, will silently skip everything written since its last run. Either raise offsets.retention.minutes on the brokers or use earliest with idempotent processing.
The metadata string
A commit can carry a free-form metadata string (up to offset.metadata.max.bytes, 4096 by default). It is returned with the offset and is occasionally useful for audit information, such as which host or application version committed:
consumer.commitSync(Map.of(
new TopicPartition("orders", 0),
new OffsetAndMetadata(18234L, "host=worker-3;version=2.4.1")));
OffsetAndMetadata c = consumer.committed(Set.of(new TopicPartition("orders", 0)))
.get(new TopicPartition("orders", 0));
System.out.println(c.offset() + " " + c.metadata()); // 18234 host=worker-3;version=2.4.1
Example: inspecting raw commits
# Normal tool: committed offsets and lag
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
# Deep debugging: decode commits as they are written
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic __consumer_offsets \
--formatter org.apache.kafka.tools.consumer.OffsetsMessageFormatter \
--consumer-property exclude.internal.topics=false | grep '"group":"billing"'
Watching the raw stream shows how often a group commits, which is a quick way to confirm whether auto commit or manual commits are actually happening.
Key Takeaways
- Committed offsets are records in the compacted internal topic
__consumer_offsets, keyed by(group, topic, partition). - The group coordinator appends commits, waits for replication, and serves offset fetches from an in-memory cache.
- Offsets of an empty group expire after
offsets.retention.minutes(7 days by default). Infrequent jobs need a longer retention or a safeauto.offset.reset. - A commit can carry a small metadata string, returned with the offset.
Practice
- (Beginner) Run a consumer in group
lab-group, stop it, and usekafka-consumer-groups.sh --describeto read its committed offsets. Then decode the raw commits forlab-groupfrom__consumer_offsets.- (Intermediate) A reporting job runs on the first day of each month. Explain what happens to its offsets between runs with the default broker configuration, and propose two different fixes with their trade-offs.
- Interview question: "Why does
__consumer_offsetsuse log compaction rather than time-based deletion?" Hint: think about what a consumer needs after a restart, and how many commits a busy group writes per day.
Auto Commit
Theory
The simplest way to commit is not to think about it. With enable.auto.commit=true (the Java default), the consumer commits automatically every auto.commit.interval.ms (5 seconds by default). What it commits is the important detail: the position after the records returned by the previous poll(), and the commit happens inside the next poll() (and in close() and before a rebalance).
This makes auto commit safe for one specific shape of code: you finish processing every record from one poll() before calling poll() again. Then anything committed has been processed, and the worst case after a crash is reprocessing up to 5 seconds of records (at-least-once). If processing continues after poll() returns, for example in another thread, auto commit can commit records that were never processed.
Analogy: auto commit is a diary that a friend fills in for you every five minutes, writing "finished up to page N" where N is the page you were handed last time. If you always read every page before asking for more, the diary is accurate or slightly behind. If you hand pages to someone else to read later, the diary can claim pages nobody has read.
When auto commit is safe and when it is not
SAFE: synchronous processing in the poll loop
poll() -> records 0..499 process 0..499 poll() -> [auto commit 500] -> records 500..999
crash while processing 500..999 -> restart at 500, records 500.. reprocessed (duplicates)
UNSAFE: processing handed to another thread
poll() -> records 0..499 queue them to a worker pool, return immediately
poll() -> [auto commit 500] ... workers have only finished 0..120
crash -> restart at 500, records 121..499 never processed (data loss)
| Processing style | Auto commit outcome |
|---|---|
Every record processed before the next poll() | At-least-once; duplicates up to one interval after a crash |
| Records handed to other threads, queues, or async calls | Possible data loss |
Batches accumulated across several poll() calls before a write (for example, to a data warehouse) | Possible data loss if the process crashes before the write |
The librdkafka difference
In librdkafka-based clients (Python, Go, .NET, C/C++), auto commit runs on a background thread on a timer, and the offset of a message is stored as "ready to commit" as soon as poll() returns it (enable.auto.offset.store=true). The commit can therefore happen while the message is still being processed. The safe pattern keeps auto commit but stores offsets only after processing:
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "order-service",
"enable.auto.commit": True, # background commit every auto.commit.interval.ms
"enable.auto.offset.store": False, # but only of offsets we explicitly store
})
consumer.subscribe(["orders"])
while running:
msg = consumer.poll(1.0)
if msg is None or msg.error():
continue
process(msg)
consumer.store_offsets(message=msg) # mark as done; committed by the next background commit
This gives at-least-once behavior with the efficiency of periodic commits.
Example: auto commit in Java
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true"); // default
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000"); // smaller window of duplicates
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> rec : records) {
process(rec); // synchronous: finished before the next poll()
}
} // close() performs a final commit
Key Takeaways
- Java auto commit commits the position from the previous
poll(), during the nextpoll(), everyauto.commit.interval.ms. - It is at-least-once only if all records are processed before the next
poll(). Asynchronous processing with auto commit can lose data. - In
librdkafkaclients, setenable.auto.offset.store=falseand callstore_offsets()after processing. - A shorter interval reduces duplicates after a crash at the cost of more commit traffic.
Practice
- (Beginner) Consume with auto commit and a 10-second interval, print each offset, and kill the process with
kill -9a few seconds after it starts printing. Restart it and count the duplicate offsets.- (Intermediate) Modify the consumer to hand each record to an
ExecutorServiceand return immediately. Introduce a 500 ms delay per record, kill the process, and show that some records are never processed.- Interview question: "Is Kafka's auto commit at-most-once or at-least-once?" Hint: the answer depends on where processing happens relative to the next
poll()call, and on the client library.
Manual Synchronous Commit
Theory
Manual commits put the application in charge of the one decision that matters: when is a record done? With enable.auto.commit=false, nothing is committed until you call a commit method. commitSync() sends the commit and blocks until the coordinator confirms it or a non-retriable error occurs. It retries retriable errors itself (for example, a coordinator move) until default.api.timeout.ms (60 seconds) passes.
The guarantee is strong: when commitSync() returns, the offset is durably stored. The cost is latency: each call is a round trip plus replication of __consumer_offsets, during which the poll thread does nothing.
Analogy: a synchronous commit is a registered letter. You wait at the counter until you get the stamped receipt. It is certain, but you cannot do anything else while you wait.
Forms of commitSync
| Call | Commits |
|---|---|
commitSync() | The current position of every assigned partition (everything returned by the last poll()) |
commitSync(Map<TopicPartition, OffsetAndMetadata>) | Exactly the given offsets (remember: next offset to read = last processed + 1) |
commitSync(Duration) / commitSync(Map, Duration) | Same, with an explicit timeout |
How often to commit
| Frequency | Duplicates after a crash | Cost |
|---|---|---|
| After every record | At most one record | One blocking round trip per record: throughput drops sharply |
After every poll() batch | At most one batch (max.poll.records) | Usually a good balance |
| Every N records or T seconds | At most N records or T seconds of work | Fewer commits, larger replay window |
Example: commit per batch, at-least-once
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("payments"));
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
if (records.isEmpty()) continue;
for (ConsumerRecord<String, String> rec : records) {
ledger.apply(rec); // side effect first...
}
try {
consumer.commitSync(); // ...then record progress
} catch (CommitFailedException e) {
// A rebalance took our partitions; the new owner will reprocess from the last commit.
log.warn("commit failed after rebalance", e);
}
}
}
Errors to know
| Exception | Meaning | Action |
|---|---|---|
CommitFailedException | The member was removed from the group (for example, max.poll.interval.ms exceeded) | Records since the last commit will be reprocessed by another member. Fix the slow processing. |
RebalanceInProgressException | A rebalance started; the commit was not applied | Call poll() to complete the rebalance; commit in onPartitionsRevoked to avoid this |
TimeoutException | No success within the timeout | The commit may or may not have been applied; retry or let the next commit supersede it |
WakeupException, InterruptException | The call was interrupted | Shut down (Section 5.1) |
Key Takeaways
- Disable auto commit and call
commitSync()after processing to get precise at-least-once behavior. commitSync()blocks and retries until success, a non-retriable error, or the timeout.- Commit per batch, not per record, unless each record is very expensive to reprocess.
CommitFailedExceptionmeans the group rebalanced without you. Processing was correct, but some records will be reprocessed.
Practice
- (Beginner) Convert an auto-commit consumer to manual
commitSync()after each batch. Measure throughput with commit per record and commit per batch on 100,000 records.- (Intermediate) Write a consumer that commits every 1,000 records or every 2 seconds, whichever comes first, using
commitSync(Map)with exact offsets per partition.- Interview question: "Why should you commit after processing rather than before, and what kind of delivery guarantee does each order give?" Hint: imagine a crash between the two steps in each order.
Manual Asynchronous Commit
Theory
commitAsync() sends the commit and returns immediately, calling an optional callback when the result arrives. The poll thread keeps working, so throughput is close to that of auto commit, while you still control exactly which offsets are committed.
The catch is retries. commitAsync() does not retry failed commits, deliberately. Imagine commit A (offset 2000) fails with a transient error, then commit B (offset 3000) succeeds. If A were retried and succeeded now, the committed offset would move back to 2000, and 1000 records would be reprocessed after a crash. Because a later commit always supersedes an earlier one, the next successful commit "fixes" a failed one anyway.
Analogy: async commits are like sending progress updates by text message without waiting for a reply. If one message is lost, it does not matter much, because the next message reports newer progress. Resending an old message late would only confuse the recipient.
The standard combined pattern
while running: poll -> process -> commitAsync() cheap, a lost commit is superseded
on revoke: commitSync(offsets of revoked parts) ownership moves: must not be lost
on shutdown: commitSync() last chance: must not be lost
Example: async in the loop, sync at the edges
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
@Override public void onPartitionsRevoked(Collection<TopicPartition> parts) {
consumer.commitSync(currentOffsets(parts)); // blocking: before ownership moves
}
@Override public void onPartitionsAssigned(Collection<TopicPartition> parts) { }
});
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
records.forEach(this::process);
consumer.commitAsync((offsets, exception) -> { // runs on the poll thread, in a later poll()
if (exception != null) {
log.warn("async commit failed for {}; next commit will supersede", offsets, exception);
}
});
}
} catch (WakeupException e) {
// shutting down
} finally {
try {
consumer.commitSync(); // final, blocking commit
} finally {
consumer.close();
}
}
Retrying safely, if you must
If you want to retry async commits, guard against going backwards with a monotonically increasing sequence number:
private final AtomicLong commitSeq = new AtomicLong();
void commitWithRetry(Map<TopicPartition, OffsetAndMetadata> offsets) {
long seq = commitSeq.incrementAndGet();
consumer.commitAsync(offsets, (o, e) -> {
if (e instanceof RetriableException && seq == commitSeq.get()) {
commitWithRetry(o); // retry only if no newer commit was issued since
}
});
}
Sync vs async
| Aspect | commitSync() | commitAsync() |
|---|---|---|
| Blocks the poll thread | Yes, for a round trip | No |
| Retries | Yes, until timeout | No (by design) |
| Result | Return or exception | Callback, executed on the poll thread during a later poll() or commit |
| Throughput impact | Noticeable when frequent | Minimal |
| Best used | On revocation, on shutdown, when certainty is required | In the main loop |
In Python, consumer.commit(asynchronous=True) is the asynchronous form; results are delivered to the on_commit callback configured on the consumer, served by poll().
Key Takeaways
commitAsync()does not block and does not retry, because a retried old commit could overwrite a newer one.- Use async commits in the poll loop and sync commits on revocation and shutdown.
- Async commit callbacks run on the poll thread. Keep them short.
- If you add retries, use a sequence number so you never retry a superseded commit.
Practice
- (Beginner) Replace
commitSync()withcommitAsync()in a batch consumer and compare throughput. Add a callback that counts failed commits.- (Intermediate) Construct a timeline in which retrying a failed
commitAsync()causes 1,000 records to be reprocessed after a crash. Then show how the sequence-number guard prevents it.- Interview question: "Why doesn't
commitAsync()retry on failure whilecommitSync()does?" Hint: think about two commits in flight at the same time and the order in which they could be applied.
auto.offset.reset
Theory
A consumer group resumes from its committed offsets. But sometimes there is no valid committed offset to resume from:
- the group is new and has never committed,
- the group's offsets expired (
offsets.retention.minutes, topic "The __consumer_offsets Topic"), or - the committed offset is out of range, typically because retention deleted the data it pointed to.
auto.offset.reset decides what to do in these cases. It is not used when a valid committed offset exists, a common misunderstanding: changing it from latest to earliest on an existing group does nothing until one of the cases above happens.
Analogy: when you lose your bookmark, you either start the book from the first page that is still in it (
earliest), skip to the newest chapter (latest), or stop and ask someone what to do (none).
Values
| Value | Behavior | Risk |
|---|---|---|
latest (default) | Start at the end: only records written from now on | Silent data skip for new groups and expired offsets |
earliest | Start at the log start offset: all retained history | Large replay (possibly days of data) on first start or after expiry |
none | Throw NoOffsetForPartitionException | Requires explicit handling, but nothing happens silently |
by_duration:<ISO-8601> (Kafka 4.0+, KIP-1106) | Start at the first offset with a timestamp within the duration, for example by_duration:P1D | Same as a time-based seek; requires a 4.0+ client |
Decision flow on partition assignment
partition assigned
|
does the group have a committed offset?
| |
yes no
| |
is it within [log start, high watermark]? |
| | |
yes no |
| | |
resume at committed +-----+----------------+
|
auto.offset.reset
earliest -> log start offset
latest -> high watermark
none -> NoOffsetForPartitionException
Choosing
| Use case | Recommended value | Why |
|---|---|---|
| Business processing where every event matters (orders, payments) | earliest | Never skip; make processing idempotent to tolerate replays |
| Live dashboards, monitoring, notifications where old events are useless | latest | Old events would be stale or harmful (for example, sending week-old alerts) |
| Critical pipelines where any reset needs human judgment | none | Fail loudly and let an operator choose |
| A new consumer that should backfill a bounded period | by_duration:PT6H or none + seek by timestamp | Avoids both full replay and data skip |
Example: handling none
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
consumer.subscribe(List.of("payments"));
try {
consumer.poll(Duration.ofSeconds(1));
} catch (NoOffsetForPartitionException e) {
// Decide explicitly, here: start from 24 hours ago for the affected partitions
long since = Instant.now().minus(Duration.ofHours(24)).toEpochMilli();
Map<TopicPartition, Long> query = new HashMap<>();
e.partitions().forEach(tp -> query.put(tp, since));
consumer.offsetsForTimes(query).forEach((tp, found) -> {
if (found != null) consumer.seek(tp, found.offset());
else consumer.seekToEnd(List.of(tp)); // nothing that recent
});
alerting.notify("payments consumer reset offsets for " + e.partitions());
}
Key Takeaways
auto.offset.resetapplies only when there is no valid committed offset: a new group, expired offsets, or an out-of-range offset.latest(the default) can silently skip data.earliestreplays everything retained.nonemakes the application decide.- Kafka 4.0 adds
by_duration:<duration>for time-bounded starting points. - Changing the setting on a running group with valid offsets has no effect.
Practice
- (Beginner) Produce 10 records, then start a new group with
latestand another withearliest. Produce 5 more and record what each group reads.- (Intermediate) Set
retention.ms=60000on a test topic, stop a consumer group, wait until its committed offset is below the log start offset, and restart it with each of the three classic values. Record the consumer's log messages and behavior.- Interview question: "A team changed
auto.offset.resetfromlatesttoearliestto reprocess data, but nothing happened. Why, and what should they have done?" Hint: when is the setting consulted, and which tool moves committed offsets?
Seeking to Specific Offsets
Theory
The consumer's position is just a number per partition, so the application can change it at any time with seek() (Section 2.1 introduced the basics). Seeking is how consumers implement replay, skip poison records, retry a failed record, or resume from offsets stored outside Kafka. A seek changes only this consumer's in-memory position. It becomes permanent only when the new position is committed.
Analogy: seeking is moving your finger to a different line on the page. The bookmark (the committed offset) stays where it was until you deliberately move it too.
The seek methods
| Method | Moves the position to | Notes |
|---|---|---|
seek(tp, offset) | An exact offset | Applied immediately; the next poll() fetches from there |
seek(tp, OffsetAndMetadata) | An offset with a leader epoch | Lets the consumer detect log truncation after leader changes |
seekToBeginning(partitions) | The log start offset | Evaluated lazily, at the next poll() or position() |
seekToEnd(partitions) | The high watermark (or last stable offset with read_committed) | Evaluated lazily |
offsetsForTimes(map) + seek | The first offset with timestamp at or after a time | Returns null for partitions with no such record |
You can only seek on partitions currently assigned to the consumer. With subscribe(), that means inside onPartitionsAssigned or after a poll() has returned an assignment; otherwise you get IllegalStateException: No current assignment for partition.
Pattern 1: replay from a time on startup
Instant replayFrom = Instant.parse("2026-10-22T08:00:00Z"); // e.g., from a command-line flag
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
private boolean done = false;
@Override public void onPartitionsRevoked(Collection<TopicPartition> parts) { }
@Override public void onPartitionsAssigned(Collection<TopicPartition> parts) {
if (done) return; // only on the first assignment
Map<TopicPartition, Long> query = new HashMap<>();
parts.forEach(tp -> query.put(tp, replayFrom.toEpochMilli()));
consumer.offsetsForTimes(query).forEach((tp, found) -> {
if (found != null) consumer.seek(tp, found.offset());
else consumer.seekToEnd(List.of(tp));
});
done = true;
}
});
Pattern 2: retry a failed record without skipping it
for (TopicPartition tp : records.partitions()) {
for (ConsumerRecord<String, String> rec : records.records(tp)) {
try {
process(rec);
} catch (TransientException e) {
consumer.seek(tp, rec.offset()); // next poll() returns this record again
consumer.pause(List.of(tp)); // back off this partition only
scheduler.schedule(() -> resumeQueue.add(tp), 5, TimeUnit.SECONDS);
break; // skip the rest of this partition's batch
}
}
}
// In the poll loop: drain resumeQueue and call consumer.resume(...) on the poll thread
Pattern 3: offsets stored outside Kafka
When results go to a database, storing the offset in the same database transaction as the results makes processing effectively exactly-once (Section 5.4). Kafka's committed offsets are then unused, and the consumer seeks to the database's offsets on every assignment:
@Override
public void onPartitionsAssigned(Collection<TopicPartition> parts) {
for (TopicPartition tp : parts) {
Long next = db.queryForLong(
"SELECT next_offset FROM consumer_progress WHERE topic = ? AND part = ?",
tp.topic(), tp.partition());
if (next != null) consumer.seek(tp, next);
else consumer.seekToBeginning(List.of(tp));
}
}
Inspecting positions
TopicPartition tp = new TopicPartition("orders", 0);
long position = consumer.position(tp); // next offset to fetch
Long committed = Optional.ofNullable(consumer.committed(Set.of(tp)).get(tp))
.map(OffsetAndMetadata::offset).orElse(null); // group's bookmark
long end = consumer.endOffsets(List.of(tp)).get(tp); // high watermark
System.out.printf("position=%d committed=%s lag=%d%n", position, committed, end - position);
Key Takeaways
seek()changes only this consumer's in-memory position. Commit afterwards to make it stick.- You can only seek on assigned partitions. With
subscribe(), seek inonPartitionsAssignedor after assignment. - Seeking enables time-based replay, retrying a record without skipping it (combined with
pause()), and resuming from offsets stored in an external system. seekToBeginningandseekToEndare lazy; they take effect on the nextpoll()orposition().
Practice
- (Beginner) Write a consumer that, on startup, seeks every assigned partition to 100 records before its end and prints them.
- (Intermediate) Implement Pattern 2 completely, including the resume queue drained on the poll thread. Verify with a processing function that fails the first two attempts for every tenth record.
- Interview question: "If a consumer seeks backwards but never commits, what happens after it restarts?" Hint: separate the in-memory position from the committed offset, and consider what auto commit would do.
Resetting Consumer Group Offsets
Theory
Seeking changes one consumer's position from inside the application. Sometimes operators need to change a whole group's committed offsets from outside: reprocess data after a bug fix, skip a flood of bad records, move a new deployment to the latest data, or recover after a mistaken deployment. Kafka supports this with kafka-consumer-groups.sh --reset-offsets (covered in detail in Section 3.2) and with the Admin API.
The essential rule: a group's offsets can only be changed while the group has no active members. If consumers were running, they would immediately overwrite the new offsets with their own commits, and their in-memory positions would not change anyway.
Analogy: resetting group offsets is like changing everyone's bookmarks in a reading club. It only works while the club is not meeting; otherwise members keep reading from where their fingers are and write their old pages back over your changes.
Choosing the reset method
| Goal | Method |
|---|---|
| Reprocess the last 2 hours after a bug fix | --by-duration PT2H or --to-datetime |
| Skip everything up to now (bad records flooded in) | --to-latest |
| Skip exactly one poison record on partition 3 | --topic orders:3 --shift-by 1, or --to-offset (prefer handling it in code, Section 5.1) |
| Rebuild a downstream store from scratch | --to-earliest |
| Copy one group's progress to a new group | --from-file with an exported CSV, or the Admin API |
| Undo a reset | --from-file with the CSV exported before the reset |
A safe procedure
1. Check downstream impact: is processing idempotent (Section 5.4)? Will reprocessing double-count?
2. Stop every member of the group; confirm state Empty (--describe --state)
3. Export current offsets (--reset-offsets --to-current --export > before.csv)
4. Dry run the reset (default: prints NEW-OFFSET per partition)
5. Execute (--execute)
6. Verify (--describe: CURRENT-OFFSET and LAG)
7. Restart the application and watch lag decrease
Example: the Admin API
The Admin API does the same thing from code, which is useful in deployment tooling and runbooks:
try (Admin admin = Admin.create(Map.of("bootstrap.servers", "localhost:9092"))) {
String group = "order-service";
// 1. Current committed offsets
Map<TopicPartition, OffsetAndMetadata> current =
admin.listConsumerGroupOffsets(group).partitionsToOffsetAndMetadata().get();
// 2. Offsets as of 2 hours ago, for the same partitions
long ts = Instant.now().minus(Duration.ofHours(2)).toEpochMilli();
Map<TopicPartition, OffsetSpec> query = new HashMap<>();
current.keySet().forEach(tp -> query.put(tp, OffsetSpec.forTimestamp(ts)));
Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> found =
admin.listOffsets(query).all().get();
// 3. Apply (fails if the group still has active members)
Map<TopicPartition, OffsetAndMetadata> target = new HashMap<>();
found.forEach((tp, info) -> {
if (info.offset() >= 0) target.put(tp, new OffsetAndMetadata(info.offset())); // -1: no record that new
});
admin.alterConsumerGroupOffsets(group, target).all().get();
}
Other Admin calls for group offsets: deleteConsumerGroupOffsets(group, partitions) removes a group's offsets for specific partitions (for example, after it stops reading a topic), and deleteConsumerGroups(groups) removes empty groups entirely.
Special cases
| Case | Note |
|---|---|
| Kafka Streams applications | Use kafka-streams-application-reset.sh, which also handles internal topics and local state (Chapter 10) |
| Kafka Connect sink connectors | Offsets belong to the group connect-<connector name>; stop the connector first, or use Connect's offsets REST API (Chapter 9) |
| Groups using externally stored offsets | Kafka's committed offsets are ignored; change them in the external store |
| Share groups | Reset with kafka-share-groups.sh --reset-offsets (Section 5.4) |
Key Takeaways
- Group offset resets change committed offsets for every partition of a group, from outside the application.
- The group must be empty: stop all members first, or the reset is refused.
- Always export current offsets before resetting, dry-run first, and check that downstream processing tolerates reprocessing.
- The Admin API (
listConsumerGroupOffsets,listOffsets,alterConsumerGroupOffsets) automates the same procedure.
Practice
- (Beginner) Reset a test group to
--to-earliestwith a dry run, then execute it, then undo it with the exported CSV.- (Intermediate) Write a small Admin API program that copies the committed offsets of group
orders-v1to a new grouporders-v2, so a new deployment can start exactly where the old one stopped.- Interview question: "A bug corrupted records written to a database between 10:00 and 10:30. How would you reprocess just that window with a consumer group, and what risks would you check first?" Hint: think about time-based offset lookup, stopping at the end of the window, idempotent writes, and records outside the window that would also be replayed.
<a id="54-consumer-patterns-and-semantics"></a>
5.4 Consumer Patterns and Semantics
The order in which a consumer processes records and commits offsets determines what happens after a crash: records are lost, processed twice, or processed effectively once. This section covers the delivery semantics you can build, how to make duplicates harmless, how to measure whether consumers keep up, how to process in parallel beyond one thread per partition, and the queue-style share groups added in Kafka 4.x.
At-Most-Once Processing
Theory
Every consumer does two things per record: a side effect (write to a database, call an API, send an email) and a commit (record progress). A crash can happen between them, and the order of the two steps decides what the crash costs.
At-most-once commits first, then processes. If the process crashes after committing but before finishing, the record is never processed: it is lost, but never duplicated. Each record is processed zero or one times.
Analogy: crossing items off a to-do list before doing them. If you are interrupted, the crossed-off item is never done, but you never do anything twice.
Timeline
poll() -> records 100..199
commitSync() -> committed offset 200 (progress recorded first)
process 100, 101, ... 150 CRASH
restart: resume at 200 -> records 151..199 are never processed (lost)
When it is acceptable
| Use case | Why loss is acceptable |
|---|---|
| High-volume metrics or telemetry sampling | A few missing points do not change aggregates |
| Real-time UI updates, presence, typing indicators | Old updates are useless anyway; the next one replaces them |
| Cache invalidation hints with periodic full refresh | The refresh repairs missed hints |
| Systems where a duplicate is worse than a gap (for example, sending a push notification twice is unacceptable, missing one is tolerable) | The business prefers a gap |
Example: at-most-once in Java
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
if (records.isEmpty()) continue;
consumer.commitSync(); // 1. commit the whole batch first
for (ConsumerRecord<String, String> rec : records) {
try {
sendPushNotification(rec); // 2. then process; failures are not retried
} catch (Exception e) {
log.warn("dropped notification at {}-{}@{}", rec.topic(), rec.partition(), rec.offset(), e);
}
}
}
Accidental at-most-once
Teams often get at-most-once without choosing it:
| Cause | How records are lost |
|---|---|
| Auto commit + asynchronous processing | Offsets committed before worker threads finish (Section 5.3) |
librdkafka auto commit with automatic offset store | Offsets stored when poll() returns, committed in the background before processing ends (Section 5.3) |
| Catching an exception, logging it, and moving on | The record is committed as if processed |
auto.offset.reset=latest on new groups or after offset expiry | Records written before the start are skipped (Section 5.3) |
Producer side with acks=0 or low durability | Records never reach the topic at all (Section 4.3) |
Key Takeaways
- At-most-once means commit, then process: a crash loses records but never duplicates them.
- It is appropriate only where loss is cheaper than duplication, such as sampled metrics or ephemeral UI events.
- Many systems are at-most-once by accident: auto commit with async processing, swallowed exceptions, or
latestresets.
Practice
- (Beginner) Implement the at-most-once consumer above, kill it mid-batch with
kill -9, restart it, and identify exactly which offsets were never processed.- (Intermediate) Review a consumer you have written (or the examples in Section 5.1) and list every path through which a record could be lost.
- Interview question: "Give an example where at-most-once is the correct choice, and explain why at-least-once would be worse there." Hint: think about side effects that cannot be undone or deduplicated and whose value expires quickly.
At-Least-Once Processing
Theory
At-least-once processes first, then commits. A crash between the two means the record is processed again after restart: never lost, but possibly duplicated. This is the default goal of most Kafka applications, because duplicates can be made harmless (topic "Idempotent Consumers") while lost data usually cannot be recovered.
Analogy: crossing items off a to-do list after doing them. If you are interrupted, you might redo the last item, but nothing is ever forgotten.
Timeline
poll() -> records 100..199
process 100 ... 199 (side effects done)
CRASH before commitSync()
restart: resume at 100 -> records 100..199 processed a second time (duplicates)
The rules for at-least-once
- Disable auto commit, or use it only with fully synchronous processing (Section 5.3).
- Commit an offset only after every record before it in that partition is fully processed.
- Commit
last processed offset + 1. - Never let an exception skip a record silently: retry, or send the record to a dead letter topic (DLT), and only then commit past it.
- Commit in
onPartitionsRevokedand on shutdown so normal rebalances and restarts cause few duplicates.
Handling failures without blocking or losing data
process(record)
|
ok? -----+------ failed
| |
continue transient? -- yes --> retry with backoff (bounded; pause partition)
|
no / retries exhausted
|
produce to orders.dlt (with error headers)
|
wait for the DLT write to succeed, then continue
Example: at-least-once with a dead letter topic
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> rec : records) {
try {
withRetries(3, () -> process(rec)); // bounded retries
} catch (Exception e) {
ProducerRecord<String, String> dlt = new ProducerRecord<>("orders.dlt", rec.key(), rec.value());
dlt.headers().add("x-original-topic", rec.topic().getBytes(UTF_8))
.add("x-original-partition", Integer.toString(rec.partition()).getBytes(UTF_8))
.add("x-original-offset", Long.toString(rec.offset()).getBytes(UTF_8))
.add("x-exception", String.valueOf(e).getBytes(UTF_8));
dltProducer.send(dlt).get(); // must succeed before committing past rec
}
}
consumer.commitAsync(); // after the whole batch is handled
}
How many duplicates to expect
| Event | Records reprocessed |
|---|---|
Clean shutdown or cooperative rebalance with commit in onPartitionsRevoked | None |
Crash (kill -9, out of memory, node loss) | Up to everything since the last commit (one batch with per-batch commits) |
Member exceeded max.poll.interval.ms | The whole uncommitted batch, processed by both the old and the new owner |
| Offset reset or replay | Everything in the replayed range |
Key Takeaways
- At-least-once means process, then commit: a crash causes duplicates but never loss.
- Commit only offsets whose preceding records are all done, and commit
offset + 1. - Failed records go to a dead letter topic (after bounded retries) before the offset moves past them.
- Duplicates are unavoidable after crashes, so pair at-least-once with idempotent processing.
Practice
- (Beginner) Kill an at-least-once consumer mid-batch and count the duplicates after restart. Change the commit frequency and repeat.
- (Intermediate) Implement the dead letter flow above and a small tool that reads
orders.dltand republishes selected records toordersafter a fix.- Interview question: "Your consumer is at-least-once, but you still lost a record in production. What could have caused it?" Hint: look for swallowed exceptions, auto commit with asynchronous work, offset resets, retention deleting unread data, and the producer side.
Idempotent Consumers
Theory
At-least-once delivery guarantees duplicates will eventually happen. Instead of trying to prevent them everywhere, make processing idempotent: processing the same record twice has the same effect as processing it once. Idempotent processing plus at-least-once delivery gives effectively-once results, which is what most systems really need.
Analogy: pressing an elevator call button is idempotent. Pressing it five times calls the elevator once. Adding money to an account balance is not idempotent; setting the balance to a value, or recording "deposit #8812" in a ledger that rejects duplicate numbers, is.
Techniques
| Technique | How it works | Good for |
|---|---|---|
| Naturally idempotent operations | Upsert by key, set a value, delete by ID | State updates ("customer 77's address is X") |
| Deduplication by event ID | Store processed event IDs with a unique constraint; skip on conflict | Commands and financial events with a unique ID |
| Deduplication by position | Store (topic, partition, offset) processed; skip if seen | Any record, but replays after topic recreation reset offsets |
| Offsets stored with the results | Write results and next offset in one database transaction; seek on assignment (Section 5.3) | Consumers writing to a transactional database |
| Conditional writes / versions | Apply only if version > stored version | Out-of-order or replayed updates |
| Kafka transactions | Consume-transform-produce with offsets committed in the producer's transaction | Kafka-to-Kafka pipelines (Chapter 8) |
Choosing a deduplication key
Producer retries are deduplicated by the idempotent producer (Chapter 8), but an application that sends the same business event twice (for example, after its own crash) creates two records with different offsets. A business event ID generated once at the source (an order ID, a UUID in a header) catches both kinds of duplicates. Offsets only catch redelivery of the same record.
Example: deduplication with a unique constraint
CREATE TABLE processed_events (
event_id VARCHAR(64) PRIMARY KEY, -- business ID from the record
processed_at TIMESTAMP NOT NULL DEFAULT now()
);
void process(ConsumerRecord<String, String> rec) {
String eventId = new String(rec.headers().lastHeader("event-id").value(), UTF_8);
jdbc.inTransaction(tx -> {
int inserted = tx.update(
"INSERT INTO processed_events (event_id) VALUES (?) ON CONFLICT DO NOTHING", eventId);
if (inserted == 0) return; // duplicate: already applied, do nothing
tx.update("UPDATE accounts SET balance = balance + ? WHERE id = ?",
amountOf(rec), accountOf(rec)); // the side effect, in the same transaction
});
}
The dedup insert and the side effect commit or roll back together, so a crash can never leave one without the other.
Example: offsets stored with the results
void processBatch(TopicPartition tp, List<ConsumerRecord<String, String>> batch) {
jdbc.inTransaction(tx -> {
for (ConsumerRecord<String, String> rec : batch) {
tx.update("INSERT INTO orders (id, body) VALUES (?, ?) ON CONFLICT (id) DO UPDATE SET body = ?",
rec.key(), rec.value(), rec.value());
}
long next = batch.get(batch.size() - 1).offset() + 1;
tx.update("INSERT INTO consumer_progress (topic, part, next_offset) VALUES (?, ?, ?) " +
"ON CONFLICT (topic, part) DO UPDATE SET next_offset = ?",
tp.topic(), tp.partition(), next, next);
});
// No Kafka commit needed; onPartitionsAssigned seeks to consumer_progress (Section 5.3)
}
Side effects outside your database
Calls to external systems (payment providers, email services) cannot join your database transaction. Use the provider's idempotency key feature (send the event ID with the request) or record the intent first and reconcile later (the outbox and inbox patterns, Chapter 15).
Key Takeaways
- Idempotent processing turns at-least-once delivery into effectively-once results.
- Prefer naturally idempotent writes (upserts, sets). Otherwise deduplicate by a business event ID stored in the same transaction as the side effect.
- Storing offsets with the results in one transaction removes the gap between processing and committing.
- For external APIs, pass an idempotency key. For Kafka-to-Kafka pipelines, use transactions (Chapter 8).
Practice
- (Beginner) Classify each operation as idempotent or not: "set order status to SHIPPED", "increment page views", "insert row with auto-generated ID", "delete user 42", "send welcome email".
- (Intermediate) Implement the deduplication table example with PostgreSQL. Replay a topic with
--to-earliestand verify that balances do not change. Then design a cleanup policy forprocessed_eventsthat remains safe.- Interview question: "How do you achieve exactly-once processing when a Kafka consumer writes to a relational database?" Hint: consider where the offset is stored relative to the results, and what single atomic operation covers both.
Consumer Lag
Theory
Consumer lag is how far a consumer group is behind the data available in a partition: the high watermark minus the group's committed offset (Section 2.1). It is the single most important consumer health metric. Zero or stable lag means the group keeps up. Growing lag means records arrive faster than they are processed, and eventually either latency becomes unacceptable or retention deletes records before they are read.
Analogy: lag is the length of the queue at a checkout. A short, stable queue is fine. A queue that keeps getting longer means more cashiers are needed, or a cashier is stuck, and if it grows long enough, customers leave (retention deletes the data).
Measuring lag
partition orders-2
log start committed (group) position (consumer) high watermark
| | | |
1 000 9 120 9 620 21 502
offset lag (committed) = 21 502 - 9 120 = 12 382 what kafka-consumer-groups.sh shows
offset lag (position) = 21 502 - 9 620 = 11 882 what the consumer's records-lag metric shows
time lag = now - timestamp(record at 9 120) = e.g. 14 minutes
distance to deletion = 9 120 - 1 000 = 8 120 records-lead: how close to losing data
| Measure | Source | Use |
|---|---|---|
| Offset lag per partition | kafka-consumer-groups.sh --describe, Admin API, exporters | Spot hot or stuck partitions |
records-lag-max (consumer metric) | Each consumer's JMX metrics | Alert from the application side |
| Time lag (seconds behind) | Lag exporters, or computed from record timestamps | Alerts that match SLAs ("data older than 5 minutes") |
records-lead-min (consumer metric) | Each consumer's metrics | Warns when the position approaches the log start offset (data about to be deleted) |
Offset lag is hard to alert on directly: 10,000 records is nothing for a topic receiving 50,000 per second and hours for one receiving 2 per second. Time lag is usually the better alerting metric. Tools such as Burrow, Kafka Exporter for Prometheus, and managed-service dashboards compute lag for all groups from the outside, which keeps working when the consumer itself is down.
Reading lag patterns
| Pattern | Likely cause | Next step |
|---|---|---|
| All partitions growing evenly | Throughput too low for the input rate | Scale out (up to partition count), speed up processing, batch writes |
| One partition growing, others near zero | Hot key (Section 4.2) or a slow, stuck member | Check key distribution; check the owning member's logs |
| Lag grows, then drops to zero repeatedly | Rebalance storms, GC pauses, periodic slow dependency | Correlate with rebalance logs and dependency latency |
| Lag frozen at a constant value, consumer running | Poison pill or infinite retry on one record | Find the offset; handle it (Section 5.1) |
Lag on a partition with no owner (CONSUMER-ID is -) | Fewer active members than expected, or a static member that has not returned | Check the deployment and group membership |
| Lag spikes after a deployment | Rebalances and cold caches | Static membership, cooperative rebalancing (Section 5.2) |
Example: computing lag with the Admin API
try (Admin admin = Admin.create(Map.of("bootstrap.servers", "localhost:9092"))) {
Map<TopicPartition, OffsetAndMetadata> committed =
admin.listConsumerGroupOffsets("order-service").partitionsToOffsetAndMetadata().get();
Map<TopicPartition, OffsetSpec> latest = new HashMap<>();
committed.keySet().forEach(tp -> latest.put(tp, OffsetSpec.latest()));
Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> ends = admin.listOffsets(latest).all().get();
committed.forEach((tp, om) -> {
long lag = ends.get(tp).offset() - om.offset();
System.out.printf("%s lag=%d%n", tp, lag);
});
}
Example: a lag alert (Prometheus rule)
groups:
- name: kafka-consumers
rules:
- alert: ConsumerGroupFallingBehind
# kafka_exporter metric; sum over partitions, growing for 10 minutes
expr: sum by (consumergroup, topic) (kafka_consumergroup_lag) > 100000
and deriv(sum by (consumergroup, topic) (kafka_consumergroup_lag)[10m:]) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.consumergroup }} is falling behind on {{ $labels.topic }}"
Key Takeaways
- Lag is the high watermark minus the committed offset, per partition. It is the primary measure of consumer health.
- Alert on sustained growth and on time lag, not on a fixed offset count alone.
- The pattern of lag across partitions (uniform, one partition, frozen, oscillating) points to the cause.
- Watch the distance to the log start offset: a consumer that falls behind retention loses data.
Practice
- (Beginner) Produce 100,000 records, consume with a deliberately slow consumer, and watch lag with
kafka-consumer-groups.sh --describeevery few seconds. Then add a second consumer and observe the change.- (Intermediate) Extend the Admin API example to compute time lag: fetch the record at each committed offset with a temporary consumer (using
assign()andseek()) and subtract its timestamp from now.- Interview question: "Lag on one partition of a 24-partition topic has been constant at 3,412 for an hour, while the consumer process is healthy and other partitions are at zero. What is happening?" Hint: constant (not growing) lag on one partition while others drain suggests the consumer is not advancing on that partition at all.
Multithreaded Consumption
Theory
A KafkaConsumer must be used from one thread, and within a group, a partition has only one owner. Together, these limit a group's parallelism to the number of partitions. That is fine when processing is CPU-light, but when each record waits on I/O (an HTTP call, a slow database), one thread per partition leaves most of the machine idle. There are three main ways to add parallelism, each with different trade-offs for ordering and commit correctness.
Analogy: a post office with one clerk per counter (partition). You can open more counters (more partitions and consumers), or let each clerk hand parcels to several helpers in the back room. Helpers speed things up, but then someone must track which parcels are actually finished before telling customers "done".
Model 1: one consumer per thread
process (or several processes)
+-----------------------+ +-----------------------+ +-----------------------+
| thread 1: consumer A | | thread 2: consumer B | | thread 3: consumer C |
| P0, P1 | | P2, P3 | | P4, P5 |
| poll -> process -> | | poll -> process -> | | poll -> process -> |
| commit | | commit | | commit |
+-----------------------+ +-----------------------+ +-----------------------+
Simple and correct: ordering per partition, normal commits, normal rebalancing. Parallelism is capped by the partition count.
Model 2: one poller, a pool of workers
poll thread worker pool
+------------------------------+ +---------------------+
| poll() -> records | submit | worker 1 |
| per-partition queue -------> | -----------> | worker 2 |
| pause(tp) if queue is full | | ... |
| resume(tp) when drained | completed | worker N |
| commit highest contiguous | <----------- | |
| completed offset per tp | offsets +---------------------+
+------------------------------+
The poll thread keeps calling poll() (so the consumer never exceeds max.poll.interval.ms), and uses pause() to stop fetching partitions whose workers are busy. Commits must only cover offsets where all earlier records are complete: if offsets 10, 11, and 13 are done but 12 is still running, the commit is 12.
Model 3: key-level parallelism (Confluent Parallel Consumer)
The open-source Confluent Parallel Consumer library processes records with different keys in parallel within one partition, while keeping records with the same key in order. It tracks per-record completion and encodes incomplete offsets in the commit metadata.
| Model | Max parallelism | Ordering | Commit complexity | Typical use |
|---|---|---|---|---|
| One consumer per thread | Partition count | Per partition | Low | CPU-bound work, most applications |
| Poller + worker pool | Number of workers | Per partition only if one worker per partition; otherwise none | High (contiguous tracking, pause/resume, rebalances) | I/O-bound work with relaxed ordering |
| Key-level (Parallel Consumer) | Number of distinct keys in flight | Per key | Handled by the library | I/O-bound work that needs per-key order |
| Share groups | Number of consumers, independent of partitions | None | Per-record acknowledgment (topic "Share Groups") | Work queues |
Example: Model 2 sketch with per-partition ordering
// One single-threaded executor per partition: order kept within a partition,
// partitions processed in parallel. Commits only after each batch completes.
Map<TopicPartition, ExecutorService> lanes = new ConcurrentHashMap<>();
Map<TopicPartition, Future<Long>> inFlight = new HashMap<>();
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (TopicPartition tp : records.partitions()) {
List<ConsumerRecord<String, String>> batch = records.records(tp);
ExecutorService lane = lanes.computeIfAbsent(tp, k -> Executors.newSingleThreadExecutor());
inFlight.put(tp, lane.submit(() -> {
batch.forEach(r -> process(r)); // slow, I/O-bound work
return batch.get(batch.size() - 1).offset() + 1; // next offset to commit
}));
consumer.pause(List.of(tp)); // no more records until this batch is done
}
// Commit and resume partitions whose batch finished (non-blocking check)
Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();
for (Iterator<Map.Entry<TopicPartition, Future<Long>>> it = inFlight.entrySet().iterator(); it.hasNext(); ) {
Map.Entry<TopicPartition, Future<Long>> e = it.next();
if (e.getValue().isDone()) {
toCommit.put(e.getKey(), new OffsetAndMetadata(e.getValue().get()));
consumer.resume(List.of(e.getKey()));
it.remove();
}
}
if (!toCommit.isEmpty()) consumer.commitAsync(toCommit, null);
}
// A complete implementation must also, in onPartitionsRevoked: wait for (or cancel) in-flight
// batches of revoked partitions, commit their results, and remove their lanes.
Pitfalls of hand-rolled multithreading
| Pitfall | Consequence |
|---|---|
| Committing the highest completed offset instead of the highest contiguous one | Data loss on crash: unfinished earlier records are skipped |
| Not handling revocation | Two members process the same partition at once after a rebalance |
| Unbounded internal queues | Out-of-memory under load; use pause() for backpressure |
| Calling consumer methods from worker threads | ConcurrentModificationException |
| Losing per-key ordering by spreading one partition over many workers | Updates applied out of order (for example, "shipped" before "paid") |
Key Takeaways
- Start with one consumer per thread; add partitions and instances to scale. It is simple and correct.
- For I/O-bound work, a poller with a worker pool increases parallelism beyond the partition count, but requires
pause()/resume(), contiguous offset tracking, and careful rebalance handling. - Key-level parallel processing (for example, Confluent Parallel Consumer) keeps per-key order while using many threads per partition.
- Only the poll thread may call consumer methods.
Practice
- (Beginner) Run three threads, each with its own
KafkaConsumerin the same group, on a 6-partition topic. Log which thread processes which partition.- (Intermediate) Complete the Model 2 sketch: implement the
onPartitionsRevokedhandling and test it by adding and removing consumers while records are in flight. Verify no record is lost or processed concurrently by two members.- Interview question: "Each record takes 200 ms of I/O and the topic has 12 partitions. You need 300 records per second. What are your options?" Hint: compute the maximum throughput of one consumer per partition, then compare adding partitions, worker pools, key-level parallelism, and share groups, including their ordering implications.
Share Groups (Queues for Kafka)
Theory
Consumer groups give each partition to one member. That is perfect for ordered streams, but awkward for work queues: a pool of workers that should take whichever job is next, scale independently of partition count, and acknowledge or retry jobs one at a time. Before Kafka 4.x, teams either over-partitioned topics or used a separate queueing system for this.
Share groups (KIP-932, "Queues for Kafka") add queue semantics on top of ordinary topics. Members of a share group cooperatively consume the same partitions: records are distributed to consumers individually, each record is acquired by one consumer for a time-limited lock, and the consumer acknowledges it as processed, released for redelivery, or rejected. Share groups were introduced as early access in Kafka 4.0 and matured over the following 4.x releases; check the release notes of your version before relying on them in production.
Analogy: a consumer group is a set of cashiers, each with their own line of customers (partitions). A share group is a single line with a "next available cashier" sign: any free cashier takes the next customer, and a customer whose transaction failed goes back into the line.
Consumer groups vs share groups
| Aspect | Consumer group | Share group |
|---|---|---|
| Unit of assignment | Whole partition, one owner | Individual records; partitions shared by many consumers |
| Max useful consumers | Partition count | Not limited by partition count |
| Ordering | Per partition | Not guaranteed |
| Progress tracking | One committed offset per partition | Per-record state within an in-flight window |
| Acknowledgment | Implicit via offset commit | Per record: accept, release, or reject |
| Redelivery | Only by rewinding offsets | Automatic on release or lock timeout, up to a delivery limit |
| Client | KafkaConsumer | KafkaShareConsumer |
| State stored in | __consumer_offsets | __share_group_state |
Record states
share-partition orders-0, in-flight window between start offset (SPSO) and end offset (SPEO)
SPSO SPEO
| |
... | 40 ACKED | 41 ACQUIRED(c1) | 42 AVAILABLE | 43 ACQUIRED(c2) | 44 ARCHIVED | ...
AVAILABLE --deliver to a consumer------------------------------> ACQUIRED (lock: 30 s)
ACQUIRED --ACCEPT-------------------------------------------> ACKNOWLEDGED
ACQUIRED --RELEASE or lock timeout--------------------------> AVAILABLE (delivery count + 1)
ACQUIRED --REJECT or delivery count limit (5) reached -------> ARCHIVED (never delivered again)
| Broker setting | Default | Meaning |
|---|---|---|
group.share.record.lock.duration.ms | 30000 | How long a consumer holds an acquired record before it is released automatically |
group.share.delivery.count.limit | 5 | Delivery attempts before a record is archived |
share.auto.offset.reset (group config) | latest | Where a new share group starts: earliest or latest |
Example: a share consumer with explicit acknowledgment
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "email-workers"); // a share group
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
props.put("share.acknowledgement.mode", "explicit"); // default "implicit": accept all on next poll
try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props)) {
consumer.subscribe(List.of("email-jobs"));
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> rec : records) {
try {
sendEmail(rec.value());
consumer.acknowledge(rec, AcknowledgeType.ACCEPT); // done
} catch (TransientMailException e) {
consumer.acknowledge(rec, AcknowledgeType.RELEASE); // redeliver, maybe to another worker
} catch (InvalidAddressException e) {
consumer.acknowledge(rec, AcknowledgeType.REJECT); // never deliver again
}
}
consumer.commitSync(); // send the acknowledgments now
}
}
Operating share groups
BS=localhost:9092
# Kafka 4.1+: enable the feature on the cluster (4.0 used an early-access configuration instead)
bin/kafka-features.sh --bootstrap-server $BS upgrade --feature share.version=1
# Start new share groups from the beginning of the topic
bin/kafka-configs.sh --bootstrap-server $BS --alter --entity-type groups \
--entity-name email-workers --add-config share.auto.offset.reset=earliest
# Inspect members and progress
bin/kafka-share-groups.sh --bootstrap-server $BS --describe --group email-workers
bin/kafka-share-groups.sh --bootstrap-server $BS --describe --group email-workers --members
When to use which
| Requirement | Choose |
|---|---|
| Per-key ordering (account events, CDC, state machines) | Consumer group |
| Stateful stream processing, Kafka Streams, exactly-once pipelines | Consumer group |
| Independent jobs (emails, image resizing, webhooks), workers scaled on demand | Share group |
| More workers than partitions, without repartitioning the topic | Share group |
| Per-message retry and poison-message isolation without blocking the partition | Share group |
Key Takeaways
- Share groups let many consumers read the same partitions, with records delivered individually and acknowledged one by one.
- Records are acquired under a time-limited lock; release or timeout causes redelivery, and reject or the delivery limit archives the record.
- They remove the "consumers no more than partitions" limit, but give up ordering.
- Use consumer groups for ordered streams and processing state; use share groups for work queues.
Practice
- (Beginner) On a Kafka 4.x cluster with share groups enabled, create a topic with 1 partition, produce 100 records, and consume them with three
KafkaShareConsumerinstances in the same share group. Record how the records are distributed.- (Intermediate) Make one worker release every record whose value contains
retry, and another reject records containingbad. Observe redelivery counts and confirm that rejected records are not delivered again. What happens after the delivery count limit?- Interview question: "When would you use a Kafka share group instead of a consumer group, and what do you give up?" Hint: think about the relationship between consumers and partitions, ordering, and how failed records are handled in each model.
<a id="6-data-serialization-and-schemas"></a>