8. Delivery Semantics and Transactions
Chapters 4 and 5 showed how acknowledgments, retries, and offset commits give you at-most-once or at-least-once delivery, and why neither is enough when a duplicate payment or a lost order is unacceptable. This chapter covers the machinery Kafka adds on top: the idempotent producer, which removes duplicates caused by retries; transactions, which make writes to many partitions and offset commits succeed or fail together; and the exactly-once semantics built from both. It also marks where the guarantee ends, because "exactly-once" in Kafka has a precise scope and most production bugs come from assuming it reaches further than it does.
<a id="81-idempotence"></a>
8.1 Idempotence
This section explains how a producer can retry as often as it needs without ever writing the same batch twice, using producer IDs, epochs, and per-partition sequence numbers checked by the partition leader.
Idempotent Producer
Theory
Every network request has an ambiguous failure mode. The producer sends a batch, the leader appends it and replicates it, and then the response is lost (a timeout, a dropped connection, a leader failover). The producer cannot tell "the write never happened" apart from "the write happened but I never heard back". It has two choices, and before idempotence both were bad:
Producer Leader (partition orders-0)
| Produce(batch: "order 42") |
|------------------------------->| append at offset 900, replicate, commit
| ack (offset 900) |
| X<------------------------| response lost / request timed out
| |
| Option 1: give up | -> the record may or may not be in the log; the app sees an error
| Option 2: retry |
|------------------------------->| append again at offset 901 -> DUPLICATE
An operation is idempotent when doing it twice has the same effect as doing it once. The idempotent producer makes Produce idempotent from the broker's point of view: every batch carries an identity, and the leader refuses to append a batch it has already written, acknowledging it as successful instead. Retries stop being dangerous, so the producer can retry indefinitely within delivery.timeout.ms.
Analogy: a bank that records the number of every cheque it cashes. If the same cheque (#1042) is presented twice, because the courier was not sure the first delivery arrived, the bank cashes it once and answers "already paid" the second time. The payer can resend as many times as needed without paying twice.
What you get and what you need
Idempotence is a producer setting, enable.idempotence, and the default in the Java client since Kafka 3.0. (A bug, KAFKA-13598, meant the default only took effect from 3.0.1, 3.1.1, and 3.2.0, so set it explicitly if you support older clients.) It needs three other settings to be compatible:
| Setting | Required value | Why |
|---|---|---|
acks | all | A batch acknowledged by the leader alone could vanish on failover, and the deduplication state would vanish with it |
retries | greater than 0 (default Integer.MAX_VALUE) | Without retries there is nothing to deduplicate |
max.in.flight.requests.per.connection | 5 or less | The leader remembers only the last 5 batches per producer and partition (see Duplicate Detection) |
If these conflict and you did not set enable.idempotence explicitly, the Java producer silently turns idempotence off. If you did set it explicitly, construction fails with a ConfigException (Section 4.3).
Guarantees and their scope
| Covered | Not covered |
|---|---|
| Duplicates caused by the producer's own internal retries | The application calling send() again for the same data (a new batch with a new sequence number) |
| Reordering caused by retries with up to 5 requests in flight | A producer restart: the new instance gets a new producer ID and the broker cannot link it to the old one |
| Leader failover (the state is replicated with the data) | Atomicity across partitions (one batch may succeed and another fail) |
| Duplicates created upstream of Kafka (a client retrying an HTTP request) |
Idempotence costs almost nothing. The producer ID, epoch, and sequence fields exist in every record batch header anyway (Section 7.1), and the leader's check is an in-memory lookup.
Example: configuring an idempotent producer
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // explicit: fail fast if a setting conflicts
props.put(ProducerConfig.ACKS_CONFIG, "all"); // default since 3.0, required anyway
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000); // total time budget for retries of one batch
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("orders", "order-42", "{\"amount\": 99.90}"),
(metadata, e) -> {
if (e != null) {
// The producer has already retried internally. Do NOT send again here:
// a new send() is a new batch and would not be deduplicated.
log.error("order-42 not confirmed", e);
}
});
}
from confluent_kafka import Producer
# librdkafka defaults enable.idempotence to False, unlike the Java client
producer = Producer({
"bootstrap.servers": "localhost:9092",
"enable.idempotence": True, # also forces acks=all, retries, and max.in.flight <= 5
})
producer.produce("orders", key="order-42", value=b'{"amount": 99.90}')
producer.flush()
Permissions: since Kafka 3.0 (KIP-679) an idempotent producer only needs WRITE on the topic. The older IDEMPOTENT_WRITE cluster ACL is deprecated.
Key Takeaways
- A lost acknowledgment makes a retry look like a duplicate. The idempotent producer lets the leader recognize and drop retried batches.
enable.idempotence=trueis the Java default since 3.0 (not in librdkafka). It needsacks=all, retries, and at most 5 in-flight requests.- The guarantee is per partition and per producer session. It does not cover application-level resends, producer restarts, or multi-partition atomicity.
- Never resend from a send callback: the resend is a new batch and is not deduplicated.
Practice
- (Beginner) Create a producer with
acks=1and no explicitenable.idempotence, then again withenable.idempotence=true. Note what happens in each case and explain why.- (Intermediate) A service calls
producer.send(record).get()and, onTimeoutException, callssend(record)again. Can the topic end up with two copies of the record? Explain using what the broker knows about each attempt.- Interview question: "Idempotence is enabled on all producers. Name three ways a topic can still contain duplicate business events." Hint: think about who creates a new batch (the application, a restarted process, an upstream client) as opposed to who retries an existing one.
Producer IDs and Sequence Numbers
Theory
To recognize a retried batch, the leader needs to know two things: who sent it, and which batch in that sender's stream it is. Kafka encodes both in the record batch header:
| Field | Type | Meaning |
|---|---|---|
producerId (PID) | int64 | Identifies one producer session. Assigned by the cluster |
producerEpoch | int16 | Generation of that PID. Bumped to reset sequences or fence older instances |
baseSequence | int32 | Sequence number of the first record in the batch, counted per PID and per partition |
| (derived) last sequence | baseSequence + lastOffsetDelta, the sequence of the last record |
Batches without idempotence carry producerId = -1 and baseSequence = -1, and the leader skips the check.
Where producer IDs come from
When an idempotent producer starts, it sends an InitProducerId request to any broker and receives a PID and an epoch (0). The PID must be unique across the cluster, so in KRaft mode brokers request blocks of 1,000 IDs from the active controller (AllocateProducerIds, KIP-730). The controller records each allocated block in the metadata log, so no two brokers ever hand out the same ID, even after a controller failover. In ZooKeeper mode the same block reservation was stored in a ZooKeeper node.
How sequence numbers are assigned
The producer keeps a counter per partition, starting at 0. When a batch is drained from the accumulator and sent for the first time, it gets the current counter value as its baseSequence, and the counter advances by the number of records in the batch. The numbers are then fixed: a retry resends the same bytes with the same sequence. This is exactly why internal retries are deduplicated while a second send() call is not, because the second call produces a new batch with new sequence numbers.
Producer PID=4000, epoch=0
partition orders-0: batch(5 records) seq 0-4 batch(3 records) seq 5-7 batch(1 record) seq 8
partition orders-1: batch(2 records) seq 0-1 batch(4 records) seq 2-5
^ each partition has its own independent sequence counter
Sequence numbers wrap to 0 after Integer.MAX_VALUE. The epoch is bumped by the producer itself (KIP-360) when it can no longer be sure which sequence the broker expects, for example after a batch expired with delivery.timeout.ms. A new epoch starts every partition's sequence at 0 again, without needing a new PID.
State on the broker
Each partition leader keeps a producer state map: for every PID that wrote to the partition, its current epoch and the metadata (first and last sequence, offsets, timestamp) of its last 5 batches. Followers build the same map because they replicate the same batches, so it survives leader failover. The map is checkpointed to the .snapshot files next to each segment (Section 2.1, Section 7.1) and rebuilt on startup from the latest snapshot plus a replay of the log after it.
State does not live forever:
producer.id.expiration.ms(broker, default 1 day, KIP-854) removes PIDs of idempotent producers that have not written for that long.- When retention deletes all segments containing a PID's batches, the state for that PID can be removed as well.
A producer whose state has expired gets UNKNOWN_PRODUCER_ID on its next write and recovers by bumping its epoch. The cost of this design is memory: every distinct PID that wrote recently occupies an entry on every partition it touched. An application that creates a new KafkaProducer per request creates a new PID per request and can fill broker heap with producer state. Reuse producer instances (Section 4.1).
Example: seeing the fields on disk and on the broker
# Batch headers in a segment: producerId, producerEpoch, and sequence ranges
bin/kafka-dump-log.sh --files /var/lib/kafka/data/orders-0/00000000000000000000.log
# baseOffset: 0 lastOffset: 4 count: 5 baseSequence: 0 lastSequence: 4 producerId: 4000
# producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: false isControl: false ...
# baseOffset: 5 lastOffset: 7 count: 3 baseSequence: 5 lastSequence: 7 producerId: 4000
# producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: false isControl: false ...
# Live producer state held by the leader of orders-0
bin/kafka-transactions.sh --bootstrap-server localhost:9092 \
describe-producers --topic orders --partition 0
# ProducerId ProducerEpoch LatestCoordinatorEpoch LastSequence LastTimestamp CurrentTransactionStartOffset
# 4000 0 -1 7 1761210942120 None
LatestCoordinatorEpoch is -1 and CurrentTransactionStartOffset is None because this producer is idempotent but not transactional.
Key Takeaways
- Every idempotent batch carries a producer ID, an epoch, and a base sequence number in its header.
- PIDs are assigned by
InitProducerId. In KRaft, the controller hands brokers unique blocks of IDs. - Sequences are per PID and per partition, assigned once when a batch is first sent, and reused unchanged on retry.
- Leaders keep the last 5 batches per PID per partition, replicate this state implicitly, and checkpoint it in
.snapshotfiles. - Producer state expires (
producer.id.expiration.ms). Many short-lived producers mean many PIDs and more broker memory.
Practice
- (Beginner) A producer sends three batches to partition 0 with 4, 10, and 1 records, and two batches to partition 1 with 6 and 6 records. Write the base and last sequence of every batch.
- (Intermediate) Produce a few records to a test topic with idempotence enabled, then with it disabled. Use
kafka-dump-log.shto compare theproducerIdandbaseSequencefields, andkafka-transactions.sh describe-producersto see which producers the leader tracks.- Interview question: "A service creates a new
KafkaProducerfor every incoming HTTP request and closes it afterwards. Functionally it works. What problem will the Kafka operators eventually see, and why?" Hint: count how many producer IDs exist per hour, and think about where the brokers keep state for each one and for how long.
Duplicate Detection
Theory
With the producer's identity in every batch, deduplication becomes a check the leader runs before each append. The leader compares the incoming batch against the stored state for its PID and decides one of five outcomes:
| Condition | Leader's response | Producer's reaction |
|---|---|---|
| Epoch lower than the stored epoch | INVALID_PRODUCER_EPOCH / PRODUCER_FENCED | This instance is stale. Fatal for transactional producers (see Zombie Fencing) |
| Sequence range equals one of the last 5 cached batches | Success, with the offset of the original write. Nothing is appended | Treats the batch as delivered |
baseSequence = last stored sequence + 1 (or 0 for a new epoch) | Append | Normal success |
baseSequence greater than expected (a gap) | OUT_OF_ORDER_SEQUENCE_NUMBER | Retries once the earlier batch has been written. If that batch is gone for good, bumps the epoch |
| PID unknown and sequence not 0 | UNKNOWN_PRODUCER_ID | Bumps the epoch and resets sequences |
The gap check is what preserves ordering with several requests in flight. Without it, a failed batch retried after a later batch succeeded would land after it in the log (Section 4.3). With it, the later batch is rejected until the earlier one is written, and the producer resends them in order.
max.in.flight = 3, all to partition orders-0
t1 send A(seq 0-4) B(seq 5-9) C(seq 10-14)
t2 A fails (NOT_LEADER_OR_FOLLOWER) B, C reach the new leader
t3 leader: B -> OUT_OF_ORDER (expected 0) C -> OUT_OF_ORDER (expected 0)
t4 producer resends A, B, C in sequence order
t5 leader: A appended, B appended, C appended -> log order A, B, C
Why only 5 batches?
The producer never has more than 5 unacknowledged batches per partition (that is what max.in.flight.requests.per.connection <= 5 guarantees, since all of a partition's requests go over the one connection to its leader). Any retry must therefore be one of the last 5 batches the leader accepted, or a batch it has not seen yet. A larger cache would cost memory for every PID on every partition without catching any additional duplicates.
When the producer gives up
Deduplication only removes duplicates the producer knows about. If a batch exceeds delivery.timeout.ms, the producer reports a TimeoutException to the application. The batch may or may not be in the log, and the producer no longer knows which sequence the leader expects next, so it bumps its epoch and restarts that partition's sequences at 0. If the application now resends the data, it may create a duplicate: at that boundary you are back to at-least-once, and only an idempotent consumer (Section 5.4) or a business key can catch it.
Example: simulating the leader's check
from collections import deque
from dataclasses import dataclass, field
@dataclass
class ProducerEntry:
epoch: int
batches: deque = field(default_factory=lambda: deque(maxlen=5)) # (first_seq, last_seq, base_offset)
class PartitionLeader:
def __init__(self):
self.log_end_offset = 0
self.producers: dict[int, ProducerEntry] = {}
def append(self, pid: int, epoch: int, base_seq: int, count: int) -> str:
last_seq = base_seq + count - 1
entry = self.producers.get(pid)
if entry is None: # first batch from this producer (or state expired)
if base_seq != 0:
return "UNKNOWN_PRODUCER_ID"
elif epoch < entry.epoch: # an older incarnation: a zombie
return "INVALID_PRODUCER_EPOCH"
elif epoch > entry.epoch: # new epoch: sequences restart at 0
if base_seq != 0:
return "OUT_OF_ORDER_SEQUENCE_NUMBER"
else:
for first, last, offset in entry.batches: # 1. is it one of the last 5 batches?
if (first, last) == (base_seq, last_seq):
return f"DUPLICATE -> ack with original offset {offset}"
expected = entry.batches[-1][1] + 1
if base_seq != expected: # 2. is it the next batch in line?
return f"OUT_OF_ORDER_SEQUENCE_NUMBER (expected {expected})"
if entry is None or epoch > entry.epoch:
entry = self.producers[pid] = ProducerEntry(epoch) # new epoch: forget old batches
offset = self.log_end_offset
entry.batches.append((base_seq, last_seq, offset))
self.log_end_offset += count
return f"APPENDED at offsets {offset}-{offset + count - 1}"
leader = PartitionLeader()
PID = 4000
steps = [
("batch A, seq 0-4", PID, 0, 0, 5),
("batch C arrives before B", PID, 0, 10, 5),
("batch B, seq 5-9", PID, 0, 5, 5),
("retry of B (ack was lost)", PID, 0, 5, 5),
("batch C retried in order", PID, 0, 10, 5),
("producer bumped to epoch 1", PID, 1, 0, 2),
("old instance still on 0", PID, 0, 15, 1),
]
for label, pid, epoch, seq, n in steps:
print(f"{label:<28} -> {leader.append(pid, epoch, seq, n)}")
# batch A, seq 0-4 -> APPENDED at offsets 0-4
# batch C arrives before B -> OUT_OF_ORDER_SEQUENCE_NUMBER (expected 5)
# batch B, seq 5-9 -> APPENDED at offsets 5-9
# retry of B (ack was lost) -> DUPLICATE -> ack with original offset 5
# batch C retried in order -> APPENDED at offsets 10-14
# producer bumped to epoch 1 -> APPENDED at offsets 15-16
# old instance still on 0 -> INVALID_PRODUCER_EPOCH
Key Takeaways
- The leader accepts a batch only if its sequence is exactly the next one expected for that PID and epoch.
- A batch matching one of the last 5 cached batches is acknowledged with its original offset and not appended again.
- Gaps are rejected with
OUT_OF_ORDER_SEQUENCE_NUMBER, which keeps per-partition order with up to 5 requests in flight. - Older epochs are rejected. The same mechanism later fences zombie transactional producers.
- After a delivery timeout the producer resets its sequences, and an application resend can create a duplicate.
Practice
- (Beginner) Using the simulator, predict and then check the result of sending batch A (seq 0-4) a third time after batch C has been appended.
- (Intermediate) The simulator caches only the last 5 batches per producer. Append six batches, then retry the first one, and note how the leader classifies it. Then explain why a correctly configured producer can never send such a retry.
- Interview question: "How does deduplication keep working when the partition leader fails and a follower takes over in the middle of a producer's retries?" Hint: think about how the follower learned about the batches the old leader accepted, and what
acks=allguarantees about those batches.
<a id="82-transactions"></a>
8.2 Transactions
Idempotence protects one producer writing to one partition. This section covers transactions, which make a group of writes across many partitions, together with consumer offset commits, become visible all at once or not at all, and which use a broker-side coordinator, an internal log, and epochs to keep that promise through crashes.
Transactional Producer API
Theory
Many applications write more than one record per logical operation:
- A transfer service writes a debit to
account-Aand a credit toaccount-B, which hash to different partitions. - An order service writes to
orders,inventory-reservations, andnotifications. - A stream processor reads a record, writes a result, and commits the input offset. If the result is written but the offset is not, the record is processed again after a crash, and the result is duplicated.
With plain idempotent sends, each partition succeeds or fails on its own, so a crash halfway leaves half an operation in the log. A transaction groups writes to any number of partitions (and, optionally, offset commits) into one atomic unit. Consumers configured with isolation.level=read_committed see either all of the transaction's records or none of them.
Analogy: writing several letters at your desk and sealing them in one envelope. Until you post it, nobody sees any of them. If you change your mind, you shred the envelope, and the recipients never learn the letters existed.
The API
A producer becomes transactional by setting transactional.id, a stable name you choose. Setting it also enables idempotence.
| Call | What it does |
|---|---|
initTransactions() | Once per producer instance. Registers the transactional.id, obtains a PID and a bumped epoch, fences older instances with the same ID, and finishes (aborts) any transaction they left open |
beginTransaction() | Starts a transaction. Purely local: no request is sent |
send(...) | Normal sends. Each partition touched is registered with the transaction |
sendOffsetsToTransaction(offsets, groupMetadata) | Adds consumer offset commits to the transaction (used by consume-transform-produce, Section 8.3) |
commitTransaction() | Flushes pending sends and commits. Blocks until the commit is durable |
abortTransaction() | Discards everything sent in the transaction |
One producer instance runs one transaction at a time. Threads that need independent transactions need separate producer instances with different transactional.id values.
| Setting | Default | Notes |
|---|---|---|
transactional.id | none | Must be unique per producer instance and stable across restarts of that instance (see Zombie Fencing) |
transaction.timeout.ms (producer) | 60000 | The coordinator aborts a transaction open longer than this |
transaction.max.timeout.ms (broker) | 900000 (15 min) | Upper bound. A producer asking for more fails in initTransactions() |
Error handling
Errors fall into two groups, and treating them the same way is the most common bug in transactional code:
| Exception | Meaning | What to do |
|---|---|---|
ProducerFencedException | Another instance with the same transactional.id called initTransactions() | Fatal: close the producer. This instance is a zombie |
OutOfOrderSequenceException | The broker received a sequence gap it cannot resolve | Fatal: close the producer |
AuthorizationException | Missing ACLs for the topic, the transactional ID, or the group | Fatal: close the producer and fix the ACLs |
Other KafkaException (including a failed send() inside the transaction) | The transaction cannot commit | Call abortTransaction(), then retry the whole unit of work |
TimeoutException from commitTransaction() | The outcome is not yet known | Call commitTransaction() again. It is safe to repeat |
If any send() in the transaction failed, commitTransaction() throws and the transaction must be aborted, so you do not need to check every send future yourself.
Required ACLs: WRITE on the topics, WRITE and DESCRIBE on the TransactionalId resource, and READ on the consumer group if offsets are sent to the transaction.
Example: atomic write to two topics (Java)
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "transfer-service-" + instanceId); // stable per instance
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions(); // fence older instances, recover their open transaction
try {
producer.beginTransaction();
producer.send(new ProducerRecord<>("ledger", "account-A", "{\"debit\": 100}"));
producer.send(new ProducerRecord<>("ledger", "account-B", "{\"credit\": 100}"));
producer.send(new ProducerRecord<>("audit", "transfer-881", "{\"from\":\"A\",\"to\":\"B\"}"));
producer.commitTransaction(); // all three become visible together
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
producer.close(); // unrecoverable: another instance owns this ID
throw e;
} catch (KafkaException e) {
producer.abortTransaction(); // none of the three records will be visible
// retry the transfer, or report it as failed
}
Example: the same in Python (confluent-kafka)
from confluent_kafka import Producer, KafkaException
producer = Producer({
"bootstrap.servers": "localhost:9092",
"transactional.id": f"transfer-service-{instance_id}",
"transaction.timeout.ms": 60000,
})
producer.init_transactions()
def transfer(src: str, dst: str, amount: int) -> None:
producer.begin_transaction()
try:
producer.produce("ledger", key=src, value=f'{{"debit": {amount}}}')
producer.produce("ledger", key=dst, value=f'{{"credit": {amount}}}')
producer.commit_transaction()
except KafkaException as e:
err = e.args[0] # a KafkaError
if err.txn_requires_abort(): # abortable: roll back and let the caller retry
producer.abort_transaction()
raise
if err.retriable(): # for example, a commit timeout: committing again is safe
producer.commit_transaction()
return
raise # fatal (fenced, auth): recreate the producer
Key Takeaways
- Transactions make writes to many partitions, plus offset commits, atomic for
read_committedconsumers. - Set
transactional.id, callinitTransactions()once, then repeat begin, send, commit or abort. - Fenced, out-of-sequence, and authorization errors are fatal and require closing the producer. Other errors require aborting and retrying.
transaction.timeout.msbounds how long a transaction may stay open. The broker caps it withtransaction.max.timeout.ms.- One producer instance runs one transaction at a time.
Practice
- (Beginner) Write a transactional producer that sends three records to two topics and then aborts. Read both topics with
kafka-console-consumer.sh, once with the default isolation level and once with--isolation-level read_committed, and compare.- (Intermediate) Predict what happens in each case, then verify: calling
send()withoutbeginTransaction(); keeping a transaction open for 90 seconds with the defaulttransaction.timeout.msand then committing.- Interview question: "How would you choose
transactional.idvalues for a service that runs as 6 replicas in Kubernetes?" Hint: an ID must outlive a single process so that the next incarnation can fence the previous one, yet two live replicas must never share one.
Transaction Coordinator
Theory
Something has to remember which partitions a transaction touched, decide whether it commits, and make sure every partition learns the outcome, even if the producer or a broker crashes in the middle. That role belongs to the transaction coordinator, a component running inside every broker, much like the group coordinator for consumer groups (Section 2.2).
Each transactional.id is owned by exactly one coordinator: the broker that leads partition abs(hash(transactional.id)) % 50 of the internal __transaction_state topic (50 is the default transaction.state.log.num.partitions). Because ownership follows partition leadership, a coordinator failover is just a leader election on __transaction_state.
// How the broker maps a transactional.id to a __transaction_state partition
int partition = Utils.abs("order-processor-0".hashCode()) % 50; // = 28
The protocol, step by step
Producer Coordinator Partition leaders Group coordinator
(txn.id=T) (leader of (orders-0, payments-3) (__consumer_offsets)
__transaction_state-28)
| 1 FindCoordinator(T) ---->|
| 2 InitProducerId(T) ----->| bump epoch, abort T's open txn, log state Empty
|<---- PID=7000, epoch=5 ---|
| beginTransaction() (local only)
| 3 AddPartitionsToTxn ---->| log: Ongoing {orders-0, payments-3}
| 4 Produce (PID, epoch, seq, isTransactional) ---------> | records appended, not yet visible
| 5 AddOffsetsToTxn(group)->| log: + __consumer_offsets-12
| 6 TxnOffsetCommit ----------------------------------------------------------->| pending offsets
| 7 EndTxn(COMMIT) -------->| log: PrepareCommit <- point of no return
|<--------- OK -------------|
| | 8 WriteTxnMarkers(COMMIT) --> | control record in each partition
| |----------------------------------------------------------->| offsets applied
| | log: CompleteCommit
- FindCoordinator locates the broker that owns
T. - InitProducerId returns the PID and a bumped epoch, fencing older instances (see Zombie Fencing).
- AddPartitionsToTxn registers each new partition before the producer first writes to it, so the coordinator knows where markers must go.
- Produce requests write the data directly to the partitions. The records are in the log right away, marked as transactional.
- AddOffsetsToTxn and 6. TxnOffsetCommit add consumer offsets to the transaction (only for consume-transform-produce).
- EndTxn asks to commit or abort.
- WriteTxnMarkers: the coordinator writes a commit or abort control record into every partition in the transaction, then records the transaction as complete.
Two phases and the point of no return
The commit is a two-phase protocol. Phase 1 writes PrepareCommit to __transaction_state (replicated with acks=all). From that moment the decision is final: if the coordinator crashes, the new leader of that __transaction_state partition reloads the state, sees PrepareCommit, and resends the markers. Phase 2 writes the markers and then CompleteCommit. commitTransaction() returns once phase 1 is durable, and the markers follow asynchronously, usually within milliseconds.
The coordinator also runs a periodic check (transaction.abort.timed.out.transaction.cleanup.interval.ms, default 10 seconds) that aborts any transaction open longer than its transaction.timeout.ms, bumping the epoch so the slow producer is fenced.
Server-side defense (KIP-890)
Older protocol versions had a gap: a delayed Produce request from a previous transaction could arrive after that transaction had ended and silently start a new, never-finishing ("hanging") transaction on that partition. KIP-890 closes it in two parts:
- Partition leaders verify with the coordinator that the partition belongs to an ongoing transaction before accepting a transactional write (Kafka 3.6 and later).
- With transaction protocol version 2 (Kafka 4.0, the
transaction.versionfeature), the epoch is bumped at the end of every transaction, so late writes from a finished transaction are rejected outright. The client also stops sendingAddPartitionsToTxnfor data partitions, because the partition leader registers them with the coordinator while handling the produce request.
Example: inspecting transactions with the CLI
# All transactional IDs the cluster knows about
bin/kafka-transactions.sh --bootstrap-server localhost:9092 list
# TransactionalId Coordinator ProducerId TransactionState
# order-processor-0 2 7000 Ongoing
# order-processor-1 1 7001 CompleteCommit
# One transaction in detail
bin/kafka-transactions.sh --bootstrap-server localhost:9092 \
describe --transactional-id order-processor-0
# CoordinatorId TransactionalId ProducerId ProducerEpoch TransactionState TransactionTimeoutMs
# CurrentTransactionStartTimeMs TransactionDurationMs TopicPartitions
# 2 order-processor-0 7000 5 Ongoing 60000
# 1761210942120 850 orders-0,payments-3
# Transaction protocol version in use (Kafka 4.x)
bin/kafka-features.sh --bootstrap-server localhost:9092 describe | grep transaction.version
Key Takeaways
- Each
transactional.idmaps to one__transaction_statepartition. Its leader is the coordinator for that ID. - Data is written to partitions during the transaction. The coordinator only tracks metadata and the outcome.
- Commit is two-phase:
PrepareCommitin the transaction log is the point of no return, then markers are written to every partition. - A coordinator failover resumes unfinished commits from the log. Timed-out transactions are aborted automatically.
- KIP-890 (3.6 and later, fully in 4.0) prevents hanging transactions caused by late produce requests.
Practice
- (Beginner) List, in order, the requests a transactional producer sends to write two records to two partitions and commit. Mark which go to the coordinator and which to partition leaders.
- (Intermediate) The coordinator broker crashes after writing
PrepareCommitbut before writing the marker topayments-3. Describe what aread_committedconsumer ofpayments-3sees during the failover and after it.- Interview question: "Why does Kafka need two phases to commit a transaction, and what is the point of no return?" Hint: consider a crash after some partitions received a commit marker and others did not, and which component must remember the decision so it can finish the job.
Transaction Log
Theory
The coordinator's memory is not enough: if its broker dies, the next coordinator must know every open transaction, its partitions, and whether a commit was already decided. That state lives in __transaction_state, an internal compacted topic. Each record is keyed by transactional.id, and its value is the latest metadata for that ID.
| Value field | Meaning |
|---|---|
producerId, producerEpoch | Current identity and generation of the producer |
transactionTimeoutMs | Timeout the producer asked for |
transactionStatus | State machine position (table below) |
transactionPartitions | Partitions that will need markers |
transactionStartTimestampMs, transactionLastUpdateTimestampMs | Used for timeouts and expiration |
Compaction keeps only the latest state per ID, so the topic stays small no matter how many transactions run. It holds metadata only: the transaction's records live in their own topics, and the outcome is recorded there by control records and in each segment's .txnindex of aborted ranges (Section 7.1).
The state machine
initTransactions()
(none) ----------------------------> Empty
| first AddPartitionsToTxn
v
+------------------ Ongoing ------------------+
EndTxn(COMMIT)| |EndTxn(ABORT) or timeout
v v
PrepareCommit PrepareAbort
| markers written | markers written
v v
CompleteCommit --- next transaction ---> Ongoing <--- CompleteAbort
PrepareEpochFence: the coordinator is aborting an ongoing transaction to fence its producer
Dead: the transactional.id expired and its state is being removed
| State | Durable meaning |
|---|---|
Empty | Producer registered, no transaction in progress |
Ongoing | Transaction open. Partitions listed |
PrepareCommit / PrepareAbort | Outcome decided, markers being written |
CompleteCommit / CompleteAbort | Outcome written to all partitions |
PrepareEpochFence | Timeout or fencing in progress |
Dead | Expired and being deleted |
Configuration and lifecycle
| Broker setting | Default | Notes |
|---|---|---|
transaction.state.log.num.partitions | 50 | Fixed after creation. Changing it remaps every ID to a different coordinator |
transaction.state.log.replication.factor | 3 | The topic is created lazily, and creation fails while fewer brokers than this are available |
transaction.state.log.min.isr | 2 | Writes to the log use acks=all and need this many in-sync replicas |
transaction.state.log.segment.bytes | 104857600 (100 MB) | Small segments keep compaction and coordinator loading fast |
transactional.id.expiration.ms | 604800000 (7 days) | A transactional ID idle this long is expired and its state removed |
A classic development-cluster pitfall: on a single-broker cluster with the default replication factor of 3, __transaction_state cannot be created, and initTransactions() blocks until it times out. Set transaction.state.log.replication.factor=1 and transaction.state.log.min.isr=1 on single-node clusters (never in production).
Example: reading the transaction log
# Kafka 4.x formatter class (3.x: kafka.coordinator.transaction.TransactionLog\$TransactionLogMessageFormatter)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic __transaction_state --from-beginning \
--formatter org.apache.kafka.tools.consumer.TransactionLogMessageFormatter
# Output (abbreviated; the exact shape depends on the version):
# {"key":{"version":0,"data":{"transactionalId":"order-processor-0"}},
# "value":{"version":1,"data":{"producerId":7000,"producerEpoch":5,"transactionTimeoutMs":60000,
# "transactionStatus":1,"transactionPartitions":[{"topic":"orders","partitionIds":[0]}],
# "transactionLastUpdateTimestampMs":1761210942970,"transactionStartTimestampMs":1761210942120}}}
Several records per transaction appear for each ID (Ongoing as partitions are added, then PrepareCommit, then CompleteCommit). After compaction only the last one remains.
Key Takeaways
__transaction_stateis a compacted topic keyed bytransactional.idthat stores each transaction's state, partitions, and timestamps.- It holds metadata only. Data and outcomes live in the data partitions as transactional records and control markers.
- The state machine moves through
Ongoing,Prepare*, andComplete*. ThePrepare*record makes the outcome durable. - Replication factor and
min.isrof the log gate every transaction. On single-node clusters, lower them orinitTransactions()hangs. - Idle transactional IDs expire after
transactional.id.expiration.ms(7 days by default).
Practice
- (Beginner) Explain why
__transaction_stateuses compaction rather than time-based retention, and what would go wrong withcleanup.policy=deleteand a 1-hour retention.- (Intermediate) A developer's transactional producer hangs in
initTransactions()on a one-broker Docker setup and eventually throws aTimeoutException. List the commands you would run to confirm the cause, and the fix.- Interview question: "How many transactions can be in progress at the same time for one
transactional.id, and what does this imply for scaling a transactional application?" Hint: look at what the coordinator stores per key, and at the relationship between producer instances and IDs.
Read Committed vs Read Uncommitted
Theory
Transactional records are appended to partitions as they are sent, long before the outcome is known, and they stay there whether the transaction commits or aborts. Kafka does not move or delete them. Instead, the consumer decides what to show the application, using isolation.level:
read_uncommitted (default) | read_committed | |
|---|---|---|
| Reads up to | High watermark | Last stable offset (LSO) |
| Records of open transactions | Returned | Held back until the outcome is known |
| Records of aborted transactions | Returned | Filtered out |
| Non-transactional records | Returned | Returned, but only once they are below the LSO |
| Control records (markers) | Never returned | Never returned |
| Typical use | Producers do not use transactions, or tools that need to see everything | Any consumer of transactional output |
The last stable offset is the first offset of the earliest still-open transaction in the partition, or the high watermark when no transaction is open (Section 2.4). Everything below it has a known outcome.
partition orders-0
offset: 0 1 2 3 4 5 6 7 8 9 10 11
record: n T1 T1 T2 T1 C1 T3 n T3 A3 T2 ...
^ HW = 11
^ LSO = 3 (T2 is still open)
n = non-transactional T1, T3 = records of transactions 1 and 3
C1 = commit marker of T1 A3 = abort marker of T3
read_uncommitted sees: 0 1 2 3 4 6 7 8 10 (everything except the markers, including T2 and aborted T3)
read_committed sees: 0 1 2 then waits at offset 3 until T2 commits or aborts
after T2 commits: 3 4 7 10 ... (T3 at 6 and 8 filtered out, markers 5 and 9 skipped)
How filtering works
For a read_committed fetch, the leader returns data only up to the LSO, plus a list of aborted transactions (producer ID and first offset) that overlap the returned range. It reads that list from the segments' .txnindex files. The consumer then drops batches belonging to aborted transactions as it iterates. The broker never rewrites batches, which keeps zero-copy transfer intact (Section 7.1).
Consequences to plan for
- Offset gaps: control records take offsets, so offsets seen by the application are not contiguous, even on a non-compacted topic. Never compute "number of records" as an offset difference.
- Head-of-line blocking: an open transaction holds back everything after it on that partition, including non-transactional records and other producers' committed transactions, for up to
transaction.timeout.ms. Long transactions directly add latency for everyread_committedreader. - Lag looks different: a
read_committedconsumer cannot get past the LSO, so its lag reflects open transactions as well as slow processing. - Hanging transactions: a transaction that never completes (a client bug, or the pre-KIP-890 race described under Transaction Coordinator) pins the LSO. Consumers stall, and compaction and tiered storage uploads cannot pass that point.
kafka-transactions.shcan find and abort it.
Example: consumer configuration and hanging-transaction recovery
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); // hide open and aborted transactional data
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
# Transactions open longer than the max timeout (default 15 min) on partitions led by broker 1
bin/kafka-transactions.sh --bootstrap-server localhost:9092 find-hanging --broker-id 1
# Topic Partition ProducerId ProducerEpoch CoordinatorEpoch StartOffset LastTimestamp Duration(min)
# orders 0 7000 5 12 3 1761210942120 47
# Abort it by writing an ABORT marker; the LSO can then advance
bin/kafka-transactions.sh --bootstrap-server localhost:9092 \
abort --topic orders --partition 0 --start-offset 3
Key Takeaways
- Transactional records are in the log immediately.
isolation.leveldecides whether consumers see them before the outcome is known and whether aborted ones are hidden. read_committedreads up to the LSO. The consumer filters aborted batches using the aborted-transaction list returned with each fetch.- Markers occupy offsets, so offsets of visible records have gaps.
- An open transaction blocks
read_committedreaders of that partition. Keep transactions short. - Use
kafka-transactions.sh find-hangingandabortto clear a transaction that pins the LSO.
Practice
- (Beginner) For the log in the diagram, list what each isolation level returns if T2 aborts instead of committing.
- (Intermediate) A
read_committedconsumer group shows growing lag on partition 7 only, and the producers look healthy. Describe the commands and metrics you would use to tell slow processing apart from a hanging transaction.- Interview question: "Why does Kafka filter aborted transactional records in the consumer instead of having the broker remove them before sending?" Hint: think about how the broker sends data from disk to the network (Chapter 7), and what removing records would cost on every fetch.
Zombie Fencing
Theory
A zombie is an instance the rest of the system believes is dead but which is still running: a process stuck in a long GC pause, or a pod cut off by a network partition while its replacement starts. When it wakes up, it carries on with its work, and without protection two instances now write the same output and commit the same offsets. Fencing makes sure that only the newest instance can complete transactions, the same idea as controller epochs (Section 2.3) and leader epochs (Section 2.4).
Producer fencing with epochs
Fencing is tied to the transactional.id. When an instance calls initTransactions(), the coordinator:
- Bumps the epoch for that ID (for example, from 5 to 6).
- Aborts any transaction the previous epoch left open, writing abort markers with the new epoch, which also updates the producer state on those partitions.
- Returns the PID and new epoch to the new instance.
From then on, the old instance's requests carry epoch 5 and are rejected: by the coordinator (AddPartitionsToTxn, EndTxn) and by partition leaders (Produce). The zombie sees ProducerFencedException and must shut down.
time ->
instance A (epoch 5): begin send send [GC pause ..........................] commit -> ProducerFencedException
instance B (same ID): start initTransactions()
coordinator: epoch 5 -> 6, abort A's open txn
B: begin send commit (succeeds)
This only works if the replacement uses the same transactional.id as the instance it replaces. A random ID per process start (UUID.randomUUID()) fences nothing: the zombie keeps its epoch, and the dead instance's open transaction pins the LSO until transaction.timeout.ms expires. Derive the ID from a stable identity, such as a Kubernetes StatefulSet pod name (orders-2) or an assigned shard number.
Consumer-group fencing (KIP-447)
In a consume-transform-produce application there is a second kind of zombie. After a rebalance, partition orders-3 moves from instance A to instance B. Instances A and B have different transactional IDs, so epochs cannot fence A's work on orders-3.
- Before Kafka 2.5, the fix was one producer per input partition, with a
transactional.idderived from the partition (for example,app-orders-3). Whoever owned the partition used its ID and fenced the previous owner, but this meant one producer and one set of connections per partition. - Since Kafka 2.5 (KIP-447),
sendOffsetsToTransaction(offsets, consumer.groupMetadata())sends the consumer's group generation and member ID with the offsets. The group coordinator rejects a transactional offset commit from a member of an older generation, so a zombie's transaction can never commit. In addition, a new owner's offset fetch waits while a transactional offset commit for that partition is still pending, so it never starts from a stale position. One producer per application instance is enough.
Example: watching a fence happen
static KafkaProducer<String, String> newProducer(String txnId) {
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, txnId);
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
return new KafkaProducer<>(p);
}
KafkaProducer<String, String> a = newProducer("payments-0");
a.initTransactions(); // epoch N
a.beginTransaction();
a.send(new ProducerRecord<>("payments", "p-1", "charge 10"));
KafkaProducer<String, String> b = newProducer("payments-0"); // the "replacement" instance
b.initTransactions(); // epoch N+1: A's open transaction is aborted
try {
a.commitTransaction(); // A is now a zombie
} catch (ProducerFencedException e) {
System.out.println("A was fenced: " + e.getMessage());
a.close(); // the only valid reaction
}
b.beginTransaction();
b.send(new ProducerRecord<>("payments", "p-1", "charge 10"));
b.commitTransaction(); // exactly one "charge 10" is visible
Key Takeaways
- A zombie is a stale instance that still runs. Fencing ensures it cannot complete transactions.
initTransactions()bumps the epoch of thetransactional.idand aborts the previous owner's open transaction. Older epochs are rejected everywhere.- Fencing needs a
transactional.idthat is stable across restarts and unique among live instances. Random IDs defeat it. - Since Kafka 2.5, passing
consumer.groupMetadata()tosendOffsetsToTransactionalso fences zombies after rebalances, so one producer per instance is enough. ProducerFencedExceptionis fatal: close the producer and stop the instance's processing.
Practice
- (Beginner) Draw a timeline for two instances sharing a
transactional.idin which the zombie's records end up invisible toread_committedconsumers. Mark where the epoch changes.- (Intermediate) A service uses
transactional.id = "svc-" + UUID.randomUUID(). An instance crashes in the middle of a transaction and is restarted. Describe what downstreamread_committedconsumers experience, and for how long.- Interview question: "Kafka uses epochs for controllers, partition leaders, and producers. What single problem do they all solve, and why is a timeout alone not enough?" Hint: a paused process does not know it was paused, so the rest of the system must be able to reject its requests whenever they arrive.
<a id="83-exactly-once-semantics"></a>
8.3 Exactly-Once Semantics
This section combines idempotence and transactions into exactly-once processing: first the hand-written consume-transform-produce loop, then the one-line version in Kafka Streams, and finally the boundaries beyond which Kafka's guarantee no longer applies.
Consume-Transform-Produce Pattern
Theory
Most stream processing has the same shape: read records from input topics, compute something, write results to output topics, and record progress by committing offsets. Section 5.4 showed why committing offsets separately from producing cannot be exactly-once:
at-least-once loop: poll -> process -> produce results -> commit offsets
^ crash here: results are in the output topic,
offsets are not committed, so after restart the
records are processed again -> duplicate output
The fix is to make "results written" and "input consumed" a single atomic step. In Kafka, consumer offsets are themselves records in __consumer_offsets, so they can join a transaction like any other write. sendOffsetsToTransaction does exactly that. Either the output records and the new offsets are committed together, or neither is, and the next attempt starts from the old offsets with no visible output from the failed attempt.
+--------------------------- one transaction ---------------------------+
poll() -> | transform -> send(output-1) ... send(output-n) -> sendOffsetsToTransaction | -> commit
+-----------------------------------------------------------------------+
crash or abort anywhere inside: output aborted, offsets unchanged
This is what Kafka calls exactly-once semantics (EOS). Precisely: for a pipeline that reads from Kafka and writes to Kafka, the effect of each input record appears in the committed output exactly once, as seen by read_committed consumers. The processing code itself may run more than once (after a crash, a record is processed again), but only one attempt's results are ever committed. That is why it is also called effectively-once processing.
Ingredients
| Component | Setting | Why |
|---|---|---|
| Consumer | enable.auto.commit=false | Offsets must be committed only through the transaction |
| Consumer | isolation.level=read_committed | Do not consume aborted or open output of an upstream transactional stage |
| Producer | transactional.id stable per instance | Enables transactions and producer fencing |
| Loop | sendOffsetsToTransaction(offsets, consumer.groupMetadata()) | Commits offsets atomically with output and enables group fencing (KIP-447) |
| Loop | After abortTransaction(), seek back to the committed offsets | The consumer's in-memory position already moved past the records whose output was discarded |
The last row is easy to miss. Aborting the transaction rolls back the output and the offset commit, but not the consumer's position. If the loop simply polls again, it skips the records of the aborted batch, and they are lost.
The offsets committed are the next offsets to read: the last processed offset plus one, per partition. (The older sendOffsetsToTransaction(offsets, String groupId) overload, which cannot fence zombies, was deprecated and is removed in Kafka 4.0.)
Example: an exactly-once order validator (Java)
Properties c = new Properties();
c.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
c.put(ConsumerConfig.GROUP_ID_CONFIG, "order-validator");
c.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // offsets go through the transaction
c.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); // ignore aborted upstream data
c.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
c.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-validator-" + podName); // stable per instance
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(c);
KafkaProducer<String, String> producer = new KafkaProducer<>(p);
consumer.subscribe(List.of("orders"));
producer.initTransactions();
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
if (records.isEmpty()) continue;
producer.beginTransaction();
try {
for (ConsumerRecord<String, String> r : records) {
String target = isValid(r.value()) ? "orders-valid" : "orders-rejected";
producer.send(new ProducerRecord<>(target, r.key(), enrich(r.value())));
}
producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
producer.commitTransaction(); // output + offsets, atomically
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
producer.close(); // fatal: another instance took over
consumer.close();
throw e;
} catch (KafkaException e) { // includes CommitFailedException after a rebalance
producer.abortTransaction();
rewindToCommitted(consumer); // without this, the aborted records are skipped
}
}
static Map<TopicPartition, OffsetAndMetadata> nextOffsets(ConsumerRecords<String, String> records) {
Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
for (TopicPartition tp : records.partitions()) {
List<ConsumerRecord<String, String>> batch = records.records(tp);
long last = batch.get(batch.size() - 1).offset();
offsets.put(tp, new OffsetAndMetadata(last + 1)); // the NEXT offset to read
}
return offsets;
}
static void rewindToCommitted(KafkaConsumer<?, ?> consumer) {
Set<TopicPartition> assigned = consumer.assignment();
Map<TopicPartition, OffsetAndMetadata> committed = consumer.committed(assigned);
for (TopicPartition tp : assigned) {
OffsetAndMetadata om = committed.get(tp);
if (om != null) consumer.seek(tp, om);
else consumer.seekToBeginning(List.of(tp)); // no commit yet: apply your reset policy
}
}
Choosing the transaction size
One transaction per poll() is the simplest choice. Larger transactions (several polls, or a time window) reduce overhead: each commit costs a round of requests to the coordinator plus a marker per partition. Smaller transactions reduce end-to-end latency, because downstream read_committed consumers see nothing until the commit, and they shorten the head-of-line blocking described in Read Committed vs Read Uncommitted. Commit intervals of 100 ms to a few seconds are typical.
Key Takeaways
- Exactly-once for Kafka-to-Kafka pipelines means committing output records and input offsets in one transaction.
- Use
enable.auto.commit=false,read_committed, a stabletransactional.id, andsendOffsetsToTransaction(offsets, consumer.groupMetadata()). - Committed offsets are last processed offset + 1.
- After an abort, seek the consumer back to the committed offsets, or records are silently skipped.
- Processing may run more than once. Only one attempt's output is committed.
Practice
- (Beginner) Explain why the offsets passed to
sendOffsetsToTransactionarelast offset + 1, and what happens downstream if you pass the last offset itself.- (Intermediate) Change the example to commit one transaction per second instead of per poll. Measure throughput and end-to-end latency (producer timestamp to downstream
read_committedconsumer) for both versions.- Interview question: "A transactional pipeline sometimes aborts, logs the error, and continues polling. A reconciliation job finds that some input orders never produced any output. What is the bug?" Hint: an abort undoes the writes and the offset commit, but something else in the application moved forward and was not undone.
Exactly-Once in Kafka Streams
Theory
The hand-written loop gets complicated once you add state: a running count or a join table is held in a local state store, and every change to it is also written to a changelog topic (Chapter 10) so the store can be rebuilt after a failure. For exactly-once, the store's changes must commit atomically with the output and the offsets. Otherwise, a crash could leave a count that already includes a record whose output was rolled back.
Kafka Streams does all of this for you with one setting:
processing.guarantee=exactly_once_v2
Each stream thread then uses one transactional producer, and every commit includes, in one transaction:
+---------------------- one Kafka Streams commit (one transaction) ----------------------+
| output topic records e.g. order-counts |
| changelog records e.g. order-enricher-orders-per-customer-changelog |
| repartition topic records e.g. order-enricher-KSTREAM-...-repartition |
| input offsets for every source and repartition topic partition read |
+-----------------------------------------------------------------------------------------+
Streams also sets the consumer to read_committed, disables auto-commit, and handles fencing, aborts, and rewinding. The only visible API is the configuration.
Versions of the setting
| Value | Status | How it works |
|---|---|---|
at_least_once | Default | Offsets committed after output is flushed. Duplicates possible after failures |
exactly_once (v1) | Deprecated in 3.0, removed in 4.0 | One producer per task, transactional.id derived from the task. Many producers, high overhead |
exactly_once_beta | Renamed to v2 in 3.0, removed in 4.0 | Early name of v2 |
exactly_once_v2 | Recommended (brokers 2.5 or later) | One producer per stream thread, fenced through consumer group metadata (KIP-447) |
What changes when you enable it
commit.interval.msdrops from 30000 to 100 ms by default, because downstreamread_committedconsumers see output only after each commit. Raising it improves throughput and increases latency.- State after a crash: a local store may contain writes from an aborted transaction. On an unclean shutdown under EOS, Streams wipes the affected stores and restores them from their changelogs (read with
read_committed). For large stores this can take minutes, so configurenum.standby.replicasto keep warm copies on other instances. Work on transactional state stores (KIP-892) aims to remove the need for this wipe. - Broker requirements: production clusters need at least 3 brokers, since the internal transaction log and Streams' internal topics should use replication factor 3 with
min.insync.replicas=2. - Overhead: extra coordinator requests and markers per commit. With the 100 ms default, throughput is typically a little lower than with
at_least_once.
Example: an exactly-once counting application
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-enricher"); // also the consumer group ID
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);
props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3); // internal topics
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1); // faster recovery after a wipe
// props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100); // the EOS default, shown for clarity
StreamsBuilder builder = new StreamsBuilder();
builder.stream("orders", Consumed.with(Serdes.String(), Serdes.String()))
.groupByKey() // key = customer ID
.count(Materialized.as("orders-per-customer")) // state store + changelog
.toStream()
.to("order-counts", Produced.with(Serdes.String(), Serdes.Long()));
KafkaStreams streams = new KafkaStreams(builder.build(), props);
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
streams.start();
Consumers of order-counts must use isolation.level=read_committed. Otherwise they can see counts from aborted attempts, and the guarantee does not reach them.
Key Takeaways
processing.guarantee=exactly_once_v2makes output, changelog, repartition writes, and input offsets commit in one transaction per thread.exactly_onceandexactly_once_betaare removed in Kafka 4.0. Useexactly_once_v2, which needs brokers 2.5 or later.- The default
commit.interval.msbecomes 100 ms. It trades throughput against end-to-end latency. - After an unclean failure, local state is wiped and restored from changelogs. Standby replicas shorten recovery.
- Downstream consumers must read with
read_committed.
Practice
- (Beginner) List every setting that changes, explicitly or by default, when an application switches from
at_least_oncetoexactly_once_v2.- (Intermediate) Run the counting application, kill it with
kill -9while producing input, restart it, and verify with aread_committedconsole consumer that no count skips or repeats. Repeat withat_least_onceand compare.- Interview question: "A Kafka Streams application with
exactly_once_v2sends a confirmation email in aforeachfor every order. Can a customer receive two emails? Why?" Hint: separate what is inside the transaction (Kafka writes and offsets) from what the processing code does to the outside world each time it runs.
End-to-End Exactly-Once Limitations
Theory
Kafka's exactly-once guarantee is real but scoped. It covers state that Kafka itself stores and commits atomically: records in topics, consumer offsets, and Streams changelogs. Anything outside that boundary is not covered by transactions and must be made safe by other means.
[ source system ] --1--> [ producer ] ==> [ Kafka: topics + offsets + changelogs ] ==> [ consumer ] --3--> [ sink ]
^ exactly-once inside this box ^
2: side effects during processing (HTTP, email, DB)
| Boundary | What can go wrong | How to handle it |
|---|---|---|
| 1. Into Kafka | A client retries an HTTP request, or the producing service crashes after sending but before recording that it did, and sends again. Idempotence cannot link the two sends | Generate a business ID at the source (order ID, UUID) and deduplicate on it. For data in a database, use the transactional outbox with CDC (Chapter 9) |
| 2. Side effects during processing | Processing runs again after an abort or crash, so external calls repeat | Pass idempotency keys to external APIs, or move side effects to a downstream consumer of the committed output |
| 3. Out of Kafka | A sink writes to a database and then crashes before committing offsets, so the write is repeated | Idempotent writes (upsert by key), or store offsets in the sink in the same transaction as the data (example below) |
| Readers | A read_uncommitted consumer sees aborted and open data | Use read_committed everywhere downstream of transactional producers |
| Across clusters | Transactions are scoped to one cluster. MirrorMaker 2 replication is at-least-once | Deduplicate after replication, or design consumers to be idempotent |
| Data loss settings | unclean.leader.election.enable=true, acks=1, or min.insync.replicas=1 can lose acknowledged data, and with it committed output or offsets | Keep RF 3, min.insync.replicas=2, acks=all, unclean election off |
| Non-deterministic processing | A retried attempt may compute a different result (current time, random values, external lookups). Only one result is committed, but side effects of the other may already have happened | Make processing deterministic from record content, for example by using event time instead of wall-clock time |
| Timeouts | A transaction exceeding transaction.timeout.ms is aborted. An idle idempotent producer's state expires after producer.id.expiration.ms | Keep transactions short. Treat expiration as an at-least-once boundary |
Connectors
Kafka Connect extends the boundary at both ends where connectors support it:
- Source connectors: since Kafka 3.3 (KIP-618), distributed workers with
exactly.once.source.support=enabledwrite source records and source offsets in one transaction. Each connector can require it withexactly.once.support=required, which fails if the connector cannot provide it. - Sink connectors: exactly-once depends on the connector. Some write idempotently (upserts by key), and others store Kafka offsets in the target system and resume from them (Chapter 9).
"Exactly-once delivery" versus what Kafka promises
In a distributed system with failures, no protocol can guarantee that a message is delivered to and acted on by a remote party exactly once, because the sender can never be sure an action happened if the acknowledgment is lost (the two generals problem). Kafka sidesteps this by not promising delivery at all. It promises atomic commits of state it owns: retries happen, but the committed result reflects each input exactly once. To extend that to an external system, the external system must also take part, by being idempotent or by committing its data together with the Kafka offsets.
Example: an exactly-once sink by storing offsets in the database
import psycopg
from confluent_kafka import Consumer, OFFSET_BEGINNING
db = psycopg.connect("dbname=shop", autocommit=True) # explicit transactions below
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "order-sink",
"enable.auto.commit": False, # Kafka's committed offsets are not the source of truth
"isolation.level": "read_committed", # never apply aborted upstream records
})
def on_assign(c, partitions):
# Resume each partition from the offset stored with the data, not from __consumer_offsets
with db.cursor() as cur:
for p in partitions:
cur.execute("SELECT next_offset FROM kafka_offsets WHERE topic = %s AND part = %s",
(p.topic, p.partition))
row = cur.fetchone()
p.offset = row[0] if row else OFFSET_BEGINNING
c.assign(partitions)
consumer.subscribe(["orders-valid"], on_assign=on_assign)
while True:
msg = consumer.poll(1.0)
if msg is None or msg.error():
continue
with db.transaction(): # data and offset commit together, or not at all
db.execute(
"INSERT INTO orders (id, body) VALUES (%s, %s) "
"ON CONFLICT (id) DO UPDATE SET body = EXCLUDED.body", # idempotent as a second safety net
(msg.key().decode(), msg.value().decode()))
db.execute(
"INSERT INTO kafka_offsets (topic, part, next_offset) VALUES (%s, %s, %s) "
"ON CONFLICT (topic, part) DO UPDATE SET next_offset = EXCLUDED.next_offset",
(msg.topic(), msg.partition(), msg.offset() + 1))
If the process crashes before the database transaction commits, neither the row nor the offset is saved, and the record is read again. If it crashes after the commit, the stored offset already points past the record. Committing to Kafka as well (asynchronously) is still useful so lag monitoring keeps working, but those offsets are not used for recovery.
Key Takeaways
- Kafka's exactly-once covers topics, offsets, and changelogs committed in one cluster, as seen by
read_committedconsumers. - Entry into Kafka, side effects during processing, and writes to external sinks are outside the guarantee.
- Extend it with business IDs and deduplication at the source, idempotency keys for external calls, and idempotent or offset-storing sinks.
- Connect source connectors can be exactly-once since 3.3 (KIP-618). Sink connectors depend on their implementation.
- Data loss settings, cross-cluster replication, and non-deterministic processing all weaken the end-to-end result.
Practice
- (Beginner) For each case, state whether Kafka transactions alone prevent a duplicate: a Streams app retrying after a crash; an HTTP client retrying a POST that produces a record; a sink connector writing rows with plain
INSERT; aread_uncommittedconsumer after an abort.- (Intermediate) Design a pipeline from a
payments-requestedtopic to an external payment provider that guarantees a customer is never charged twice, even if the calling service crashes after the provider accepted the charge. Describe the keys, the stored state, and the recovery steps.- Interview question: "Is exactly-once delivery possible in a distributed system? What exactly does Kafka guarantee when it advertises exactly-once semantics?" Hint: separate delivering a message from committing the effect of processing it, and consider which side of the system must remember what has already been done.
<a id="9-kafka-connect"></a>