3. Getting Started and Operations Basics
Chapters 1 and 2 explained what Kafka is and how it works inside. This chapter is hands-on: installing Kafka, running it as a single node and as a multi-node cluster, using the command-line tools that come with every distribution, and managing topics day to day. These skills come before everything else in the guide. Every later experiment, from producer tuning to transactions, starts with a running cluster and a correctly configured topic, and most production incidents are first investigated with the same tools introduced here.
<a id="31-installation-and-setup"></a>
3.1 Installation and Setup
This section takes you from an empty machine to a working cluster, first with the plain Apache distribution, then with Docker, and finally with several nodes that replicate data between them.
Installing Kafka Locally
Theory
Kafka is a JVM application distributed as a single archive. "Installing" it means unpacking that archive and making sure a compatible Java runtime is available. No installer, system service, or external database is needed. Since Kafka 4.0 there is no ZooKeeper either, because KRaft (Section 2.3) is the only metadata mode.
Analogy: installing Kafka is like unpacking a portable toolbox rather than renovating a house. Everything is in one folder, it runs from wherever you put it, and deleting the folder (plus its data directory) removes it completely.
Prerequisites
| Component | Requirement (Kafka 4.x) | Notes |
|---|---|---|
| Java for brokers and controllers | Java 17 or newer | Kafka 4.0 dropped Java 8 and 11 for the server side. Use an LTS build (17 or 21). |
| Java for clients, Streams, Connect | Java 11+ for clients and Streams, Java 17 for Connect and tools | Your applications may run an older JVM than your brokers. |
| Operating system | Linux for production. macOS or Linux for development. | On Windows, use WSL2 or Docker. The bin\windows\*.bat scripts exist, but Windows file locking has historically caused problems when Kafka deletes log segments. |
| Disk | A few GB for development | log.dirs holds all topic data. In production, use dedicated disks. |
Downloading and verifying the archive
Releases are published at https://kafka.apache.org/downloads. Every archive name contains two versions: the Scala version the broker was compiled with and the Kafka version. Kafka 4.x builds only for Scala 2.13, so the archive is kafka_2.13-<version>.tgz. The Scala version does not matter to you unless you write Scala code that links against Kafka's server JARs.
java -version # must report 17 or newer
KAFKA_VERSION=4.1.0 # check the downloads page for the latest release
SCALA_VERSION=2.13
ARCHIVE="kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"
# Current releases live on downloads.apache.org; older ones move to archive.apache.org/dist/kafka/
curl -fLO "https://downloads.apache.org/kafka/${KAFKA_VERSION}/${ARCHIVE}"
curl -fLO "https://downloads.apache.org/kafka/${KAFKA_VERSION}/${ARCHIVE}.sha512"
# Apache publishes checksums in gpg --print-md format, so compare with the same tool
gpg --print-md SHA512 "${ARCHIVE}" | diff - "${ARCHIVE}.sha512" && echo "checksum OK"
tar -xzf "${ARCHIVE}"
cd "kafka_${SCALA_VERSION}-${KAFKA_VERSION}"
bin/kafka-topics.sh --version # prints the Kafka version, e.g. 4.1.0
For stronger assurance, also verify the .asc signature against the project's KEYS file with gpg --verify. A checksum only proves the download is intact. A signature proves who produced it.
Package managers are convenient for development but can lag behind releases:
brew install kafka # macOS/Linux Homebrew; config under $(brew --prefix)/etc/kafka
brew services start kafka # runs a single node in the background
What is in the distribution
kafka_2.13-4.1.0/
|-- bin/ shell scripts: servers and every CLI tool (Section 3.2)
| |-- kafka-server-start.sh
| |-- kafka-storage.sh
| |-- kafka-topics.sh kafka-console-producer.sh kafka-console-consumer.sh ...
| `-- windows/ .bat equivalents
|-- config/
| |-- server.properties combined broker+controller node (development)
| |-- broker.properties broker-only node
| |-- controller.properties controller-only node
| |-- producer.properties consumer.properties connect-*.properties
| `-- log4j2.yaml application logging (Log4j 2 since Kafka 4.0)
|-- libs/ Kafka and dependency JARs
|-- licenses/
`-- site-docs/ offline copy of the documentation
Two kinds of "logs"
Kafka's documentation uses the word log for two unrelated things, and beginners regularly delete the wrong one:
| Data logs | Application logs | |
|---|---|---|
| What | Topic partitions: segment, index, and checkpoint files (Section 2.1) | Text files of what the server did: server.log, controller.log, state-change.log |
| Configured by | log.dirs in the properties file | LOG_DIR environment variable (default <kafka>/logs) and log4j2.yaml |
| Safe to delete? | No. This is your data. | Yes. Only diagnostics are lost. |
Environment variables read by the scripts
| Variable | Purpose | Example |
|---|---|---|
KAFKA_HEAP_OPTS | JVM heap size. The server default is 1 GB. | -Xms6g -Xmx6g (production brokers rarely need more than 6 to 8 GB, because data is cached in the OS page cache, not the heap) |
KAFKA_OPTS | Extra JVM flags such as Java agents or JAAS files | -javaagent:/opt/jmx_exporter.jar=7071:/opt/jmx.yaml |
LOG_DIR | Where application logs go | /var/log/kafka |
JMX_PORT | Opens a JMX port for metrics tools | 9999 |
KAFKA_JVM_PERFORMANCE_OPTS | Garbage collector flags (G1 by default) | Rarely changed |
Example: making the tools easy to call
# ~/.bashrc or ~/.zshrc
export KAFKA_HOME="$HOME/opt/kafka_2.13-4.1.0"
export PATH="$KAFKA_HOME/bin:$PATH"
Now kafka-topics.sh works from any directory. The rest of this guide writes bin/kafka-topics.sh to make clear where the tools live.
Key Takeaways
- Kafka is a self-contained archive. Installation is download, verify, and extract. Kafka 4.x needs Java 17+ on the server and no ZooKeeper.
- The archive name
kafka_2.13-4.1.0means Scala 2.13 build of Kafka 4.1.0. Only the second number matters to most users. bin/holds the servers and all CLI tools.config/holds sample properties for combined, broker-only, and controller-only nodes.log.dirsholds topic data.LOG_DIRholds application logs. Do not confuse them.- Size the heap modestly. Kafka relies on the OS page cache for performance.
Practice
- (Beginner) Download a Kafka release, verify its checksum, and print the version with
bin/kafka-topics.sh --version. Then list the files inconfig/and explain the difference betweenserver.properties,broker.properties, andcontroller.properties.- (Intermediate) A colleague freed disk space on a development machine by running
rm -rfon "the Kafka logs folder", and afterwards every topic was gone. Explain which directory they most likely deleted, how you would tell the two directories apart on that machine, and how to prevent the confusion in the future.- Interview question: "Why do Kafka brokers typically run with a heap of only a few gigabytes, even on machines with 64 GB or more of RAM?" Hint: think about where recently written and read segment data is cached, and what a very large heap does to garbage collection pauses.
Running Kafka with Docker
Theory
Docker removes the "works on my machine" problem: the image already contains the right Java version, the Kafka distribution, and a startup script that turns environment variables into configuration. It is the fastest way to get a disposable cluster for development, demos, and integration tests. The one Kafka-specific complication is networking. A container has its own hostname and network, so advertised.listeners must be set carefully (see Broker Discovery and Bootstrap Servers in Section 2.2).
Analogy: running Kafka in Docker is like renting a fully furnished apartment instead of buying furniture yourself. You move in immediately, but you still need to give visitors the right street address (the advertised listener), not the internal room number.
Choosing an image
| Image | Maintainer | Notes |
|---|---|---|
apache/kafka | Apache Kafka project | Official JVM image, KRaft only. With no configuration it starts a single combined node on localhost:9092. Tools are in /opt/kafka/bin. |
apache/kafka-native | Apache Kafka project | GraalVM native build that starts in well under a second with a small memory footprint. Intended for local development and tests, not production. |
confluentinc/cp-kafka | Confluent | Confluent Platform's packaging of Apache Kafka, with similar environment-variable conventions. Common in Confluent-based stacks. |
bitnami/kafka | Broadcom (Bitnami) | Widely used in older tutorials. In 2025 Bitnami moved most free images to a legacy repository that no longer receives updates, so check its status before relying on it. |
This guide uses apache/kafka. Always pin an exact tag (apache/kafka:4.1.0), never latest, so a teammate's machine and your CI run the same version.
Configuration through environment variables
The apache/kafka image turns every environment variable that starts with KAFKA_ into a broker property:
| Rule | Property | Environment variable |
|---|---|---|
Prefix KAFKA_, uppercase, . becomes _ | offsets.topic.replication.factor | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR |
_ becomes __ (double underscore) | listener.name.external_ssl.ssl.keystore.location (listener named EXTERNAL_SSL) | KAFKA_LISTENER_NAME_EXTERNAL__SSL_SSL_KEYSTORE_LOCATION |
- becomes ___ (triple underscore) | listener.name.internal.scram-sha-256.sasl.jaas.config | KAFKA_LISTENER_NAME_INTERNAL_SCRAM___SHA___256_SASL_JAAS_CONFIG |
| Cluster ID (not a property, used by the format step) | (none) | CLUSTER_ID |
Important: if you pass any configuration through environment variables (or a mounted properties file), the image no longer applies its built-in defaults. You must then provide a complete, consistent configuration, as in the Compose file below.
Example: the 30-second quick start
# Start a single combined broker+controller node; port 9092 is published to the host
docker run -d --name kafka -p 9092:9092 apache/kafka:4.1.0
# Run the CLI tools that ship inside the image
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 --create --topic quickstart
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 --describe --topic quickstart
docker rm -f kafka # stop and remove; all data is gone
This works from the host because the default configuration advertises localhost:9092, and the published port maps the host's localhost:9092 to the container. Another container, however, cannot use localhost to reach it, which is why real projects use a Compose file with two listeners.
Example: a single-node Compose file for development
# docker-compose.yml
services:
kafka:
image: apache/kafka:4.1.0
hostname: kafka
container_name: kafka
ports:
- "9092:9092" # host -> EXTERNAL listener
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller # combined mode (development only)
KAFKA_LISTENERS: INTERNAL://:29092,EXTERNAL://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
# Internal topics default to replication factor 3, impossible with one broker
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 # faster consumer startup in development
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false" # typos in topic names fail loudly
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: 4L6g3nShT-eMCtK--X86sw # any ID from kafka-storage.sh random-uuid
volumes:
- kafka-data:/var/lib/kafka/data # data survives container re-creation
healthcheck:
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
interval: 10s
timeout: 10s
retries: 10
orders-app: # an application in the same Compose network
image: example/orders-app:dev
environment:
BOOTSTRAP_SERVERS: kafka:29092 # containers use the INTERNAL listener
depends_on:
kafka:
condition: service_healthy # wait until the broker answers requests
volumes:
kafka-data:
host machine Docker network
+--------------------------+ +-------------------------------------+
| IDE / app on the host | | kafka container |
| bootstrap=localhost:9092 |--9092--->| EXTERNAL :9092 -> advertises |
+--------------------------+ | localhost:9092 |
| INTERNAL :29092 -> advertises |
+--------------------------+ | kafka:29092 |
| orders-app container |--29092-->| CONTROLLER :9093 (KRaft only) |
| bootstrap=kafka:29092 | +-------------------------------------+
+--------------------------+
docker compose up -d
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
docker compose down # stop; the named volume keeps the data
docker compose down -v # stop and delete the volume: a completely fresh cluster next time
On first start, the image formats the storage directory with CLUSTER_ID. On later starts it finds the existing meta.properties in the volume and skips formatting. If you change CLUSTER_ID while keeping the old volume, the broker refuses to start because the IDs do not match.
Example: Kafka in integration tests
Testcontainers starts a throwaway broker from JUnit and hands your test the correct bootstrap address (Chapter 15 covers testing in depth):
import org.testcontainers.kafka.KafkaContainer; // module org.testcontainers:kafka (1.20+)
class OrderPipelineIT {
@Test
void publishesOrderEvents() throws Exception {
// Native image: starts in well under a second, ideal for tests
try (KafkaContainer kafka = new KafkaContainer("apache/kafka-native:4.1.0")) {
kafka.start();
String bootstrap = kafka.getBootstrapServers(); // e.g. PLAINTEXT://localhost:54721
// ... create producer/consumer with this bootstrap address and assert on results
}
}
}
Key Takeaways
apache/kafkais the official image. Pin an exact version tag.apache/kafka-nativeis for development and tests only.- Environment variables starting with
KAFKA_map to properties (.to_,_to__,-to___). Supplying any of them disables the image defaults. - Use two listeners: one advertising the container hostname for other containers, one advertising
localhostfor the host. - On a single broker, lower the replication factor of internal topics to 1, or consumer groups and transactions will not work.
- Mount a volume at
log.dirsif data should survive re-creating the container.docker compose down -vgives you a clean slate.
Practice
- (Beginner) Start Kafka with the one-line
docker runcommand, create a topic, and list topics from inside the container. Then remove the container and start it again. Is the topic still there? Explain why.- (Intermediate) Using the Compose file above, remove the
EXTERNALlistener so onlyINTERNAL://kafka:29092is advertised, and publish port 29092 instead. Run a client on the host withbootstrap.servers=localhost:29092. Describe what happens, and explain it using the metadata response.- Interview question: "Your service's integration tests pass on developers' laptops but time out in CI, where tests run inside a container that talks to Kafka in another container." Hint: compare the address the client bootstraps to with the address the broker advertises in each environment.
Starting a Single-Node Cluster
Theory
A single-node cluster runs one process that is both broker (stores partitions, serves clients) and controller (manages metadata through a one-member Raft quorum). This is called combined mode (process.roles=broker,controller). It has no fault tolerance, since every partition has exactly one replica, but it behaves like a real cluster from a client's point of view. That makes it ideal for learning and for developing applications.
Analogy: a single-node cluster is a one-person company. The same person is the CEO (controller) and the only employee (broker). Everything works, decisions are fast, but if that person is sick, the company is closed.
Before a KRaft node can start for the first time, its storage directory must be formatted. Formatting writes a meta.properties file containing the cluster ID and node ID. This is a safety check: a node will not accidentally join the wrong cluster, or start on an empty disk and believe it has no data.
The startup sequence
kafka-storage.sh format kafka-server-start.sh
----------------------- ---------------------
1. reads node.id and log.dirs 4. reads meta.properties, checks cluster ID
2. writes meta.properties 5. controller starts the Raft quorum (1 voter)
(cluster.id, node.id, directory.id) and becomes leader of __cluster_metadata
3. writes the bootstrap metadata 6. broker registers with the controller
(initial metadata.version, single voter) 7. broker catches up on metadata and is unfenced
8. listeners open: clients can connect on :9092
Essential configuration
The shipped config/server.properties is already set up for a single combined node. These are the lines that matter:
process.roles=broker,controller # combined mode
node.id=1 # unique per node in the cluster
# Raft quorum: either a static voter list or bootstrap servers for a dynamic quorum
controller.quorum.bootstrap.servers=localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093 # where the process listens
advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
log.dirs=/tmp/kraft-combined-logs # move this out of /tmp for anything you want to keep
num.partitions=1 # default partitions for new topics
# Internal topics would default to replication factor 3, impossible with one broker
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
share.coordinator.state.topic.replication.factor=1
share.coordinator.state.topic.min.isr=1
Example: format, start, verify, stop
cd "$KAFKA_HOME"
# 1. Generate a cluster ID (once per cluster) and format the storage directory
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties
# --standalone: this node is the only voter of a new dynamic controller quorum
cat /tmp/kraft-combined-logs/meta.properties
# node.id=1
# directory.id=pgCGpQ6sQ2SgOgYIcc8mEw
# version=1
# cluster.id=4L6g3nShT-eMCtK--X86sw
# 2. Start in the foreground (Ctrl+C stops it), or add -daemon to run in the background
bin/kafka-server-start.sh config/server.properties
# ... [KafkaRaftServer nodeId=1] Kafka Server started
In a second terminal:
# 3. Verify: the controller quorum has a leader, and the broker answers API requests
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
# ClusterId: 4L6g3nShT-eMCtK--X86sw
# LeaderId: 1
# LeaderEpoch: 1
# HighWatermark: 152
# ...
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 | head -1
# localhost:9092 (id: 1 rack: null) -> ( one entry per broker, followed by its supported API versions
# 4. Smoke test: create a topic, write a record, read it back
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic smoke-test
echo "hello kafka" | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic smoke-test
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic smoke-test \
--from-beginning --max-messages 1
# hello kafka
# 5. Stop gracefully. Note: this script stops EVERY Kafka server process on the machine
bin/kafka-server-stop.sh
# 6. Reset to a completely empty cluster (development only)
rm -rf /tmp/kraft-combined-logs
A graceful stop (SIGTERM) lets Kafka perform a controlled shutdown: it moves leadership away, flushes data, and writes a clean-shutdown marker so the next start does not need to rebuild indexes. Avoid kill -9 except as a last resort.
Common startup problems
| Symptom (messages abridged) | Cause | Fix |
|---|---|---|
No meta.properties found in /tmp/kraft-combined-logs (have you run kafka-storage.sh to format the directory?) | Storage not formatted | Run kafka-storage.sh format |
Invalid cluster.id ... Expected X, but read Y | Directory was formatted for a different cluster | Use the original cluster ID, or wipe and re-format (development only) |
Log directory ... is already formatted when formatting | Formatting twice | Add --ignore-formatted, or leave the existing format alone |
Address already in use | Another process holds 9092 or 9093 | Stop it or change the ports in listeners |
| Server starts, but clients hang or time out | advertised.listeners not reachable from the client | Advertise an address the client can resolve and reach |
Consumers never join a group (COORDINATOR_NOT_AVAILABLE) | offsets.topic.replication.factor is greater than the number of brokers | Set it to 1 on a single node, then delete log.dirs and re-format so __consumer_offsets can be created |
| Every topic disappears after a reboot | log.dirs is under /tmp, which the OS cleans | Point log.dirs at a persistent directory |
Key Takeaways
- Combined mode runs broker and controller in one process. It is perfect for development but has no fault tolerance.
- A KRaft node must be formatted before its first start. Formatting writes
meta.propertieswith the cluster ID and node ID. - The shipped
config/server.propertiesworks for one node, but its data lives in/tmpby default. - Verify a new cluster with
kafka-metadata-quorum.sh describe --statusand a produce/consume smoke test. - Stop with SIGTERM (
kafka-server-stop.sh) so the broker can perform a controlled shutdown.
Practice
- (Beginner) Format and start a single node, run the smoke test above, then stop the node. Inspect
log.dirsand identify the directory that holds partition 0 ofsmoke-testand the files inside it.- (Intermediate) Copy
config/server.properties, changelog.dirsto a directory in your home folder, and setnum.partitions=3. Format with a new cluster ID, start the node, create a topic without--partitions, and confirm how many partitions it has. Then try starting the node against the directory with a different cluster ID and record the error.- Interview question: "Why does KRaft require an explicit format step with a cluster ID, when ZooKeeper-based brokers could simply be started?" Hint: consider what happens if a broker with an empty or replaced disk starts and joins a cluster, or if a node is accidentally pointed at another cluster's controllers.
Starting a Multi-Node Cluster
Theory
Replication, failover, leader election, and ISR behavior (Section 2.4) cannot be observed on one node. A multi-node cluster lets you set a replication factor of 3, stop a broker, and watch the cluster recover. For production, multiple nodes are not optional: they are what makes Kafka fault tolerant.
Analogy: going from one node to several is like going from a solo musician to an orchestra. More can be played, and the show goes on if one violinist is absent, but now there is a conductor (the active controller), everyone must follow the same score (the cluster ID and quorum configuration), and each player needs their own seat (node ID, ports, data directory).
Topologies
| Combined mode | Isolated mode | |
|---|---|---|
| Roles | Each node is broker and controller | Separate controller nodes and broker nodes |
| Typical size | 1 to 3 nodes | 3 or 5 controllers plus any number of brokers |
| Use | Development, testing, small edge deployments | Production |
| Pros | Fewer processes, simpler setup | Controllers are not slowed by broker load. Brokers can be scaled or restarted independently. |
| Cons | Heavy client traffic can delay metadata work. You cannot scale brokers without changing the quorum. | More processes to run |
The controller quorum needs a majority to make progress, so its size determines how many controller failures the cluster survives:
| Controllers | Majority | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
Use 3 controllers for most clusters and 5 when you need to survive two simultaneous controller failures (or one failure during maintenance). An even number adds cost without adding tolerance.
What must be the same and what must differ on every node
| Must be identical on every node | Must be unique per node |
|---|---|
| Cluster ID used to format storage | node.id (brokers and controllers share one ID space) |
controller.quorum.voters or controller.quorum.bootstrap.servers | listeners and advertised.listeners (host and/or port) |
controller.listener.names and the listener security protocol map | log.dirs (when several nodes share a machine) |
| Kafka version (except during a rolling upgrade) | LOG_DIR for application logs (when several nodes share a machine) |
Example A: three combined-mode nodes on one machine (no Docker)
cd "$KAFKA_HOME"
CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
VOTERS="1@localhost:19093,2@localhost:29093,3@localhost:39093" # static quorum: id@host:port
for i in 1 2 3; do
cat > "/tmp/kafka-node-$i.properties" <<EOF
process.roles=broker,controller
node.id=$i
controller.quorum.voters=$VOTERS
listeners=PLAINTEXT://localhost:${i}9092,CONTROLLER://localhost:${i}9093
advertised.listeners=PLAINTEXT://localhost:${i}9092,CONTROLLER://localhost:${i}9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
log.dirs=$HOME/kafka-lab/data/node-$i
default.replication.factor=3
min.insync.replicas=2
EOF
# Same cluster ID for every node; no --standalone because the voter set is static
bin/kafka-storage.sh format -t "$CLUSTER_ID" -c "/tmp/kafka-node-$i.properties"
# Separate application-log directory per node, then start in the background
LOG_DIR="$HOME/kafka-lab/logs/node-$i" \
bin/kafka-server-start.sh -daemon "/tmp/kafka-node-$i.properties"
done
Node 1 listens on ports 19092 and 19093, node 2 on 29092 and 29093, and node 3 on 39092 and 39093. To stop a single node, target its process instead of using kafka-server-stop.sh, which would stop all three:
pkill -f kafka-node-2.properties # SIGTERM -> controlled shutdown of node 2 only
Example B: isolated mode with Docker Compose (3 controllers, 3 brokers)
YAML anchors (&name / <<: *name) keep the shared settings in one place:
# docker-compose.cluster.yml
x-common-env: &common-env
CLUSTER_ID: 4L6g3nShT-eMCtK--X86sw
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
x-controller-env: &controller-env
<<: *common-env
KAFKA_PROCESS_ROLES: controller
KAFKA_LISTENERS: CONTROLLER://:9093
x-broker-env: &broker-env
<<: *common-env
KAFKA_PROCESS_ROLES: broker
KAFKA_LISTENERS: INTERNAL://:29092,EXTERNAL://:9092
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
services:
controller-1: { image: "apache/kafka:4.1.0", hostname: controller-1, environment: { <<: *controller-env, KAFKA_NODE_ID: 1 } }
controller-2: { image: "apache/kafka:4.1.0", hostname: controller-2, environment: { <<: *controller-env, KAFKA_NODE_ID: 2 } }
controller-3: { image: "apache/kafka:4.1.0", hostname: controller-3, environment: { <<: *controller-env, KAFKA_NODE_ID: 3 } }
broker-1:
image: apache/kafka:4.1.0
hostname: broker-1
ports: ["19092:9092"] # each broker needs its own host port
environment:
<<: *broker-env
KAFKA_NODE_ID: 4 # must not clash with controller IDs
KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker-1:29092,EXTERNAL://localhost:19092
depends_on: [controller-1, controller-2, controller-3]
broker-2:
image: apache/kafka:4.1.0
hostname: broker-2
ports: ["29092:9092"]
environment:
<<: *broker-env
KAFKA_NODE_ID: 5
KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker-2:29092,EXTERNAL://localhost:29092
depends_on: [controller-1, controller-2, controller-3]
broker-3:
image: apache/kafka:4.1.0
hostname: broker-3
ports: ["39092:9092"]
environment:
<<: *broker-env
KAFKA_NODE_ID: 6
KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker-3:29092,EXTERNAL://localhost:39092
depends_on: [controller-1, controller-2, controller-3]
+--------------+ +--------------+ +--------------+
controller quorum | controller-1 | | controller-2 | | controller-3 | Raft voters, :9093
+------+-------+ +------+-------+ +------+-------+
| __cluster_metadata (fetched by brokers)
+------v-------+ +------v-------+ +------v-------+
brokers | broker-1 | | broker-2 | | broker-3 | observers
| id 4 | | id 5 | | id 6 |
+------+-------+ +------+-------+ +------+-------+
host ports 19092 29092 39092
^ ^ ^
host clients: bootstrap.servers=localhost:19092,localhost:29092
Note that each broker's EXTERNAL listener must advertise a different host port. If all three advertised localhost:9092, clients on the host would send every request to whichever broker owns that port.
Example: verifying the cluster and watching a failover
BS=localhost:19092,localhost:29092,localhost:39092 # bootstrap list for the Compose cluster
# Quorum health: who is the active controller, and are followers caught up?
bin/kafka-metadata-quorum.sh --bootstrap-server $BS describe --status
# LeaderId: 2
# LeaderEpoch: 1
# CurrentVoters: [1,2,3] (abridged; recent versions print each voter's endpoints)
# CurrentObservers: [4,5,6]
bin/kafka-metadata-quorum.sh --bootstrap-server $BS describe --replication
# NodeId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status (abridged)
# 2 1721 0 1761210942120 1761210942120 Leader
# 1 1721 0 ... Follower
# 3 1721 0 ... Follower
# 4 1721 0 ... Observer
# ...
# A replicated topic
bin/kafka-topics.sh --bootstrap-server $BS --create --topic payments --partitions 3
bin/kafka-topics.sh --bootstrap-server $BS --describe --topic payments
# Topic: payments Partition: 0 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6 ...
# Kill one broker and look again: leadership moves and the ISR shrinks
docker compose -f docker-compose.cluster.yml stop broker-1
bin/kafka-topics.sh --bootstrap-server localhost:29092 --describe --topic payments
# Topic: payments Partition: 0 Leader: 5 Replicas: 4,5,6 Isr: 5,6 ...
docker compose -f docker-compose.cluster.yml start broker-1 # rejoins, catches up, re-enters ISR
Production clusters follow the same principles, with one machine (or Kubernetes pod) per node, persistent disks, security on every listener (Chapter 11), rack awareness (Section 2.2), and usually a dynamic quorum (controller.quorum.bootstrap.servers) so controllers can be replaced without downtime (Section 2.3). Kubernetes operators such as Strimzi automate much of this (Chapter 14).
Key Takeaways
- Use combined mode for local multi-node experiments and isolated mode (dedicated controllers) for production.
- Run 3 or 5 controllers. The quorum needs a majority, so even numbers add no fault tolerance.
- Every node shares the cluster ID and quorum configuration. Each has a unique
node.id, its own listeners, and its own data directory. - Each broker must advertise its own reachable address. Duplicate advertised addresses break client routing.
- Verify with
kafka-metadata-quorum.sh describe --statusand--replication, then test failover by stopping a broker and describing a replicated topic.
Practice
- (Beginner) Start the three-node combined cluster from Example A. Create a topic with 3 partitions and replication factor 3, and identify the leader of each partition.
- (Intermediate) In the same cluster, stop the node that is the active controller. Use
kafka-metadata-quorum.shto find the new leader and itsLeaderEpoch. Then stop a second node and try to create a topic. Explain the result in terms of quorum majority.- Interview question: "Why is it recommended to run dedicated controller nodes in production rather than combined broker+controller nodes?" Hint: think about resource contention between client traffic and metadata operations, and about scaling or restarting brokers without touching the quorum.
<a id="32-command-line-tools"></a>
3.2 Command-Line Tools
Kafka ships with a set of shell scripts in bin/ that cover most everyday administration. This section covers the five you will use constantly.
kafka-topics
Theory
kafka-topics.sh creates, lists, describes, alters, and deletes topics. Like every modern Kafka tool, it is a thin wrapper around the Admin API: it connects to the brokers given in --bootstrap-server, discovers the cluster through metadata, and sends admin requests that the controller executes. It never edits files on disk directly, so you can run it from any machine that can reach the cluster.
Analogy: the CLI tools are like a bank's teller window. You never walk into the vault (the broker's disks). You hand a form (an admin request) to the teller (any broker), and the bank's back office (the controller) carries it out.
Conventions shared by every tool
| Convention | Meaning |
|---|---|
--bootstrap-server host:port[,host:port] | How to reach the cluster. Since Kafka 4.0 there are no --zookeeper options. |
--command-config client.properties | Client settings for the tool's admin client: security, timeouts. Used by kafka-topics, kafka-configs, kafka-consumer-groups, and others. |
--producer.config / --consumer.config | The equivalent for the console producer and consumer. Recent releases (KIP-1147) are standardizing on --command-config and print deprecation warnings for some older option names. |
--help | Every tool documents all of its options. |
| JVM startup | Each call starts a JVM (about 1 second). For scripts that make hundreds of changes, use the Admin API from code instead. |
A typical client.properties for a secured cluster (Chapter 11 explains each line):
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ops-admin" password="<secret>";
ssl.truststore.location=/etc/kafka/truststore.p12
ssl.truststore.password=<secret>
Main operations
| Operation | Command (after bin/kafka-topics.sh --bootstrap-server $BS) | Notes |
|---|---|---|
| Create | --create --topic orders --partitions 6 --replication-factor 3 --config min.insync.replicas=2 | Partitions and RF default to num.partitions and default.replication.factor. Add --if-not-exists in scripts. |
| List | --list | Add --exclude-internal to hide __consumer_offsets and similar. |
| Describe | --describe --topic orders | Without --topic, describes every topic. |
| Increase partitions | --alter --topic orders --partitions 12 | Can only increase (Section 3.3). |
| Delete | --delete --topic orders | Asynchronous. Requires delete.topic.enable=true (the default). |
| Explicit placement | --create --topic t --replica-assignment 4:5:6,5:6:4,6:4:5 | One entry per partition, the first broker is the preferred leader. Rarely needed. |
Health filters for --describe
These options turn kafka-topics.sh into a quick health check of the whole cluster:
| Option | Shows partitions where | Concern |
|---|---|---|
--under-replicated-partitions | ISR is smaller than the replica list | A replica is down or lagging |
--under-min-isr-partitions | ISR is smaller than min.insync.replicas | acks=all writes are failing right now |
--at-min-isr-partitions | ISR equals min.insync.replicas | One more failure stops acks=all writes |
--unavailable-partitions | There is no leader | Partition is offline for reads and writes |
--topics-with-overrides | (Lists topics) that have topic-level config overrides | Useful for auditing |
Example: create and describe
BS=localhost:19092
bin/kafka-topics.sh --bootstrap-server $BS --create --if-not-exists \
--topic orders --partitions 3 --replication-factor 3 \
--config min.insync.replicas=2 --config retention.ms=259200000 # 3 days
bin/kafka-topics.sh --bootstrap-server $BS --describe --topic orders
Topic: orders TopicId: q8bGv1TfRXSX0n3b1n7Y2w PartitionCount: 3 ReplicationFactor: 3 Configs: min.insync.replicas=2,retention.ms=259200000
Topic: orders Partition: 0 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6 Elr: LastKnownElr:
Topic: orders Partition: 1 Leader: 5 Replicas: 5,6,4 Isr: 5,6,4 Elr: LastKnownElr:
Topic: orders Partition: 2 Leader: 6 Replicas: 6,4,5 Isr: 6,4,5 Elr: LastKnownElr:
How to read it:
- TopicId: a unique ID that stays the same for the life of the topic. A deleted and recreated topic with the same name gets a new ID.
- Configs: only the topic-level overrides, not the inherited defaults.
- Replicas: the assigned brokers, with the preferred leader first. Kafka spreads preferred leaders across brokers (4, 5, 6) to balance load.
- Isr: replicas that are fully caught up (Section 2.4). If this is shorter than
Replicas, the partition is under-replicated. - Elr / LastKnownElr: Eligible Leader Replicas (KIP-966, Section 2.4). They are empty when the ISR is healthy.
Example: a one-line cluster health check
for f in --unavailable-partitions --under-min-isr-partitions --under-replicated-partitions; do
echo "== $f"
bin/kafka-topics.sh --bootstrap-server $BS --describe $f # prints nothing when healthy
done
A pitfall: --topic is a regular expression
For --describe, --alter, and --delete, the --topic value is treated as a regex. A . matches any character, so --delete --topic orders.v1 would also delete orders_v1 or ordersXv1 if they exist. Check first with --list --topic 'orders.v1', or escape the dot ('orders\.v1').
The same with the Admin API
from confluent_kafka.admin import AdminClient
admin = AdminClient({"bootstrap.servers": "localhost:19092"})
md = admin.list_topics(timeout=10)
for name, t in sorted(md.topics.items()):
if name.startswith("__"):
continue # like --exclude-internal
under = [p.id for p in t.partitions.values() if len(p.isrs) < len(p.replicas)]
print(f"{name}: {len(t.partitions)} partitions, under-replicated: {under or 'none'}")
Key Takeaways
kafka-topics.shuses the Admin API through--bootstrap-server. It can run from any machine that can reach the cluster.- Use
--command-configfor secured clusters and--if-not-exists/--if-existsin scripts. --describeshows overrides, replicas (preferred leader first), and ISR. The health filters find under-replicated, under-min-ISR, and leaderless partitions.--topicis a regex for describe, alter, and delete. Beware of dots in topic names.
Practice
- (Beginner) Create a topic
inventorywith 4 partitions and replication factor 2 on a three-broker cluster. Describe it, and for each partition name the preferred leader and the follower.- (Intermediate) Stop one broker and run each health filter in the table. Predict which partitions each filter will print before running it, then compare. Repeat after setting
min.insync.replicas=3on the topic.- Interview question: "Your monitoring reports zero under-replicated partitions, yet a producer using
acks=allis gettingNotEnoughReplicaserrors. How can both be true, and which command would show the problem?" Hint: compare the replica list, the ISR, andmin.insync.replicasfor the affected topic, and consider what each metric actually counts.
kafka-console-producer
Theory
kafka-console-producer.sh reads lines from standard input and sends each line as one record. It uses the normal Java producer, so every producer setting (Chapter 4) applies. It is the quickest way to put test data into a topic, reproduce a bug with a specific key, or pipe a file into Kafka.
Analogy: the console producer is a microphone plugged straight into a topic. Everything you say (each line you type) is broadcast as one message.
How a line becomes a record
input line: trace-id:abc123,source:web|CUST-77:{"orderId":"ORD-1","total":39.98}
\________________________/ \_____/ \______________________________/
headers (parse.headers) key value
split on headers.delimiter | split on key.separator :
Important options
| Option | Purpose |
|---|---|
--topic orders | Destination topic |
--property parse.key=true | Treat the text before key.separator as the key |
--property key.separator=: | Key/value separator (default is a tab) |
--property parse.headers=true | Parse headers at the start of each line |
--property headers.delimiter=| | Separates headers from the rest (default tab) |
--property headers.separator=, / headers.key.separator=: | Between headers / between a header key and its value |
--property null.marker=NULL | A key or value equal to this text is sent as null, for example to write tombstones |
--producer-property acks=all | Set any producer config inline (repeatable) |
--producer.config file | Load producer and security settings from a file |
--compression-codec zstd | Compress batches |
--sync | Wait for each send to be acknowledged (slow, but shows errors immediately) |
Recent releases (KIP-1147) are renaming some of these for consistency, for example --property to --reader-property. The old names keep working during a deprecation period.
Example: typing records interactively
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
--property parse.key=true --property key.separator=: \
--producer-property acks=all
>CUST-77:{"orderId":"ORD-1","total":39.98}
>CUST-12:{"orderId":"ORD-2","total":5.00}
>CUST-77:{"orderId":"ORD-3","total":12.50}
# Ctrl+D (end of input) flushes the remaining records and exits
Because both CUST-77 records have the same key, they land in the same partition and keep their order (Section 2.1).
Example: bulk loading, headers, and tombstones
# Load a file: one record per line
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
--property parse.key=true --property key.separator=: < orders.txt
# Headers + key + value on each line
printf 'trace-id:abc123,source:web|CUST-77:{"orderId":"ORD-4"}\n' | \
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
--property parse.headers=true --property 'headers.delimiter=|' \
--property parse.key=true --property key.separator=:
# A tombstone (null value) for key CUST-12 on a compacted topic
echo 'CUST-12:NULL' | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
--topic customer-profiles --property parse.key=true --property key.separator=: \
--property null.marker=NULL
# Check how many records each partition now holds (latest offset per partition)
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders --time -1
# orders:0:3
# orders:1:2
# orders:2:0
Limitations
- Every value is a plain string from a text line. Binary data and formats such as Avro or Protobuf need a schema-aware tool (for example
kafka-avro-console-producerfrom Confluent, Chapter 6) orkcat. - A line cannot contain a newline, so multi-line JSON must be compacted to one line first (for example with
jq -c). - It is a test and debugging tool, not a data pipeline. For moving data from files or databases, use Kafka Connect (Chapter 9).
Key Takeaways
- Each input line becomes one record. With
parse.keyandkey.separator, part of the line becomes the key. - Headers, null values (
null.marker), and any producer setting (--producer-property) are available for realistic tests. - Redirect a file into the tool for quick bulk loads, and check results with
kafka-get-offsets.sh. - Use schema-aware tools or
kcatfor binary or schema-based data.
Practice
- (Beginner) Produce five records with keys
A,B,A,C,Ato a 3-partition topic. Usekafka-get-offsets.shto see how many records landed in each partition, and explain why allArecords are together.- (Intermediate) Produce a record with two headers (
trace-idandschema-version) and a key, then verify all three with the console consumer (next topic). Then send a tombstone for the same key usingnull.marker.- Interview question: "You need to replay 10,000 failed messages captured in a file back into a topic, preserving their original keys. How would you do it quickly from the command line, and what would you check before and after?" Hint: think about key parsing, the separator appearing inside values,
acks, and how to confirm the record count afterwards.
kafka-console-consumer
Theory
kafka-console-consumer.sh reads records from a topic and prints them to standard output. It is the "eyes" of Kafka debugging: what is actually in this topic, what are the keys, which partition did a record go to, are the headers right? It uses the normal Java consumer (Chapter 5), so it can join a consumer group and commit offsets, or read on its own without affecting anyone.
Analogy: if the console producer is a microphone, the console consumer is a pair of headphones. You can listen live from now on, or rewind the tape to the beginning, since Kafka keeps the recording.
Where reading starts
| Options | Starts at | Typical use |
|---|---|---|
| (none) | The end: only records written after the tool starts | Watching live traffic |
--from-beginning | The earliest retained offset | Inspecting all data in a topic |
--partition 2 --offset 1500 | Exact offset 1500 of partition 2 | Inspecting one specific record |
--partition 2 --offset earliest or latest | Start or end of that single partition | Looking at one partition |
--group order-audit | The group's committed offsets (or auto.offset.reset if none) | Behaving like a real consumer group |
Without --group, the tool generates a throwaway group ID such as console-consumer-12345, so it does not interfere with your applications. With --group, it commits offsets and takes part in rebalances, so never reuse the group ID of a production application, or the tool will steal partitions from it and move its committed offsets.
Controlling the output
Property (--property name=value) | Effect |
|---|---|
print.key=true | Print the key |
print.partition=true / print.offset=true | Print where the record is stored |
print.timestamp=true | Print CreateTime:... or LogAppendTime:... |
print.headers=true | Print headers (NO_HEADERS when there are none) |
key.separator=" | " | Text between printed fields |
null.literal=<tombstone> | Text printed for null keys or values |
Other useful options: --max-messages N (exit after N records), --timeout-ms 5000 (exit if no record arrives for 5 seconds), --include 'orders\..*' (regex over topic names, replacing the removed --whitelist), --isolation-level read_committed (hide aborted transactional records, Chapter 8), --key-deserializer / --value-deserializer, and --formatter for internal topics.
Example: inspecting records in detail
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
--from-beginning --max-messages 4 \
--property print.timestamp=true --property print.partition=true \
--property print.offset=true --property print.headers=true --property print.key=true
CreateTime:1761210942000 Partition:0 Offset:0 NO_HEADERS CUST-77 {"orderId":"ORD-1","total":39.98}
CreateTime:1761210950412 Partition:0 Offset:1 NO_HEADERS CUST-77 {"orderId":"ORD-3","total":12.50}
CreateTime:1761210961003 Partition:0 Offset:2 trace-id:abc123,source:web CUST-77 {"orderId":"ORD-4"}
CreateTime:1761210946730 Partition:1 Offset:0 NO_HEADERS CUST-12 {"orderId":"ORD-2","total":5.00}
Processed a total of 4 messages
Notice that the output is not globally ordered by time: the consumer reads partitions independently, and order is guaranteed only within a partition.
Example: everyday debugging recipes
# Look at one exact record reported in an error log ("partition 0, offset 2")
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
--partition 0 --offset 2 --max-messages 1 --property print.key=true
# Search a topic for one customer's events (fine for small topics only)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
--from-beginning --timeout-ms 10000 --property print.key=true | grep '^CUST-77'
# Pretty-print JSON values with jq
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
--from-beginning --max-messages 10 | jq .
# Read as a named group so that progress (and lag) is tracked
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
--group order-audit --from-beginning
--from-beginning only matters when the group has no committed offset yet. Run the last command twice, and the second run continues where the first stopped instead of starting over.
Key Takeaways
- By default the console consumer starts at the end of the topic. Use
--from-beginningor--partition/--offsetto read history. print.*properties reveal keys, partitions, offsets, timestamps, and headers, which is what you need for debugging.- Without
--groupthe tool is a harmless observer. With--groupit commits offsets and joins rebalances, so never use an application's group ID. - Output is ordered within each partition, not across partitions.
Practice
- (Beginner) Read all records of a topic and print key, partition, and offset. Verify that records with the same key appear in the same partition in the order they were produced.
- (Intermediate) Run the console consumer twice with
--group test-group --from-beginning, producing new records between runs. Explain why the second run does not start from the beginning, and how you would make it do so (seekafka-consumer-groupsnext).- Interview question: "A developer debugging production joins the console consumer to the group
payment-processorto 'see what the service sees'. Shortly afterwards, the payment service reports missing messages. What happened?" Hint: think about partition assignment within a group and what the console consumer does with offsets for the partitions it receives.
kafka-consumer-groups
Theory
Consumer groups (Chapter 5) track their progress as committed offsets stored in Kafka. kafka-consumer-groups.sh is the tool for seeing and changing that state: which groups exist, which members own which partitions, how far behind each group is (lag), and, when needed, moving a group's offsets to reprocess or skip data.
Analogy: a consumer group's committed offsets are bookmarks in a shared book (the topic). This tool lets you see where each reader's bookmark is, how many pages they have left, and, with the reader's permission, move the bookmark.
Reading the describe output
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
order-service orders 0 18234 18240 6 order-service-1-5f8c... /10.0.3.17 order-service-1
order-service orders 1 17990 17990 0 order-service-1-5f8c... /10.0.3.17 order-service-1
order-service orders 2 9120 21502 12382 order-service-2-a91d... /10.0.3.18 order-service-2
| Column | Meaning |
|---|---|
| CURRENT-OFFSET | The group's committed offset: the next record it will read |
| LOG-END-OFFSET | The latest readable offset in the partition. Despite the name, this is the high watermark (Section 2.4), not the leader's raw log end. |
| LAG | LOG-END-OFFSET minus CURRENT-OFFSET: records written but not yet processed |
| CONSUMER-ID / HOST / CLIENT-ID | The member that currently owns the partition. - means no active owner. |
Here partition 2 has a lag of over 12,000 while the others are near zero. The problem is specific to order-service-2 (a slow or stuck instance, or a hot partition), not to the whole group.
Other views:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service --state
# GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
# order-service broker-2:9092 (5) uniform Stable 2
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service --members --verbose
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups
Common states are Stable (working normally), Empty (no active members, but offsets are kept), PreparingRebalance / CompletingRebalance (classic protocol) or Assigning / Reconciling (new consumer protocol, KIP-848, Section 5.2), and Dead (the group's metadata has been removed). Since Kafka 4.0, kafka-groups.sh --list lists groups of every type, including share groups (Section 5.4).
Resetting offsets
Resetting moves a group's bookmarks. It requires the group to be inactive (state Empty): stop every consumer in the group first, or the tool refuses with an error saying the group is not inactive.
| Option | New position |
|---|---|
--to-earliest / --to-latest | Start / end of each partition |
--to-offset 5000 | Exact offset |
--shift-by -100 | Current position moved by N (negative means go back) |
--to-datetime 2026-10-22T08:00:00.000 | First offset with a timestamp at or after this time (uses the time index) |
--by-duration PT2H | Same, relative: 2 hours ago |
--from-file offsets.csv | Per-partition offsets from a CSV (topic,partition,offset) |
Scope with --topic orders, --topic orders:0,2 (only partitions 0 and 2), or --all-topics. Every reset is a dry run unless you add --execute.
Example: reprocessing the last two hours after a bug fix
# 1. Stop all instances of the application, then confirm the group is Empty
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service --state
# 2. Preview (dry run is the default)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group order-service \
--topic orders --reset-offsets --by-duration PT2H
# GROUP TOPIC PARTITION NEW-OFFSET
# order-service orders 0 16012
# order-service orders 1 15877
# order-service orders 2 8001
# 3. Save the current offsets so you can undo, then apply
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group order-service \
--topic orders --reset-offsets --to-current --export > order-service-before.csv
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group order-service \
--topic orders --reset-offsets --by-duration PT2H --execute
# 4. Restart the application; it resumes from the new offsets
Reprocessing means the application sees records a second time, so it must be idempotent (Section 5.4) or its side effects must be safe to repeat.
Deleting
# Remove a group that is no longer used (must be Empty)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group old-reporting-job
# Remove only the group's offsets for one topic (e.g., the group stopped reading it)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete-offsets \
--group order-service --topic legacy-orders
Empty groups are also cleaned up automatically once their offsets expire (offsets.retention.minutes, default 7 days).
Key Takeaways
--describe --groupshows committed offset, latest offset, and lag per partition, plus which member owns it. Look for lag concentrated in one partition or one member.--stateand--membersshow the group's health and composition.- Offset resets require an inactive group, are dry runs by default, and support absolute, relative, and time-based targets. Export the current offsets first so you can undo.
- Reprocessing duplicates side effects unless consumers are idempotent.
Practice
- (Beginner) Start a console consumer with
--group lab-group, produce 20 records, stop the consumer after it has read 10 (use--max-messages 10), and describe the group. Explain each column for one partition.- (Intermediate) Reset
lab-groupto the offsets of 5 minutes ago with a dry run, then with--execute. Try the same while a consumer in the group is running and record the error. Finally, undo the reset using the exported CSV.- Interview question: "Lag for a consumer group is steadily growing on one partition but zero on all others. List possible causes and how you would narrow them down." Hint: compare the producer's key distribution, the member owning that partition, the processing time of specific records, and whether the consumer is stuck on one record.
kafka-configs
Theory
Many Kafka settings can be changed while the cluster is running, without restarts: a topic's retention, a broker's thread pool size, a client's quota, the log level of one class. These dynamic configurations are stored in the cluster metadata, so they apply cluster-wide and survive restarts. kafka-configs.sh reads and writes them.
Analogy:
server.propertiesis the building's blueprint, which you change only during renovation (a restart). Dynamic configs are the thermostat and light switches: you adjust them while people are inside, and the change takes effect immediately.
Entity types
--entity-type | Shortcut | Example settings |
|---|---|---|
topics | --topic orders | retention.ms, cleanup.policy, min.insync.replicas |
brokers | --broker 4 / --broker-defaults | log.retention.ms, num.io.threads, log.cleaner.threads, replication throttles |
broker-loggers | (none) | Log level of a class, e.g. kafka.server.ReplicaManager=DEBUG |
users / clients / ips | --user, --client | Quotas: producer_byte_rate, consumer_byte_rate, request_percentage, connection rates; SCRAM credentials |
groups | --group | Group-level overrides such as consumer.session.timeout.ms (KIP-848) |
client-metrics | (none) | Client telemetry subscriptions (KIP-714) |
Where a broker setting comes from (precedence)
highest 1. Dynamic per-broker config kafka-configs --entity-type brokers --entity-name 4
| 2. Dynamic cluster-wide default kafka-configs --entity-type brokers --entity-default
| 3. Static config server.properties on that broker
lowest 4. Kafka's built-in default
Topic configs sit on top of this chain: a topic-level override wins over all broker-level values, and a topic without an override inherits the broker value (Section 3.3).
Not every broker setting can be changed dynamically. The documentation marks each broker config with an update mode: read-only (restart needed), per-broker, or cluster-wide.
Example: topics
BS=localhost:9092
# Show only the overrides (and where values come from)
bin/kafka-configs.sh --bootstrap-server $BS --describe --topic orders
# Dynamic configs for topic orders are:
# retention.ms=259200000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=259200000, DEFAULT_CONFIG:log.retention.hours=168}
# Show every effective value, including inherited defaults
bin/kafka-configs.sh --bootstrap-server $BS --describe --topic orders --all
# Change one value and remove an override (reverting to the inherited default)
bin/kafka-configs.sh --bootstrap-server $BS --alter --topic orders \
--add-config retention.ms=604800000,max.message.bytes=2097152
bin/kafka-configs.sh --bootstrap-server $BS --alter --topic orders \
--delete-config max.message.bytes
# Values that contain commas go in square brackets
bin/kafka-configs.sh --bootstrap-server $BS --alter --topic orders \
--add-config 'cleanup.policy=[compact,delete]'
The synonyms list shows the full inheritance chain: the topic override retention.ms wins over the broker default log.retention.hours=168.
Example: brokers, loggers, and quotas
# Cluster-wide dynamic default: applies to every broker without a per-broker override
bin/kafka-configs.sh --bootstrap-server $BS --alter --broker-defaults \
--add-config log.retention.ms=259200000
# One broker only: more I/O threads on broker 4
bin/kafka-configs.sh --bootstrap-server $BS --alter --broker 4 --add-config num.io.threads=16
# Temporarily turn on DEBUG logging for one class on one broker, then turn it off again
bin/kafka-configs.sh --bootstrap-server $BS --alter --entity-type broker-loggers --entity-name 4 \
--add-config kafka.server.ReplicaManager=DEBUG
bin/kafka-configs.sh --bootstrap-server $BS --alter --entity-type broker-loggers --entity-name 4 \
--delete-config kafka.server.ReplicaManager
# Limit a noisy client to 10 MB/s of produce traffic per broker
bin/kafka-configs.sh --bootstrap-server $BS --alter --entity-type clients --entity-name batch-loader \
--add-config producer_byte_rate=10485760
Example: the same with the Java Admin API
try (Admin admin = Admin.create(Map.of(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"))) {
ConfigResource topic = new ConfigResource(ConfigResource.Type.TOPIC, "orders");
// Incremental changes: SET one value, DELETE another (others are left untouched)
Collection<AlterConfigOp> ops = List.of(
new AlterConfigOp(new ConfigEntry("retention.ms", "604800000"), AlterConfigOp.OpType.SET),
new AlterConfigOp(new ConfigEntry("max.message.bytes", ""), AlterConfigOp.OpType.DELETE));
admin.incrementalAlterConfigs(Map.of(topic, ops)).all().get();
// Print every value that does not come from the built-in default, with its source
Config config = admin.describeConfigs(List.of(topic)).all().get().get(topic);
for (ConfigEntry e : config.entries()) {
if (e.source() != ConfigEntry.ConfigSource.DEFAULT_CONFIG) {
System.out.printf("%s=%s (%s)%n", e.name(), e.value(), e.source()); // e.g. DYNAMIC_TOPIC_CONFIG
}
}
}
Always use incrementalAlterConfigs. The older alterConfigs API replaces the entire set of overrides for the resource, silently removing any you did not include.
Key Takeaways
- Dynamic configs change topics, brokers, loggers, quotas, and groups at runtime, without restarts. They are stored in cluster metadata.
- Broker precedence: per-broker dynamic, then cluster-wide dynamic default, then
server.properties, then built-in default. Topic overrides win over all of them. --describeshows overrides only. Add--allfor effective values.synonymsreveal where a value comes from.- Use
--add-configand--delete-config(orincrementalAlterConfigsin code). Wrap comma-containing values in[...].
Practice
- (Beginner) Set
retention.mson a topic to one hour, describe the topic's configs, then delete the override and describe again. Explain thesynonymsoutput in both cases.- (Intermediate) Set
log.retention.msas a cluster-wide dynamic default, then set a different value on one broker with--broker. Use--describe --broker <id> --allon two brokers to show which value each uses, and explain the precedence order.- Interview question: "A teammate changed
log.retention.hoursinserver.propertieson all brokers and did a rolling restart, but disk usage did not change for most topics. What could explain this?" Hint: check for topic-level overrides and dynamic broker configs, and remember which of them take precedence over the static file.
<a id="33-topic-management"></a>
3.3 Topic Management
Topics are the main thing you design and operate in Kafka, and some topic decisions, such as the partition count of a keyed topic, are expensive to change later. This section covers their lifecycle and the decisions that matter most.
Creating and Deleting Topics
Theory
Creating a topic tells the controller to pick a name, a number of partitions, a replication factor, and a set of brokers for each replica, and to record all of it in the metadata log. The brokers then create the partition directories. Deleting reverses this: the topic disappears from metadata, and brokers remove its files in the background.
Analogy: creating a topic is like opening a new account at a bank. You choose the account type (configuration), the bank assigns an account number (topic ID) and branches that keep copies (replicas). Closing the account is quick at the counter, but the archived paperwork is shredded later.
Explicit creation vs auto-creation
By default, brokers create a topic automatically when a client asks for metadata about a topic that does not exist (auto.create.topics.enable=true). The topic gets num.partitions and default.replication.factor.
| Explicit creation | Auto-creation | |
|---|---|---|
| Who decides partitions, RF, configs | You, per topic | Broker defaults for every topic |
Typos (oders instead of orders) | Fail with UNKNOWN_TOPIC_OR_PARTITION | Silently create a new, wrong topic |
| Review and governance | Possible (CLI, Terraform, operators, CI) | None |
| Recommendation | Production and shared clusters | Disable in production (auto.create.topics.enable=false) |
Naming rules and conventions
- Allowed characters: letters, digits,
.,_, and-. Maximum length 249. The names.and..are not allowed. .and_collide in metric names, soorders.createdandorders_createdcannot coexist. Kafka warns when you create a name containing either character. Pick one separator and use it everywhere.- Names starting with
__are reserved by convention for internal topics. - A common convention is
<domain>.<entity>.<event-type>or<team>.<dataset>.<version>, for examplesales.orders.createdorpayments.transactions.v2(topic naming is covered in Chapter 15).
What happens on deletion
kafka-topics --delete --topic orders
|
v
1. Controller writes RemoveTopicRecord to __cluster_metadata -> topic disappears for clients
2. Each broker renames the partition directories:
orders-0 -> orders-0.<uuid>-delete
3. After file.delete.delay.ms (default 60 s) the files are removed from disk
Pitfalls of deletion:
- Recreating with the same name produces a new topic (new topic ID) with offsets starting at 0. Consumer groups may still have committed offsets for the old topic's name. Those offsets are now invalid (often beyond the new log end), so consumers fall back to
auto.offset.resetand may skip or reprocess data. Delete the groups' offsets (kafka-consumer-groups --delete-offsets) when you recreate a topic. - Auto-creation can resurrect a deleted topic if any producer or consumer is still running against it.
- Deleting to "empty" a topic is heavy-handed. To discard old data but keep the topic, use
kafka-delete-records.sh, or lowerretention.mstemporarily.
Example: creating, deleting, and purging with the CLI
BS=localhost:9092
bin/kafka-topics.sh --bootstrap-server $BS --create --if-not-exists \
--topic sales.orders.created --partitions 12 --replication-factor 3 \
--config min.insync.replicas=2
bin/kafka-topics.sh --bootstrap-server $BS --delete --if-exists --topic sales.orders.test
# Purge records before offset 500 in partition 0, keeping the topic
cat > purge.json <<'EOF'
{"version": 1, "partitions": [{"topic": "sales.orders.created", "partition": 0, "offset": 500}]}
EOF
bin/kafka-delete-records.sh --bootstrap-server $BS --offset-json-file purge.json
# partition: sales.orders.created-0 low_watermark: 500
Example: idempotent topic creation from code
Applications and deployment pipelines often ensure their topics exist at startup. The code must tolerate "already exists":
from confluent_kafka import KafkaError, KafkaException
from confluent_kafka.admin import AdminClient, NewTopic
admin = AdminClient({"bootstrap.servers": "localhost:9092"})
wanted = [
NewTopic("sales.orders.created", num_partitions=12, replication_factor=3,
config={"min.insync.replicas": "2", "retention.ms": str(7 * 24 * 3600 * 1000)}),
NewTopic("sales.orders.dlq", num_partitions=3, replication_factor=3),
]
# create_topics is asynchronous: it returns one future per topic
for name, future in admin.create_topics(wanted, operation_timeout=30).items():
try:
future.result() # raises on failure
print(f"created {name}")
except KafkaException as e:
err = e.args[0]
if err.code() == KafkaError.TOPIC_ALREADY_EXISTS:
print(f"{name} already exists") # fine: creation is idempotent
else:
raise # e.g. INVALID_REPLICATION_FACTOR, POLICY_VIOLATION
Note that "already exists" does not mean "exists with the configuration you asked for". A robust tool also describes the existing topic and reports differences. This is what declarative tools such as Terraform providers, Strimzi's KafkaTopic resources, or Julie Ops do.
Key Takeaways
- Create topics explicitly and disable auto-creation in production, so partitions, RF, and configs are deliberate and typos fail loudly.
- Use letters, digits, and one separator style. Do not mix
.and_. Avoid the__prefix. - Deletion is asynchronous. A recreated topic is a new topic with offsets starting at 0, and old consumer group offsets must be cleaned up.
- To discard data but keep the topic, use
kafka-delete-records.shor temporarily lower retention.
Practice
- (Beginner) Create a topic
lab.eventswith 6 partitions, produce some records, delete it, and recreate it. Compare the topic IDs before and after.- (Intermediate) With a consumer group that has committed offsets around 1,000 on the old
lab.events, recreate the topic as above and start the consumer withauto.offset.reset=latest. Produce 50 records. How many does the consumer process, and why? Fix the situation without changing the consumer configuration.- Interview question: "Why do many organizations disable
auto.create.topics.enablein production, and how do they manage topics instead?" Hint: consider typos, default partition counts and replication factors, ownership, review processes, and infrastructure-as-code tools.
Choosing Partition Count
Theory
The partition is Kafka's unit of parallelism (Section 2.1). A topic with 6 partitions can be written by many producers but read in parallel by at most 6 consumers in one group. Too few partitions cap throughput. Too many waste resources and slow down recovery. And for keyed topics, the number is hard to change later, because adding partitions moves keys (next-but-one topic). The partition count is therefore one of the few Kafka decisions worth getting roughly right up front.
Analogy: partitions are checkout lanes in a supermarket. More lanes let more customers pay at once, but each lane needs a cashier, space, and electricity. Opening new lanes later is possible, but regular customers who always used lane 3 (keys) suddenly find themselves sent to a different lane.
Throughput-based sizing
Measure (or estimate) how much one partition can handle in your environment, for producing (p) and for consuming with your real processing logic (c). For a target throughput T:
partitions >= max( T / p , T / c )
The consumer side usually dominates, because processing (database writes, HTTP calls) is slower than Kafka itself.
import math
def partitions_needed(target_mb_s: float, produce_mb_s_per_partition: float,
consume_mb_s_per_partition: float, growth_factor: float = 2.0) -> int:
"""Throughput-based estimate, scaled for expected growth over the topic's lifetime."""
base = max(target_mb_s / produce_mb_s_per_partition,
target_mb_s / consume_mb_s_per_partition)
return math.ceil(base * growth_factor)
# 50 MB/s today; one partition can absorb 25 MB/s of produce traffic, but one consumer
# instance with its database writes handles only 5 MB/s
print(partitions_needed(50, 25, 5)) # -> 20; round to 24, which divides evenly by more group sizes
Other inputs to the decision
| Factor | Pushes the count | Why |
|---|---|---|
| Maximum consumer instances in a group | Up | In a classic consumer group, members beyond the partition count sit idle. (Share groups, Section 5.4, lift this limit.) |
| Future growth of a keyed topic | Up, from the start | Adding partitions later moves keys and breaks per-key ordering during the transition |
| Even load per consumer | Pick a number with many divisors | 12 partitions divide evenly among 1, 2, 3, 4, 6, or 12 consumers. 10 does not divide evenly among 3 or 4. |
| Low-volume topics | Down | Each partition costs memory, file handles, and metadata, even when idle. A few partitions are plenty. |
| Broker and cluster partition limits | Down | Many replicas per broker means more open files, more replication fetches, and slower leader failover and restarts |
End-to-end latency with acks=all | Down (mildly) | Followers replicate many partitions over a limited number of fetcher threads |
Rules of thumb
- Low-volume topics (a few MB/s or less): 3 to 6 partitions.
- Typical business event streams: 6 to 24 partitions.
- High-throughput topics: size with the formula, measured on your hardware, often 30 to 100+.
- Keep the total replica count per broker in the low thousands unless you have tested higher. KRaft raised cluster-wide limits dramatically (Section 2.3), but per-broker costs still apply.
- Never rely on
num.partitions=1(the broker default) for anything that needs parallelism.
Example: why divisors matter
12 partitions, 5 consumers: 12 partitions, 4 consumers:
c1: p0 p1 p2 c1: p0 p1 p2
c2: p3 p4 p5 c2: p3 p4 p5
c3: p6 p7 c3: p6 p7 p8
c4: p8 p9 c4: p9 p10 p11
c5: p10 p11
-> c1, c2 carry 50% more load -> perfectly even
Key Takeaways
- Partitions bound parallelism: a classic group's useful consumers are limited to the partition count.
- Size with
max(T/p, T/c)using measured per-partition rates, then add headroom for growth, especially for keyed topics. - Prefer counts with many divisors (6, 12, 24, 48) so load spreads evenly.
- More partitions cost memory, file handles, replication work, and recovery time. Do not over-partition low-volume topics.
Practice
- (Beginner) A topic receives 8 MB/s. One partition handles 10 MB/s for producing and one consumer handles 2 MB/s. How many partitions do you need today, and what number would you choose to allow for doubling traffic?
- (Intermediate) Use
kafka-producer-perf-test.shandkafka-consumer-perf-test.shon your local cluster to measure throughput for a topic with 1 partition and with 6 partitions. Explain why the consumer measurement does not reflect your real application's per-partition rate.- Interview question: "How would you decide the number of partitions for a new topic that carries customer events keyed by customer ID?" Hint: combine throughput math, consumer scaling limits, divisibility, per-partition costs, and the cost of changing the count for a keyed topic later.
Topic-Level Configuration
Theory
Different data needs different handling. Debug logs can be deleted after a day, payment events must keep strong durability, and a "latest customer profile" topic should keep only the newest value per key. Kafka lets each topic override the broker's defaults, so one cluster can serve all of these needs.
Analogy: broker defaults are a company's standard office policy. Topic-level configs are the exceptions written into a specific department's handbook: "Legal keeps documents for ten years," "Marketing shreds drafts after a week." Departments without exceptions follow the standard policy, including whenever it changes.
How inheritance works
effective value for topic T = T's override, if set
else the broker setting (dynamic per-broker, dynamic default,
server.properties, built-in default, as in kafka-configs)
A topic without an override follows the broker value, including future changes to it. If you rely on a value for correctness (for example min.insync.replicas=2), set it explicitly on the topic.
Topic configs and their broker counterparts usually have different names:
| Topic config | Broker default | Default value | What it controls |
|---|---|---|---|
retention.ms | log.retention.ms (or .hours) | 604800000 (7 days) | How long to keep data (-1 = forever) |
retention.bytes | log.retention.bytes | -1 (no limit) | Maximum size per partition before old segments are deleted |
cleanup.policy | log.cleanup.policy | delete | delete, compact, or both (Chapter 7) |
min.insync.replicas | min.insync.replicas | 1 | Replicas that must have a write for acks=all to succeed |
max.message.bytes | message.max.bytes (note the different word order) | 1048588 (about 1 MB) | Largest record batch the topic accepts |
segment.bytes / segment.ms | log.segment.bytes / log.roll.ms | 1 GB / 7 days | When the active segment is rolled. Retention and compaction only act on closed segments. |
compression.type | compression.type | producer | Keep the producer's codec, or recompress with a specific one |
message.timestamp.type | log.message.timestamp.type | CreateTime | Event time vs broker append time (Section 2.1) |
unclean.leader.election.enable | same | false | Availability over durability (Section 2.4) |
delete.retention.ms | log.cleaner.delete.retention.ms | 86400000 (1 day) | How long tombstones survive on compacted topics |
remote.storage.enable / local.retention.ms | (tiered storage) | false / -2 | Tiered storage (Chapter 7) |
Example: three topics, three profiles
BS=localhost:9092
# Critical business events: strong durability, keep 30 days
bin/kafka-topics.sh --bootstrap-server $BS --create --topic payments.transactions \
--partitions 12 --replication-factor 3 \
--config min.insync.replicas=2 \
--config retention.ms=2592000000 \
--config unclean.leader.election.enable=false
# Latest state per key: compacted, never expires by time
bin/kafka-topics.sh --bootstrap-server $BS --create --topic customers.profiles \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.compaction.lag.ms=3600000 # keep every update for at least 1 hour
# High-volume diagnostic data: short retention, size cap, compressed on the broker
bin/kafka-topics.sh --bootstrap-server $BS --create --topic ops.debug-logs \
--partitions 6 --replication-factor 2 \
--config retention.ms=86400000 \
--config retention.bytes=5368709120 \
--config compression.type=zstd
# Which topics have overrides at all?
bin/kafka-topics.sh --bootstrap-server $BS --describe --topics-with-overrides
Changes made later with kafka-configs.sh --alter --topic ... take effect without restarts. Retention changes are applied the next time the broker checks for deletable segments (log.retention.check.interval.ms, default 5 minutes), and only to closed segments.
A common surprise: retention and segments
retention.ms = 1 hour, segment.ms = 7 days (default), low traffic
[ active segment: records from the last 6 days ............ ] <- cannot be deleted yet
-> data remains for up to ~7 days, even though retention is 1 hour
Retention deletes whole closed segments. On low-traffic topics with short retention, also lower segment.ms (for example to the retention period) so segments roll and become eligible for deletion.
Key Takeaways
- Topic-level settings override broker defaults. Topics without an override follow the broker value, including later changes to it.
- Topic and broker names often differ (
retention.msvslog.retention.ms,max.message.bytesvsmessage.max.bytes). - The most important per-topic settings are retention, cleanup policy,
min.insync.replicas, maximum message size, and segment rolling. - Retention removes only closed segments. Pair short retention with a smaller
segment.mson quiet topics.
Practice
- (Beginner) For each topic, choose
cleanup.policy,retention.ms, andmin.insync.replicas: (a) clickstream for analytics, (b) bank transfers, (c) current shipping address per customer.- (Intermediate) Create a topic with
retention.ms=60000, produce records once, and check after five minutes whether they are still readable. Then setsegment.ms=60000, produce again, and repeat. Explain the difference.- Interview question: "A producer gets
RecordTooLargeExceptionafter you raisedmax.message.byteson the topic to 5 MB. What else needs to change?" Hint: follow the record through the producer, the broker, followers replicating it, and the consumers fetching it, and find the size limit at each step.
Adding Partitions
Theory
Kafka lets you increase a topic's partition count at any time, online. It never lets you decrease it, because removing a partition would mean deleting or moving data and rewriting offsets. Adding partitions is easy for unkeyed topics. For keyed topics it changes where keys go, because the default partitioner computes murmur2(key) % numPartitions, and changing numPartitions changes the result for most keys.
Analogy: imagine mail sorted into 3 bins by "house number mod 3". Add a fourth bin and switch to "mod 4", and most houses' mail now goes to a different bin. Anyone who assumed "all mail for house 7 is in one bin" now has to look in two bins: old letters in the old one and new letters in the new one.
Example: how many keys move
This Python port of Kafka's default key-to-partition mapping shows what happens when a topic grows from 3 to 4 partitions:
def murmur2(data: bytes) -> int:
"""Port of Kafka's Utils.murmur2 (32-bit arithmetic), used by the default partitioner."""
length, m, mask = len(data), 0x5BD1E995, 0xFFFFFFFF
h = (0x9747B28C ^ length) & mask # fixed seed XOR length
for i in range(length // 4): # mix 4 bytes at a time
k = int.from_bytes(data[i * 4:i * 4 + 4], "little")
k = (k * m) & mask
k ^= k >> 24
k = (k * m) & mask
h = ((h * m) & mask) ^ k
tail, rem = length & ~3, length % 4 # mix the remaining 1-3 bytes
if rem == 3:
h ^= data[tail + 2] << 16
if rem >= 2:
h ^= data[tail + 1] << 8
if rem >= 1:
h ^= data[tail]
h = (h * m) & mask
h ^= h >> 13
h = (h * m) & mask
return h ^ (h >> 15)
def partition_for(key: str, num_partitions: int) -> int:
return (murmur2(key.encode()) & 0x7FFFFFFF) % num_partitions # clear sign bit, then modulo
for key in [f"CUST-{n}" for n in range(1, 11)]:
before, after = partition_for(key, 3), partition_for(key, 4)
print(f"{key:<8} {before} -> {after}{' moved' if before != after else ''}")
CUST-1 1 -> 3 moved
CUST-2 2 -> 1 moved
CUST-3 2 -> 2
CUST-4 0 -> 1 moved
CUST-5 1 -> 1
CUST-6 0 -> 1 moved
CUST-7 0 -> 3 moved
CUST-8 1 -> 1
CUST-9 1 -> 3 moved
CUST-10 2 -> 3 moved
Seven of these ten keys move. Across 100,000 keys, about 75% move when going from 3 to 4 partitions. The partitioner has no memory of where a key used to go.
What breaks for keyed topics
before (3 partitions) after (4 partitions)
CUST-7 -> p0: [e1][e2][e3] CUST-7 -> p3: [e4][e5]
p0 still holds [e1][e2][e3]
Two consumers may process p0 and p3 in parallel:
e4 ("address changed") can be handled before e3 ("account created") -> ordering violated
| Affected | Problem |
|---|---|
| Per-key ordering | During the transition, a key's old events (old partition) and new events (new partition) can be processed concurrently, out of order |
| Compacted topics | The "latest value" for a key may now exist in two partitions, and compaction never merges across partitions |
| Kafka Streams and stateful consumers | State stores and co-partitioned joins assume a fixed key-to-partition mapping. Changing the count of an input topic breaks them (Chapter 10). |
Consumers with auto.offset.reset=latest | A group discovers new partitions on its next metadata refresh (metadata.max.age.ms, default 5 minutes). Records written to a new partition before then have no committed offset, and latest skips them. |
Safer ways to grow a keyed topic
- Over-provision from the start (previous topic), so you never need to add partitions.
- Pause, drain, then add: stop producers, let consumers reach zero lag, add partitions, then restart producers. No key has events in two partitions that are processed at the same time. This requires downtime.
- Migrate to a new topic: create
orders.v2with more partitions, dual-write or replay from the old topic, and switch consumers once caught up. This is also the only way to reduce partitions. - Use
auto.offset.reset=earliestfor groups that read topics which may grow, so records in new partitions are not skipped.
Example: adding partitions
BS=localhost:9092
bin/kafka-topics.sh --bootstrap-server $BS --alter --topic clickstream --partitions 12
bin/kafka-topics.sh --bootstrap-server $BS --describe --topic clickstream | head -1
# Topic: clickstream TopicId: ... PartitionCount: 12 ReplicationFactor: 3 Configs:
# Trying to shrink fails
bin/kafka-topics.sh --bootstrap-server $BS --alter --topic clickstream --partitions 6
# Error while executing topic command : ... InvalidPartitionsException ... (the count can only be increased)
from confluent_kafka.admin import AdminClient, NewPartitions
admin = AdminClient({"bootstrap.servers": "localhost:9092"})
# NewPartitions takes the new TOTAL count, not the number to add
for topic, future in admin.create_partitions([NewPartitions("clickstream", 12)]).items():
future.result()
print(f"{topic} now has 12 partitions")
New partitions start empty. Existing data is not redistributed, so the new partitions will be lighter than the old ones until retention removes old data. Moving existing replicas between brokers is a separate operation (kafka-reassign-partitions.sh, Section 2.4).
Key Takeaways
- Partitions can be added online but never removed. Reducing requires migrating to a new topic.
- For unkeyed topics, adding partitions is safe. For keyed topics, most keys move to a different partition, which breaks per-key ordering during the transition and confuses compaction and stateful processing.
- Groups notice new partitions only after a metadata refresh. With
auto.offset.reset=latest, early records in new partitions can be skipped. - Prefer over-provisioning keyed topics up front. Otherwise pause and drain, or migrate to a new topic.
Practice
- (Beginner) Create a 3-partition topic, produce 10 keyed records, add a fourth partition, and produce the same 10 keys again. Use the console consumer with
print.keyandprint.partitionto list which keys now appear in two partitions.- (Intermediate) Extend the Python script to compute the fraction of moved keys for growth from 6 to 12 partitions and from 6 to 7 partitions. Explain why doubling moves fewer keys than adding one partition.
- Interview question: "Your keyed
orderstopic has 6 partitions, and consumers can no longer keep up. Walk through your options, and the risks of each, for increasing throughput." Hint: consider faster processing per partition, adding partitions with or without draining, migrating to a new topic, and consumption models that do not require more partitions.
Internal Topics
Theory
Kafka stores much of its own state in topics. Consumer group offsets, transaction state, and share group state are all kept in internal topics, using the same replicated, durable log that holds your data. This means Kafka's own bookkeeping gets replication and failover for free, but it also means these topics need the same care as your most important data: a sufficient replication factor and healthy replicas.
Analogy: a bank that records its own internal ledgers in the same secure vault as its customers' deposits. The vault's protections cover both, but if the vault is badly configured, the bank also loses track of its own books.
Kafka's internal topics
| Topic | Holds | Defaults | Notes |
|---|---|---|---|
__consumer_offsets | Committed offsets and group metadata of consumer groups | 50 partitions (offsets.topic.num.partitions), RF 3 (offsets.topic.replication.factor), compacted | Created on first use by a consumer group (Section 5.3) |
__transaction_state | State of producer transactions | 50 partitions, RF 3, min.insync.replicas=2 (transaction.state.log.*) | Created when the first transactional producer starts (Chapter 8) |
__share_group_state | Delivery state of share groups (queues for Kafka) | 50 partitions, RF 3 (share.coordinator.state.topic.*) | Kafka 4.x (Section 5.4) |
__remote_log_metadata | Metadata about segments in tiered storage | 50 partitions, RF 3 | Only with tiered storage enabled (Chapter 7) |
__cluster_metadata | KRaft metadata log | Single partition, replicated by Raft | Not a normal topic: not listed by kafka-topics.sh and not readable by clients (Section 2.3) |
Other components create topics that behave like internal topics, although Kafka does not mark them as such:
| Component | Topics | Chapter |
|---|---|---|
| Kafka Connect (distributed) | connect-configs (1 partition, compacted), connect-offsets, connect-status (names configurable) | 9 |
| Kafka Streams | <application.id>-<name>-changelog, <application.id>-<name>-repartition | 10 |
| Schema Registry | _schemas | 6 |
| MirrorMaker 2 | mm2-offset-syncs.*.internal, <source>.checkpoints.internal, heartbeats | 14 |
How a group finds its coordinator
Each consumer group's offsets live in exactly one partition of __consumer_offsets, chosen by hashing the group ID. The leader of that partition is the group's coordinator (Section 5.2):
partition = abs(groupId.hashCode()) % offsets.topic.num.partitions
"order-service".hashCode() = 562889526 -> 562889526 % 50 = 26 -> __consumer_offsets-26
"billing".hashCode() = -109829509 -> 109829509 % 50 = 9 -> __consumer_offsets-9
The leader of __consumer_offsets-26 coordinates order-service. If that partition has no leader, the group cannot commit offsets or rebalance. This is why the replication factor of internal topics matters, and why offsets.topic.num.partitions must never be changed on an existing cluster: changing it would map groups to different partitions and lose their offsets.
Example: inspecting internal topics
BS=localhost:9092
# Internal topics appear in --list unless excluded
bin/kafka-topics.sh --bootstrap-server $BS --list
# __consumer_offsets
# orders
bin/kafka-topics.sh --bootstrap-server $BS --list --exclude-internal
# orders
# Check their health like any other topic
bin/kafka-topics.sh --bootstrap-server $BS --describe --topic __consumer_offsets | head -3
# Decode raw offset commits (advanced debugging; output abridged)
bin/kafka-console-consumer.sh --bootstrap-server $BS --topic __consumer_offsets \
--from-beginning --max-messages 3 \
--formatter org.apache.kafka.tools.consumer.OffsetsMessageFormatter \
--consumer-property exclude.internal.topics=false
# {"key":{"type":1,"data":{"group":"order-service","topic":"orders","partition":0}},
# "value":{"version":3,"data":{"offset":18234,"leaderEpoch":0,"metadata":"","commitTimestamp":1761210942120}}}
For everyday questions, kafka-consumer-groups.sh (Section 3.2) is the right tool. Reading __consumer_offsets directly is for deep debugging only.
Configuration pitfalls
| Situation | Symptom | Fix |
|---|---|---|
| Single-broker cluster with default RF 3 for internal topics | Consumer groups never become ready (COORDINATOR_NOT_AVAILABLE), transactions fail | Set offsets.topic.replication.factor=1 and the transaction.state.log.* and share.coordinator.state.topic.* equivalents (development only) |
| Cluster started with one broker, later grown to three | __consumer_offsets still has RF 1. One broker failure loses some groups' offsets. | Increase its RF with kafka-reassign-partitions.sh (Section 2.4) |
Changing offsets.topic.num.partitions after creation | No effect on the existing topic, and changing the partition count by other means remaps groups | Choose it before the cluster is first used, then leave it alone |
| Deleting or producing to internal topics by hand | Lost offsets or corrupted coordinator state | Never do this. Use the dedicated tools. |
Key Takeaways
- Kafka keeps consumer offsets, transaction state, and share group state in internal topics (
__consumer_offsets,__transaction_state,__share_group_state). The KRaft metadata log__cluster_metadatais special and not a normal topic. - A group's offsets live in one partition of
__consumer_offsets, chosen by hashing the group ID. That partition's leader is the group coordinator. - Internal topics need a healthy replication factor (3 in production). On a single development broker, lower it to 1 before first use.
- Never change
offsets.topic.num.partitionson a live cluster, and never write to internal topics directly. - Connect, Streams, Schema Registry, and MirrorMaker 2 create their own critical topics, which need the same care.
Practice
- (Beginner) List all topics on your cluster with and without
--exclude-internal. Start a console consumer with--group lab-group, then describe__consumer_offsetsand note its partition count and replication factor.- (Intermediate) Compute which
__consumer_offsetspartition stores the offsets of your group, using Java'sString.hashCode()(or a reimplementation). Usekafka-consumer-groups.sh --describe --group <g> --stateto find the coordinator and confirm it is the leader of that partition.- Interview question: "Why does Kafka store consumer offsets in a compacted, replicated topic rather than in an external database or in ZooKeeper, as early versions did?" Hint: compare write throughput, durability and failover, how the latest offset per group and partition is retained, and the load that frequent offset commits put on ZooKeeper.
<a id="4-producers"></a>