11. Security
Out of the box, a Kafka cluster accepts unencrypted connections from anyone who can reach its ports and lets every client read, write, and delete every topic. This chapter covers the three layers that turn it into a system you can trust with payments, personal data, and multi-team workloads: encryption (nobody can read or tamper with data in transit or on disk), authentication (the cluster knows who is connecting), and authorization and governance (each identity can do only what it should, within resource limits, and leaves an audit trail). Security mistakes in Kafka are rarely subtle bugs; they are usually a listener left on PLAINTEXT or a missing ACL, so knowing how the pieces fit is as important as knowing the configuration keys.
<a id="111-encryption"></a>
11.1 Encryption
This section covers how Kafka protects data on the wire with TLS, for both client and cluster-internal traffic, how to issue and rotate the certificates that TLS depends on, and what options exist for protecting data stored on disk, which Kafka itself does not encrypt.
TLS for Client-Broker Traffic
Theory
By default, Kafka clients and brokers exchange records as plain bytes over TCP. Anyone with access to the network path (a compromised host in the same VPC, a misconfigured network tap, a cloud provider's shared infrastructure) can read every order, password reset token, or medical record flowing through, and could even modify requests in flight. The data is only as private as the least-trusted network segment between producer, broker, and consumer.
TLS (Transport Layer Security, the successor of SSL) solves three problems at once:
| Property | What it means | Mechanism |
|---|---|---|
| Confidentiality | Eavesdroppers see only ciphertext | Symmetric encryption (for example AES-GCM) with session keys negotiated per connection |
| Integrity | Tampered bytes are detected and the connection is dropped | Authenticated encryption (AEAD) |
| Server authentication | The client is sure it is talking to the real broker, not an impostor | The broker presents a certificate signed by a CA the client trusts, and the client checks the hostname |
Analogy: TLS is a sealed, tamper-evident armored van with the driver's ID badge checked at the door. Anyone can see that the van is moving, but not what is inside, and nobody can swap the cargo without breaking the seal.
Kafka still uses the historic name SSL in its configuration keys and security protocol names, but it negotiates modern TLS (TLS 1.3 by default on Java 11+, with TLS 1.2 as fallback).
The TLS handshake on a Kafka connection
Client (producer) Broker (EXTERNAL listener, port 9093)
| |
| ---- TCP connect -------------------------------------> |
| ---- ClientHello (TLS versions, cipher suites) -------> |
| <--- ServerHello + broker certificate chain ----------- |
| |
| verify chain against truststore (is it signed by our CA?)
| verify hostname: does SAN contain broker1.kafka.example.com?
| |
| <==== key exchange, both sides derive session keys ====> |
| ==== encrypted Kafka protocol: ApiVersions, Metadata, Produce ... ===> |
Two files drive this on the JVM:
| Store | Who needs it | Contents |
|---|---|---|
| Keystore | The side that proves its identity (always the broker; the client too with mTLS) | Private key + certificate chain |
| Truststore | The side that verifies the peer (always the client; the broker too with mTLS) | CA certificate(s) that are trusted to sign peer certificates |
Kafka accepts JKS, PKCS12 (the recommended default), and PEM stores, the last either as files or inline in the configuration (ssl.keystore.certificate.chain, ssl.keystore.key, ssl.truststore.certificates).
Hostname verification
Since Kafka 2.0, ssl.endpoint.identification.algorithm=https is the client default: the client checks that the hostname it connected to appears in the broker certificate's Subject Alternative Name (SAN). Without this check, any certificate signed by your CA (including one stolen from a test machine) could impersonate any broker. Setting this property to an empty string to "fix" a handshake error is a common and dangerous shortcut; fix the certificate instead.
Performance cost
Without TLS, brokers send log segment bytes straight from the page cache to the socket with the sendfile system call (zero-copy, Chapter 2). With TLS, data must be encrypted in user space, so the broker reads it into the JVM heap, encrypts it, and writes it out. Expect noticeably higher CPU usage on brokers, especially for consumer fetches. With AES-NI hardware acceleration and modern JDKs, the overhead is usually acceptable, but measure it (Chapter 13) rather than assume.
Example: broker configuration for a TLS listener
# Two listeners: INTERNAL for replication, EXTERNAL for clients (both encrypted)
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
advertised.listeners=INTERNAL://broker1.internal:9092,EXTERNAL://broker1.kafka.example.com:9093
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL,CONTROLLER:SSL
inter.broker.listener.name=INTERNAL
# Broker identity: private key + certificate whose SAN lists BOTH advertised hostnames
ssl.keystore.type=PKCS12
ssl.keystore.location=/etc/kafka/secrets/broker1.keystore.p12
ssl.keystore.password=${file:/etc/kafka/secrets/creds.properties:keystore.password} # config provider, no plaintext secret
ssl.key.password=${file:/etc/kafka/secrets/creds.properties:key.password}
# CA bundle used to verify peers (other brokers, controllers, mTLS clients)
ssl.truststore.type=PKCS12
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.p12
ssl.truststore.password=${file:/etc/kafka/secrets/creds.properties:truststore.password}
ssl.enabled.protocols=TLSv1.3,TLSv1.2 # refuse TLS 1.0/1.1
ssl.client.auth=none # server-side TLS only (mTLS is covered in 11.2)
# Required for the ${file:...} placeholders above
config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
Settings can be scoped to one listener with the listener.name.<listener>. prefix, for example listener.name.external.ssl.keystore.location, so internal and external listeners can present different certificates.
Example: client configuration
# client-ssl.properties (Java clients and CLI tools via --command-config)
bootstrap.servers=broker1.kafka.example.com:9093,broker2.kafka.example.com:9093
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=/etc/app/kafka-ca.crt # only the CA certificate, no private key needed
# ssl.endpoint.identification.algorithm=https # default; never set it to ""
# confluent-kafka (librdkafka) uses its own property names
from confluent_kafka import Producer
producer = Producer({
"bootstrap.servers": "broker1.kafka.example.com:9093",
"security.protocol": "SSL",
"ssl.ca.location": "/etc/app/kafka-ca.crt", # PEM file with the CA certificate
"ssl.endpoint.identification.algorithm": "https", # hostname verification (default in librdkafka 2.x)
})
producer.produce("orders", key=b"o-1001", value=b'{"amount": 42.0}')
producer.flush()
Verifying from the command line
# Inspect the certificate a listener presents: subject, validity, SANs
openssl s_client -connect broker1.kafka.example.com:9093 \
-servername broker1.kafka.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -dates -ext subjectAltName
# End-to-end check with Kafka's own tooling over TLS
kafka-broker-api-versions.sh --bootstrap-server broker1.kafka.example.com:9093 \
--command-config client-ssl.properties | head -3
Key Takeaways
- TLS gives confidentiality, integrity, and broker authentication. Kafka names it
SSLin configs but negotiates TLS 1.2/1.3. - The broker needs a keystore (its key and certificate); clients need a truststore (the CA). PKCS12 and PEM are the recommended formats.
- Hostname verification against the certificate SAN is on by default; every advertised hostname must be in the SAN.
- TLS disables zero-copy transfer, so broker CPU usage rises. Benchmark before and after enabling it.
- Keep store passwords out of plain configuration files with config providers.
Practice
- (Beginner) Enable an
SSLlistener on a local single-node cluster using a self-signed CA, then produce and consume withkafka-console-producer.shandkafka-console-consumer.shusing--command-config.- (Intermediate) Issue a broker certificate whose SAN does not include the advertised hostname. Record the exact client error, then fix the certificate without touching
ssl.endpoint.identification.algorithm.- Interview question: "Why does enabling TLS increase CPU usage on Kafka brokers more than on a typical HTTPS web server?" Hint: think about how the broker serves fetch requests without TLS, and which kernel optimization becomes impossible once bytes must be encrypted.
Inter-Broker Encryption
Theory
Encrypting only client connections leaves a gap: the same records travel again between brokers during replication, and cluster metadata (including SCRAM credential hashes and ACLs) travels between controllers and brokers. An attacker who cannot sniff client traffic may well be able to sniff the cluster's internal network, especially in multi-AZ or stretched deployments where "internal" traffic crosses data-center links.
A KRaft cluster has four kinds of connections, each of which needs its own security decision:
+-------------------- controller quorum ------------------+
| ctrl1 <------ Raft (Vote, Fetch) ------> ctrl2, ctrl3 | (4) controller <-> controller
+---------------------------^-----------------------------+
| CONTROLLER listener
(3) broker -> controller: | registration, heartbeats,
metadata fetch | metadata log replication
+---------------------------+------------------------------+
| |
+---------------+ (2) replication (follower Fetch) +---------------+
(1) clients | broker 4 | <-----------------------------------> | broker 5 |
-----------> | EXTERNAL:9093 | INTERNAL listener :9092 | EXTERNAL:9093 |
+---------------+ +---------------+
| Connection | Listener used | Controlled by |
|---|---|---|
| (1) Client to broker | Any client-facing listener | listeners, listener.security.protocol.map |
| (2) Broker to broker (replication) | Inter-broker listener | inter.broker.listener.name (or security.inter.broker.protocol, not both) |
| (3) Broker to controller | Controller listener | controller.listener.names + its protocol in the map |
| (4) Controller to controller | Controller listener | Same as above |
For SASL on internal listeners, sasl.mechanism.inter.broker.protocol selects the mechanism for (2) and sasl.mechanism.controller.protocol the one for (3) and (4). Using mTLS for (2) to (4) is common, because brokers and controllers already have certificates and no extra credentials need to be distributed. SCRAM is not supported on the controller listener, because SCRAM credentials are stored in the very metadata log the controllers serve.
Example: a broker node and a controller node with fully encrypted internal traffic
# ---------- controller node: ctrl1 ----------
process.roles=controller
node.id=1
controller.quorum.bootstrap.servers=ctrl1.internal:9094,ctrl2.internal:9094,ctrl3.internal:9094
listeners=CONTROLLER://0.0.0.0:9094
advertised.listeners=CONTROLLER://ctrl1.internal:9094
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SSL
ssl.client.auth=required # only nodes holding a cluster certificate may connect
ssl.keystore.location=/etc/kafka/secrets/ctrl1.keystore.p12
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.p12
# (passwords omitted for brevity)
# ---------- broker node: broker4 ----------
process.roles=broker
node.id=4
controller.quorum.bootstrap.servers=ctrl1.internal:9094,ctrl2.internal:9094,ctrl3.internal:9094
controller.listener.names=CONTROLLER # broker uses this protocol to reach controllers, does not listen on it
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
advertised.listeners=INTERNAL://broker4.internal:9092,EXTERNAL://broker4.kafka.example.com:9093
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SASL_SSL,CONTROLLER:SSL
inter.broker.listener.name=INTERNAL
listener.name.internal.ssl.client.auth=required # replication peers must present certificates
ssl.keystore.location=/etc/kafka/secrets/broker4.keystore.p12 # used as server AND client certificate
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.p12
The broker's keystore plays two roles: it is the server certificate on INTERNAL and EXTERNAL, and the client certificate when the broker connects to other brokers and to controllers. The certificate therefore needs both the serverAuth and clientAuth extended key usages if your CA sets them.
Migrating a running cluster from PLAINTEXT to SSL without downtime
You cannot flip all brokers at once, because a restarted broker using TLS for replication could not talk to brokers that are still plaintext. The standard sequence uses several rolling restarts (Chapter 12):
Step 1 Add an SSL listener next to the PLAINTEXT one on every broker -> rolling restart
listeners=PLAINTEXT://:9092,SSL://:9095 inter.broker.listener.name=PLAINTEXT
Step 2 Switch replication to the new listener -> rolling restart
inter.broker.listener.name=SSL
Step 3 Move all clients to the SSL port (deploy client configs)
Step 4 Remove the PLAINTEXT listener -> rolling restart
The controller listener is changed the same way: controllers first listen on both protocols, then brokers and controllers switch, then the old listener is removed. Practice this on a test cluster first, because a wrong controller listener prevents brokers from registering.
Key Takeaways
- Securing only client traffic is incomplete: replication and metadata traffic carry the same data plus credentials and ACLs.
- KRaft has four connection types; inter-broker and controller listeners each need an explicit protocol in
listener.security.protocol.map. - mTLS is a natural fit for internal listeners. SCRAM cannot be used on the controller listener.
- Migrating from plaintext to TLS is a multi-step rolling procedure: add the listener, switch internal traffic, move clients, remove the old listener.
Practice
- (Beginner) Draw the listener map for a cluster with three controllers, six brokers, internal clients in the same VPC, and external partners on the internet. Assign a security protocol to each listener and justify each choice.
- (Intermediate) On a three-broker test cluster with
PLAINTEXTeverywhere, perform the zero-downtime migration toSSLfor inter-broker traffic while a producer writes continuously. Verify withtcpdumporopenssl s_clientthat port 9092 no longer carries readable records.- Interview question: "Your security team says internal traffic is on a private network, so inter-broker TLS is unnecessary. How do you respond?" Hint: consider what flows over replication and controller connections, multi-AZ links, insider threats, and compliance frameworks that require encryption in transit everywhere.
Certificate Management
Theory
TLS is only as trustworthy as the certificates behind it. A certificate binds a public key to an identity (a hostname, or a client name for mTLS) and is signed by a Certificate Authority (CA). Clients trust the CA, so they trust anything it signs. That makes certificate management an operational discipline with three recurring tasks: issuing certificates with the right names, distributing keystores and truststores securely, and rotating them before they expire or after a compromise.
Most Kafka TLS outages are not attacks; they are expired certificates. A certificate that silently expires on a Saturday stops every client and, if internal listeners use TLS, replication itself.
Root CA (offline, 10+ years)
|
Intermediate CA "kafka-prod" (online, 2-5 years)
/ | \
broker1.kafka.example.com ctrl1.internal CN=billing-service
(server+client auth, (server+client (client auth only,
SAN: broker1.kafka..., auth, 90 days) mTLS, 30 days)
broker1.internal, 90 days)
Keep the root offline, sign with an intermediate, and put the root (and intermediate, if clients do not receive the full chain) in truststores. Leaf certificates should be short-lived so a leaked key has limited value.
What a broker certificate must contain
| Field | Requirement |
|---|---|
| SAN (DNS entries) | Every hostname in advertised.listeners that clients or peers connect to. Wildcards (*.kafka.example.com) work but widen the impact of a stolen key |
| Key usage | digitalSignature, keyEncipherment; extended key usage serverAuth and, for internal listeners, clientAuth |
| Validity | Short enough to limit exposure, long enough to rotate safely (90 days with automation is common) |
| Subject DN | Matters for mTLS principals (Section 11.3, Principal Mapping) |
Example: issuing a broker certificate with OpenSSL and keytool
# 1. Create a lab CA (in production use your PKI, Vault, or cert-manager instead)
openssl req -new -x509 -days 3650 -nodes -newkey rsa:4096 \
-keyout ca.key -out ca.crt -subj "/CN=Kafka Lab CA"
# 2. Broker key pair inside a PKCS12 keystore
keytool -genkeypair -alias broker1 -keyalg RSA -keysize 3072 -validity 90 \
-dname "CN=broker1.kafka.example.com" \
-keystore broker1.keystore.p12 -storetype PKCS12 -storepass changeit
# 3. Certificate signing request (CSR)
keytool -certreq -alias broker1 -keystore broker1.keystore.p12 \
-storepass changeit -file broker1.csr
# 4. Sign it, adding SANs and usages through an extension file
cat > broker1.ext <<'EOF'
subjectAltName = DNS:broker1.kafka.example.com, DNS:broker1.internal
extendedKeyUsage = serverAuth, clientAuth
EOF
openssl x509 -req -in broker1.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 90 -sha256 -extfile broker1.ext -out broker1.crt
# 5. Import the CA first, then the signed certificate (replaces the self-signed one)
keytool -importcert -alias ca -file ca.crt -keystore broker1.keystore.p12 \
-storepass changeit -noprompt
keytool -importcert -alias broker1 -file broker1.crt -keystore broker1.keystore.p12 \
-storepass changeit
# 6. Truststore shared by all nodes and Java clients
keytool -importcert -alias ca -file ca.crt -keystore kafka.truststore.p12 \
-storetype PKCS12 -storepass changeit -noprompt
# 7. Check what you built
keytool -list -v -keystore broker1.keystore.p12 -storepass changeit | grep -E "Alias|Owner|Valid|DNSName"
Rotation without restarts
Brokers can reload keystores and truststores dynamically (KIP-226). Write the renewed keystore to disk, then issue a dynamic config update for that listener; using the same path is allowed and triggers a reload:
kafka-configs.sh --bootstrap-server broker1.kafka.example.com:9093 \
--command-config admin.properties \
--entity-type brokers --entity-name 1 --alter \
--add-config listener.name.external.ssl.keystore.location=/etc/kafka/secrets/broker1.keystore.p12
Existing connections keep their session; new connections get the new certificate. By default the broker rejects a new keystore whose subject DN or SANs differ from the old one, which protects against loading the wrong file.
Rotating the CA itself is the delicate case, because every party must trust the new CA before anyone presents a certificate signed by it:
Phase 1 Truststores everywhere contain OLD CA + NEW CA (brokers dynamically, clients via redeploy)
Phase 2 Re-issue broker/controller/client certificates from the NEW CA
Phase 3 Remove the OLD CA from all truststores
Automation options
| Environment | Typical tooling |
|---|---|
| Kubernetes | cert-manager, or Strimzi's built-in cluster and clients CAs with automatic renewal (Chapter 14) |
| VMs / bare metal | HashiCorp Vault PKI, internal ACME, configuration management (Ansible) pushing keystores |
| Managed Kafka | Provider manages broker certificates; you manage client certificates for mTLS |
Whatever the tooling, monitor expiry: export days-until-expiry as a metric (for example with a blackbox exporter probing each listener) and alert at 30 and 7 days.
Key Takeaways
- Expired certificates are the most common TLS outage. Automate issuance and renewal, and alert on expiry.
- Broker certificates need every advertised hostname in the SAN and, for internal listeners, both
serverAuthandclientAuthusages. - Use a CA hierarchy with an offline root and short-lived leaf certificates.
- Brokers reload keystores and truststores dynamically per listener, so routine rotation needs no restart.
- Rotate a CA in phases: trust both, re-issue, then remove the old one.
Practice
- (Beginner) Using the commands above, create a CA, a broker keystore, and a truststore. Use
keytool -list -vto find the SAN entries and expiry date.- (Intermediate) Issue a broker certificate valid for 10 minutes, start the broker, renew the certificate, and rotate it with
kafka-configs.shwhile a consumer is running. Confirm withopenssl s_clientthat new connections see the new expiry date.- Interview question: "Walk me through rotating the CA for a Kafka cluster with hundreds of client applications without an outage." Hint: order matters; think about which side verifies which certificate and what happens if a client sees a certificate signed by a CA it does not yet trust.
Encryption at Rest
Theory
TLS protects data while it moves. Once a broker receives a batch, it writes the bytes to its log segments exactly as received, unencrypted. Anyone who obtains a broker disk, a volume snapshot, a backup, or a tiered-storage bucket can read every record with kafka-dump-log.sh. Apache Kafka has no built-in at-rest encryption, so you choose one or more layers outside it:
Layer Encrypts Protects against Kafka changes
-------------------------- --------------------------------- --------------------------------------- -------------
Client-side (end-to-end) Record values (and maybe keys) Disk theft, backups, broker admins, Serializers,
cloud provider, other tenants key management
Filesystem / block device Everything on the disk Stolen or decommissioned disks, None
(LUKS/dm-crypt, EBS+KMS) snapshots taken outside your account
Object storage (S3 SSE) Tiered-storage segments Bucket exposure, provider media None
Analogy: disk encryption is a locked warehouse. Once inside, anyone with a badge sees every box. End-to-end encryption locks each box individually, and only the intended recipient has the key, so even warehouse staff cannot open it.
Volume and filesystem encryption is transparent to Kafka, costs little thanks to hardware acceleration, and preserves zero-copy reads (the kernel decrypts into the page cache). It is the baseline for compliance requirements such as "data at rest must be encrypted." It does not protect against anyone who can log in to the broker or read the page cache, nor against Kafka administrators.
End-to-end (client-side) encryption encrypts the record value in the producer and decrypts it in the consumer. Brokers store and replicate ciphertext only. This protects against a wider set of threats but has real costs:
| Concern | Impact |
|---|---|
| Key management | You need a KMS (AWS KMS, Vault, Azure Key Vault) and envelope encryption: a data encryption key (DEK) encrypts records, and the KMS key (KEK) encrypts the DEK |
| Compression | Ciphertext does not compress. Compress before encrypting, or accept larger batches |
| Keys and compaction | Encrypting the record key breaks partitioning and compaction unless the encryption is deterministic. Usually keys stay in clear text (so do not put sensitive data in them) |
| Tooling | Kafka Connect sinks, ksqlDB, and stream processors need decryption keys to work with the data |
| Schema Registry | Schemas still describe the plaintext; encryption wraps the serialized payload |
A useful side effect is crypto-shredding: if each customer's data uses its own DEK, deleting that key makes their records unreadable everywhere (log segments, replicas, backups), which helps with GDPR erasure requests in an append-only log.
Example: an encrypting serializer with AES-GCM
import org.apache.kafka.common.errors.SerializationException;
import org.apache.kafka.common.header.Headers;
import org.apache.kafka.common.serialization.Serializer;
import javax.crypto.Cipher;
import javax.crypto.SecretKey;
import javax.crypto.spec.GCMParameterSpec;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.util.Map;
public class AesGcmSerializer implements Serializer<String> {
private static final int IV_BYTES = 12, TAG_BITS = 128;
private final SecureRandom random = new SecureRandom();
private SecretKey dataKey;
private String keyId;
@Override
public void configure(Map<String, ?> configs, boolean isKey) {
keyId = (String) configs.get("encryption.key.id"); // custom property passed via producer config
dataKey = KeyService.unwrapDataKey(keyId); // envelope encryption: KMS decrypts the DEK
}
@Override
public byte[] serialize(String topic, Headers headers, String data) {
if (data == null) return null; // keep tombstones as tombstones
try {
byte[] iv = new byte[IV_BYTES];
random.nextBytes(iv); // never reuse an IV with the same key
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
cipher.init(Cipher.ENCRYPT_MODE, dataKey, new GCMParameterSpec(TAG_BITS, iv));
cipher.updateAAD(topic.getBytes(StandardCharsets.UTF_8)); // ciphertext only valid for this topic
byte[] ciphertext = cipher.doFinal(data.getBytes(StandardCharsets.UTF_8));
headers.add("enc.key.id", keyId.getBytes(StandardCharsets.UTF_8)); // consumer looks up the DEK
return ByteBuffer.allocate(IV_BYTES + ciphertext.length).put(iv).put(ciphertext).array();
} catch (GeneralSecurityException e) {
throw new SerializationException("Encryption failed", e);
}
}
@Override
public byte[] serialize(String topic, String data) {
throw new UnsupportedOperationException("Headers are required to carry the key id");
}
}
The matching deserializer reads enc.key.id, obtains the DEK, splits the IV from the ciphertext, and decrypts with the same associated data (the topic name). In production, prefer a maintained library (for example a schema-aware field-level encryption feature of your Schema Registry vendor) over hand-rolled crypto; the example shows the moving parts, not a hardened implementation.
Key Takeaways
- Kafka stores records unencrypted on disk. At-rest encryption must come from the storage layer, the client, or both.
- Disk or volume encryption is cheap and transparent but does not protect against anyone with access to a running broker.
- End-to-end encryption keeps brokers blind to the data, at the cost of key management, lost compression, and harder integration with processing tools.
- Leave record keys unencrypted (or encrypt them deterministically) so partitioning and compaction keep working; do not put secrets in keys.
- Per-subject data keys enable crypto-shredding for erasure requirements.
Practice
- (Beginner) Produce a few records to a local topic, then locate the segment file under
log.dirsand print it withkafka-dump-log.sh --files <segment> --print-data-log. Note what an attacker with disk access sees.- (Intermediate) Complete the
AesGcmSerializerwith a matching deserializer, produce through it, and repeat exercise 1. Then enablecompression.type=zstdand compare batch sizes with and without encryption.- Interview question: "A customer asks you to delete all of their data from a Kafka topic with 7 years of retention. How could the system have been designed to make this feasible?" Hint: immutable logs cannot easily drop single records; think about keys that are scoped per customer and what happens to ciphertext when its key no longer exists.
<a id="112-authentication"></a>
11.2 Authentication
Encryption hides data from outsiders, but the broker still does not know who is on the other end of a connection. This section explains how Kafka combines listeners and security protocols, then walks through each authentication mechanism (SASL/PLAIN, SASL/SCRAM, Kerberos, OAuth 2.0, and mutual TLS) and when to choose it.
Listeners and Security Protocols
Theory
Different clients reach Kafka from different places and with different credentials: microservices inside the cluster network, partner systems over the internet, brokers replicating among themselves, administrators running CLI tools. Forcing all of them onto one port with one authentication method rarely works. Kafka's answer is the listener: a named endpoint (host and port) with its own security protocol and its own mechanism settings (Section 2.2 introduced listeners; this topic focuses on their security side).
Every listener uses exactly one of four security protocols:
| Security protocol | Encryption | Authentication | Resulting principal | Use |
|---|---|---|---|---|
PLAINTEXT | None | None | User:ANONYMOUS | Local development only |
SSL | TLS | None, or client certificate (mTLS) | User:ANONYMOUS, or derived from the certificate DN | Internal traffic with mTLS, or encryption without identity |
SASL_PLAINTEXT | None | SASL mechanism | User:<SASL identity> | Rarely justified; credentials or tokens travel in clear text |
SASL_SSL | TLS | SASL mechanism | User:<SASL identity> | Default choice for authenticated client traffic |
SASL (Simple Authentication and Security Layer) is a framework, not a mechanism: it defines how a client and server exchange authentication messages, and plug-in mechanisms define what is exchanged. Kafka supports five mechanisms:
| Mechanism | Credential | Credential store | Strengths | Weaknesses |
|---|---|---|---|---|
PLAIN | Username + password | Static JAAS file, or custom callback (LDAP) | Simple, easy to integrate | Password sent to broker (needs TLS); static by default |
SCRAM-SHA-256/512 | Username + password | KRaft metadata (salted hashes) | Password never sent; credentials managed at runtime | Kafka-only user database |
GSSAPI | Kerberos ticket | KDC / Active Directory | Enterprise SSO, no passwords in apps | Operational complexity (KDC, keytabs, DNS, clocks) |
OAUTHBEARER | JWT access token | External identity provider | Central identity, short-lived tokens, cloud-native | Requires an IdP; token and claim configuration |
(SSL mTLS) | X.509 client certificate | Your PKI | No passwords; strong identity | Revocation and certificate distribution |
How a SASL_SSL connection authenticates
Client Broker (listener EXTERNAL = SASL_SSL)
| -- TCP + TLS handshake (encryption, broker identity) --> |
| -- SaslHandshake(mechanism="SCRAM-SHA-512") -----------> | is it in sasl.enabled.mechanisms?
| <- mechanisms supported -------------------------------- |
| -- SaslAuthenticate(bytes) <--------------------------> | one or more rounds, mechanism-specific
| <- success: session bound to principal User:billing ---- |
| -- Metadata, Produce, Fetch (authorized as User:billing) ->|
| |
| ... after connections.max.reauth.ms the client re-authenticates on the same connection
Authentication happens once per connection, not per request. connections.max.reauth.ms (KIP-368, disabled by default) forces clients to re-authenticate periodically; without it, a revoked password or expired token keeps working on already-open connections until they close.
Configuring mechanisms per listener
Mechanism-specific settings are scoped with listener.name.<listener>.<mechanism>. (listener and mechanism in lowercase):
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
listener.security.protocol.map=INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL,CONTROLLER:SSL
inter.broker.listener.name=INTERNAL
# Which mechanisms each listener accepts
listener.name.internal.sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.external.sasl.enabled.mechanisms=OAUTHBEARER,SCRAM-SHA-512
# Mechanism the broker itself uses when it connects to other brokers
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
# JAAS login context per listener + mechanism (the broker's own credentials as a client)
listener.name.internal.scram-sha-512.sasl.jaas.config=\
org.apache.kafka.common.security.scram.ScramLoginModule required \
username="broker" password="broker-secret";
Clients choose one listener (by connecting to its port) and one mechanism:
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="billing" password="billing-secret";
ssl.truststore.location=/etc/app/kafka-ca.crt
ssl.truststore.type=PEM
The sasl.jaas.config property replaces the old practice of a separate JAAS file passed with -Djava.security.auth.login.config, which still works but applies one configuration to the whole JVM.
Troubleshooting signals
| Symptom | Likely cause |
|---|---|
Client hangs, then Connection to node -1 ... terminated during authentication | Client protocol does not match the listener (for example PLAINTEXT client on a SASL_SSL port) |
UnsupportedSaslMechanismException | Mechanism not in the listener's sasl.enabled.mechanisms |
SaslAuthenticationException: Authentication failed | Wrong credentials; the broker metric failed-authentication-total increases |
SSLHandshakeException | Certificate, truststore, or hostname problem (Section 11.1) |
Key Takeaways
- A listener is an endpoint plus a security protocol; different client populations should get different listeners.
SASL_SSLis the default choice for authenticated client traffic;PLAINTEXTandSASL_PLAINTEXTdo not belong in production.- SASL is a framework; PLAIN, SCRAM, GSSAPI, and OAUTHBEARER are mechanisms with very different credential stores.
- Scope mechanism configuration with
listener.name.<listener>.<mechanism>.and give clients their credentials withsasl.jaas.config. - Set
connections.max.reauth.msso revoked credentials stop working on existing connections.
Practice
- (Beginner) For each security protocol in the table, state what the principal of an unauthenticated client would be and whether a passive network observer could read the records.
- (Intermediate) Configure a broker with an
INTERNALlistener usingSASL_SSL+ SCRAM and anEXTERNALlistener usingSASL_SSL+ OAUTHBEARER and SCRAM. Connect a client to the wrong port with the wrong mechanism and collect the resulting error messages.- Interview question: "A user's password was revoked an hour ago, but their application is still producing successfully. Why, and how would you prevent it?" Hint: when does Kafka authenticate a client, and which broker setting changes that?
SASL/PLAIN
Theory
PLAIN is the simplest SASL mechanism: the client sends its username and password, and the server checks them. It is easy to understand and to integrate with existing username/password directories, but the password itself crosses the network, so PLAIN must always run over SASL_SSL. Do not confuse the SASL mechanism PLAIN with the security protocol PLAINTEXT; they are unrelated.
Analogy: PLAIN is showing your ID card to a guard. Safe inside a sealed room (TLS), unwise in a crowded street.
Out of the box, the broker validates passwords against a static list in its JAAS configuration. Adding a user or changing a password means editing the configuration on every broker and restarting them, so the built-in store is suitable only for a handful of service accounts. For anything larger, plug in a server callback handler that checks credentials against an external system such as LDAP or a secrets service.
Example: static users on the broker
listener.name.external.sasl.enabled.mechanisms=PLAIN
listener.name.external.plain.sasl.jaas.config=\
org.apache.kafka.common.security.plain.PlainLoginModule required \
username="broker" password="broker-secret" \
user_broker="broker-secret" \
user_billing="billing-secret" \
user_analytics="analytics-secret";
# username/password -> credentials this broker uses as a CLIENT (inter-broker, if this listener is used)
# user_<name>="..." -> accepted users and their passwords (stored in clear text on disk)
Example: client
Properties props = new Properties();
props.put("bootstrap.servers", "broker1.kafka.example.com:9093");
props.put("security.protocol", "SASL_SSL"); // never SASL_PLAINTEXT with PLAIN
props.put("sasl.mechanism", "PLAIN");
props.put("sasl.jaas.config",
"org.apache.kafka.common.security.plain.PlainLoginModule required "
+ "username=\"billing\" password=\"" + System.getenv("KAFKA_PASSWORD") + "\";");
props.put("ssl.truststore.type", "PEM");
props.put("ssl.truststore.location", "/etc/app/kafka-ca.crt");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("orders", "o-1001", "{\"amount\":42.0}"));
}
Example: validating PLAIN credentials against LDAP
import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.plain.PlainAuthenticateCallback;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.login.AppConfigurationEntry;
import java.util.List;
import java.util.Map;
public class LdapPlainCallbackHandler implements AuthenticateCallbackHandler {
private LdapClient ldap; // your LDAP wrapper, with connection pooling
@Override
public void configure(Map<String, ?> configs, String mechanism, List<AppConfigurationEntry> jaasEntries) {
ldap = new LdapClient((String) configs.get("ldap.url")); // custom broker property
}
@Override
public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
String username = null;
for (Callback callback : callbacks) {
if (callback instanceof NameCallback nc) {
username = nc.getDefaultName(); // username sent by the client
} else if (callback instanceof PlainAuthenticateCallback pc) {
boolean ok = ldap.bind("uid=" + username + ",ou=services,dc=example,dc=com",
new String(pc.password()));
pc.authenticated(ok); // false -> SaslAuthenticationException on client
} else {
throw new UnsupportedCallbackException(callback);
}
}
}
@Override
public void close() { ldap.close(); }
}
listener.name.external.plain.sasl.server.callback.handler.class=com.example.kafka.LdapPlainCallbackHandler
ldap.url=ldaps://ldap.example.com:636
The handler runs on the broker's network path for every new connection, so cache successful binds briefly and set tight LDAP timeouts; a slow directory otherwise turns into slow or failing client connections.
Key Takeaways
- PLAIN sends the password to the broker, so it is only acceptable over
SASL_SSL. - The built-in user list lives in clear text in broker configuration and requires restarts to change.
- A custom
AuthenticateCallbackHandlerlets PLAIN validate against LDAP or other directories, which is its main production use. PLAIN(a SASL mechanism) andPLAINTEXT(a security protocol with no encryption or authentication) are different things.
Practice
- (Beginner) Configure a local broker with two PLAIN users. Produce as one user, then try a wrong password and record the client exception.
- (Intermediate) Implement a callback handler that validates users against a properties file reloaded every 30 seconds, so passwords can change without broker restarts. Discuss what happens to already-connected clients.
- Interview question: "Why might a company choose SASL/PLAIN over SCRAM even though SCRAM never sends the password?" Hint: where do the credentials live in each case, and who owns that system?
SASL/SCRAM
Theory
SCRAM (Salted Challenge Response Authentication Mechanism, RFC 5802) fixes PLAIN's two main weaknesses. First, the password never crosses the network: the client proves it knows the password by answering a challenge. Second, users are stored in the cluster itself and can be created, changed, and removed at runtime with an admin command, no restarts needed. In KRaft mode, the broker stores only salted, iterated hashes of passwords in the metadata log, never the passwords themselves.
Analogy: instead of showing the guard your password, the guard gives you a random puzzle that only someone who knows the password can solve. Overhearing the answer does not help an eavesdropper with the next puzzle.
Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. The exchange looks like this:
Client Broker
| -- client-first: username=billing, client nonce ------> |
| | look up salt, iterations, StoredKey, ServerKey
| <- server-first: combined nonce, salt, iterations ----- |
| |
| derive SaltedPassword = PBKDF2(password, salt, iterations)
| compute ClientProof from SaltedPassword and the full conversation
| -- client-final: combined nonce, ClientProof ---------> |
| | verify proof against StoredKey
| <- server-final: ServerSignature ---------------------- |
| verify ServerSignature (the broker also knew the secret: mutual authentication)
Because the stored hashes are sensitive (they can be attacked offline), keep using TLS: SCRAM over SASL_PLAINTEXT resists password sniffing but still exposes all data and metadata traffic.
Example: managing credentials
# Create or update a user (iterations >= 4096; higher is slower to brute-force)
kafka-configs.sh --bootstrap-server broker1.kafka.example.com:9093 \
--command-config admin.properties --alter \
--add-config 'SCRAM-SHA-512=[iterations=8192,password=billing-secret]' \
--entity-type users --entity-name billing
# Show which mechanisms a user has credentials for (no secrets are printed)
kafka-configs.sh --bootstrap-server broker1.kafka.example.com:9093 \
--command-config admin.properties --describe --entity-type users --entity-name billing
# Remove the credential: new authentications fail immediately
kafka-configs.sh --bootstrap-server broker1.kafka.example.com:9093 \
--command-config admin.properties --alter \
--delete-config 'SCRAM-SHA-512' --entity-type users --entity-name billing
The same operations are available in the Admin API (KIP-554):
try (Admin admin = Admin.create(adminProps)) {
admin.alterUserScramCredentials(List.of(
new UserScramCredentialUpsertion(
"billing",
new ScramCredentialInfo(ScramMechanism.SCRAM_SHA_512, 8192), // mechanism + iterations
System.getenv("BILLING_PASSWORD"))))
.all().get(); // completes once committed to metadata
}
Bootstrapping the first credentials
There is a chicken-and-egg problem: if brokers use SCRAM for inter-broker traffic, they need a SCRAM user before the cluster can accept the admin command that would create it. Kafka solves this by writing credentials into the metadata log when storage is formatted (KIP-900):
kafka-storage.sh format --cluster-id "$CLUSTER_ID" --config server.properties \
--add-scram 'SCRAM-SHA-512=[name=broker,password=broker-secret]' \
--add-scram 'SCRAM-SHA-512=[name=admin,password=admin-secret]'
Client configuration
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="billing" password="billing-secret";
SCRAM also underpins Kafka delegation tokens: a client authenticated with Kerberos or mTLS can obtain a short-lived token and pass it to distributed workers, which authenticate with it through the SCRAM mechanism (tokenauth=true in the JAAS configuration), so workers never hold the long-term credential.
Key Takeaways
- SCRAM is a challenge-response mechanism: the password never crosses the wire, and broker and client authenticate each other.
- Credentials are salted hashes stored in the KRaft metadata log and managed at runtime with
kafka-configs.shor the Admin API. - Use
kafka-storage.sh format --add-scramto bootstrap credentials that brokers need before the cluster is running. - SCRAM is not supported on the controller listener; use mTLS or another mechanism there.
- Prefer
SCRAM-SHA-512with at least 4096 iterations, always over TLS.
Practice
- (Beginner) Create a user with
SCRAM-SHA-256, connect a console producer, then delete the credential and show that a new connection fails while an existing one (without re-authentication) keeps working.- (Intermediate) Write a small Java tool that rotates a service's SCRAM password using the Admin API, then rolls the new password into the client's secret store. Design the order of steps so the service never fails to authenticate.
- Interview question: "An attacker records a complete SCRAM authentication exchange. What can and cannot they do with it?" Hint: think about nonces, what the proof is derived from, and how an offline dictionary attack would work against a weak password.
SASL/GSSAPI (Kerberos)
Theory
Many enterprises already run Kerberos, usually as part of Active Directory, as the single source of identity for people and services. Kafka's GSSAPI mechanism lets clients authenticate with Kerberos tickets, so applications never hold Kafka-specific passwords, identities are managed in one place, and disabling an account in AD disables its Kafka access.
Kerberos is a ticket-based protocol with a trusted third party, the Key Distribution Center (KDC):
Analogy: a theme park. You prove who you are once at the main gate (the KDC) and receive a wristband (the ticket-granting ticket). At each ride (a service such as Kafka), you exchange the wristband for a ride-specific ticket that the ride operator can verify without calling the main gate.
+-------------------- KDC (e.g. Active Directory) --------------------+
| Authentication Service (AS) Ticket Granting Service (TGS) |
+---------^------------------|------------^-------------|--------------+
(1) AS-REQ with keytab | (3) TGS-REQ for |
for billing@EXAMPLE.COM | kafka/broker1... |
| (2) TGT | | (4) service ticket
| v | v
+------------------------------------------------------------+
| Client: billing service |
+------------------------------------------------------------+
| (5) SASL GSSAPI: present service ticket
v
+------------------------------------------------------------+
| Broker: kafka/broker1.kafka.example.com@EXAMPLE.COM |
| decrypts the ticket with its own keytab -> User:billing |
+------------------------------------------------------------+
Key vocabulary:
| Term | Meaning | Example |
|---|---|---|
| Principal | A Kerberos identity: primary/instance@REALM | kafka/broker1.kafka.example.com@EXAMPLE.COM, billing@EXAMPLE.COM |
| Service name | The primary part of the broker principals; clients must use the same value | kafka |
| Keytab | File holding a principal's long-term keys, used instead of a password | /etc/security/keytabs/kafka.service.keytab |
| Realm | Kerberos administrative domain, usually the uppercase DNS domain | EXAMPLE.COM |
The client builds the broker principal from the service name and the hostname it connected to, so DNS (forward and reverse) must resolve consistently, and clocks must be synchronized (default tolerance is 5 minutes) or tickets are rejected.
Example: creating principals and keytabs (MIT Kerberos)
# One principal per broker host, plus one per client application
kadmin.local -q "addprinc -randkey kafka/broker1.kafka.example.com@EXAMPLE.COM"
kadmin.local -q "addprinc -randkey billing@EXAMPLE.COM"
kadmin.local -q "ktadd -k /etc/security/keytabs/kafka.service.keytab kafka/broker1.kafka.example.com@EXAMPLE.COM"
kadmin.local -q "ktadd -k /etc/security/keytabs/billing.keytab billing@EXAMPLE.COM"
chmod 400 /etc/security/keytabs/*.keytab # keytabs are equivalent to passwords
klist -kt /etc/security/keytabs/kafka.service.keytab
Example: broker and client configuration
# broker1 server.properties
listener.name.external.sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
listener.name.external.gssapi.sasl.jaas.config=\
com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true storeKey=true \
keyTab="/etc/security/keytabs/kafka.service.keytab" \
principal="kafka/broker1.kafka.example.com@EXAMPLE.COM";
# JVM: export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"
# client.properties
security.protocol=SASL_SSL
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka # must match the broker principals' primary
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true storeKey=true \
keyTab="/etc/security/keytabs/billing.keytab" \
principal="billing@EXAMPLE.COM";
By default, sasl.kerberos.principal.to.local.rules=DEFAULT shortens principals from the default realm to their first component, so billing@EXAMPLE.COM becomes User:billing in ACLs. Custom rules are covered in Section 11.3 (Principal Mapping).
Common failures
| Error fragment | Typical cause |
|---|---|
Server not found in Kerberos database | Client connected via a hostname (or IP) that does not match the broker principal; check DNS and advertised.listeners |
Clock skew too great | NTP not running on client, broker, or KDC |
Checksum failed / Pre-authentication information was invalid | Keytab is stale (password or key version changed in the KDC) |
Cannot locate default realm | krb5.conf missing or not passed via -Djava.security.krb5.conf |
Add -Dsun.security.krb5.debug=true to KAFKA_OPTS for a detailed trace when debugging.
Key Takeaways
- GSSAPI integrates Kafka with Kerberos and Active Directory, giving centrally managed identities without application passwords.
- Brokers and clients authenticate with keytabs; the client derives the broker principal from
sasl.kerberos.service.nameand the broker hostname. - DNS consistency and clock synchronization are hard requirements.
- Principals are mapped to short names (by default the first component) for ACLs.
Practice
- (Beginner) Given the principals
kafka/broker3.kafka.example.com@EXAMPLE.COMandreporting@EXAMPLE.COM, write the service name the client must configure and the ACL principal the client will have withDEFAULTrules.- (Intermediate) Run an MIT KDC in a container alongside a single Kafka broker, configure GSSAPI, and deliberately break DNS (connect via IP address). Explain the resulting error.
- Interview question: "Kerberos authentication to Kafka works from some application servers but not others, with identical client configuration. What would you check?" Hint: the configuration is identical, so look at what differs per host: name resolution, clocks, keytab versions, and
krb5.conf.
SASL/OAUTHBEARER
Theory
Modern organizations increasingly centralize identity in an OAuth 2.0 / OpenID Connect identity provider (IdP) such as Keycloak, Okta, Microsoft Entra ID, or a cloud IAM service. Services already obtain short-lived access tokens (usually JWTs) from it to call HTTP APIs. OAUTHBEARER lets them use the same tokens for Kafka: no Kafka-specific credentials, centralized onboarding and offboarding, and tokens that expire in minutes rather than passwords that live for years.
Analogy: a concert wristband with a printed expiry time and a hologram. The venue staff (broker) does not need to call the ticket office; they check the hologram (signature) and the time printed on it.
+---------------------+ (1) client_credentials grant +---------------------------+
| billing service | --- client_id + client_secret ---> | Identity Provider (IdP) |
| (Kafka client) | <-- signed JWT access token ------ | token endpoint |
+----------+----------+ | JWKS endpoint (public |
| | signing keys) |
| (2) SaslAuthenticate: JWT +-------------^-------------+
v | (3) fetch & cache
+---------------------+ | public keys
| Kafka broker | ---------------------------------------------------+
| validates: signature, exp, iss, aud -> principal = value of "sub" claim
+---------------------+
(4) client refreshes the token before it expires; broker re-authentication
(connections.max.reauth.ms) closes connections whose token expired
Token validation happens entirely on the broker using cached public keys, so the IdP is not on the hot path of every connection. Kafka 3.1 added built-in OIDC support (KIP-768); before that, OAUTHBEARER required custom callback handlers. Without the OIDC callback handlers, OAUTHBEARER defaults to unsecured (unsigned) JWTs meant only for development.
Example: broker configuration
listener.name.external.sasl.enabled.mechanisms=OAUTHBEARER
listener.name.external.oauthbearer.sasl.jaas.config=\
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
listener.name.external.oauthbearer.sasl.server.callback.handler.class=\
org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler
sasl.oauthbearer.jwks.endpoint.url=https://idp.example.com/realms/kafka/protocol/openid-connect/certs
sasl.oauthbearer.expected.issuer=https://idp.example.com/realms/kafka # reject tokens from other issuers
sasl.oauthbearer.expected.audience=kafka-prod # reject tokens minted for other APIs
sasl.oauthbearer.sub.claim.name=sub # claim that becomes the principal name
connections.max.reauth.ms=900000 # re-check tokens at least every 15 minutes
# Kafka 4.x only contacts IdP URLs that are explicitly allowed (JVM system property):
# KAFKA_OPTS="-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=https://idp.example.com/realms/kafka/protocol/openid-connect/certs"
Example: Java client using the client credentials grant
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler
sasl.oauthbearer.token.endpoint.url=https://idp.example.com/realms/kafka/protocol/openid-connect/token
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
clientId="billing-service" \
clientSecret="${file:/etc/app/secrets.properties:oauth.client.secret}" \
scope="kafka";
sasl.login.refresh.window.factor=0.8 # refresh at 80% of token lifetime
The client JVM also needs the token endpoint URL in org.apache.kafka.sasl.oauthbearer.allowed.urls on Kafka 4.x.
Example: Python client
from confluent_kafka import Consumer
consumer = Consumer({
"bootstrap.servers": "broker1.kafka.example.com:9093",
"security.protocol": "SASL_SSL",
"sasl.mechanisms": "OAUTHBEARER",
"sasl.oauthbearer.method": "oidc", # librdkafka fetches and refreshes tokens itself
"sasl.oauthbearer.client.id": "billing-service",
"sasl.oauthbearer.client.secret": "s3cr3t",
"sasl.oauthbearer.token.endpoint.url":
"https://idp.example.com/realms/kafka/protocol/openid-connect/token",
"sasl.oauthbearer.scope": "kafka",
"group.id": "billing",
})
consumer.subscribe(["orders"])
Design considerations
| Topic | Guidance |
|---|---|
| Principal claim | sub is often an opaque UUID. Many IdPs let you map the client ID to a custom claim (for example client_id or azp); set sasl.oauthbearer.sub.claim.name accordingly so ACLs stay readable |
| Token lifetime | 5 to 60 minutes; shorter means faster revocation but more IdP traffic |
| IdP outage | Existing tokens remain valid until expiry; new clients and refreshes fail. Monitor the IdP as a Kafka dependency |
| Scopes and groups | The built-in validator authenticates; it does not turn scopes or group claims into permissions. That requires a custom principal builder or authorizer (Section 11.3) |
| Managed services | Cloud providers often offer their own OAUTHBEARER variants (for example Amazon MSK IAM, Azure Event Hubs with Entra ID) |
Key Takeaways
- OAUTHBEARER lets Kafka reuse an organization's OAuth 2.0 / OIDC identity provider with short-lived JWT access tokens.
- Brokers validate tokens locally using the IdP's JWKS keys plus
exp,iss, andaud; the principal comes from a configurable claim. - Use the OIDC callback handlers from KIP-768; the default handlers accept unsigned tokens and are for development only.
- Combine with
connections.max.reauth.msso connections cannot outlive their tokens. - Kafka 4.x requires IdP URLs to be allow-listed via
org.apache.kafka.sasl.oauthbearer.allowed.urls.
Practice
- (Beginner) Decode a JWT from your IdP (for example with
jq -R 'split(".") | .[1] | @base64d | fromjson') and identify the claims Kafka checks and the one that becomes the principal.- (Intermediate) Run Keycloak and Kafka locally, create two clients in Keycloak, configure OAUTHBEARER, and verify that a token with the wrong audience is rejected. Then set a 2-minute token lifetime and observe refresh and re-authentication in the client logs.
- Interview question: "Compare OAUTHBEARER and SCRAM for a company with 300 microservices across several teams." Hint: consider where identities are created and revoked, credential lifetime, blast radius of a leaked secret, and operational dependencies.
mTLS Authentication
Theory
Section 11.1 used TLS so that clients could verify the broker. Mutual TLS (mTLS) adds the reverse: the broker requires a client certificate and verifies it against its truststore. The client's identity is then the certificate's subject, so no password, token, or ticket is needed. mTLS is popular for service-to-service traffic, for internal Kafka listeners, and in environments that already issue workload certificates (service meshes, SPIFFE/SPIRE, Kubernetes with cert-manager).
Analogy: two embassies exchanging diplomatic credentials signed by a mutually recognized authority. Each side checks the other's seal before talking.
Client (keystore: CN=billing-service) Broker (ssl.client.auth=required)
| -- ClientHello -------------------------------------> |
| <- ServerHello, broker certificate, CertificateRequest |
| verify broker cert (truststore + hostname) |
| -- client certificate + proof of private key -------> |
| | verify client cert against truststore
| | principal = User:CN=billing-service,OU=payments,O=Example,C=IT
| <=========== encrypted, authenticated session ======> |
ssl.client.auth controls the behavior on SSL listeners:
| Value | Effect |
|---|---|
none | Client certificates not requested (TLS only, principal ANONYMOUS) |
requested | Certificate requested but optional; clients without one become ANONYMOUS (useful during migrations, risky if ACLs allow ANONYMOUS) |
required | Handshake fails without a valid client certificate |
On a SASL_SSL listener the principal always comes from SASL. A required client certificate there adds a second factor at the transport level but does not change the identity used for ACLs.
Example: issuing a client certificate and configuring the client
# Key + CSR for the billing service
openssl req -new -newkey rsa:3072 -nodes -keyout billing.key -out billing.csr \
-subj "/C=IT/O=Example/OU=payments/CN=billing-service"
# Sign for client authentication only, 30 days
printf "extendedKeyUsage = clientAuth\n" > client.ext
openssl x509 -req -in billing.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 30 -sha256 -extfile client.ext -out billing.crt
# broker: require certificates on the EXTERNAL SSL listener
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL,CONTROLLER:SSL
listener.name.external.ssl.client.auth=required
# client.properties: PEM files directly, no keytool needed
security.protocol=SSL
ssl.keystore.type=PEM
ssl.keystore.location=/etc/app/billing.pem # private key + certificate chain in one PEM file
ssl.truststore.type=PEM
ssl.truststore.location=/etc/app/kafka-ca.crt
The billing.pem file contains the private key followed by the certificate (and intermediates): cat billing.key billing.crt > billing.pem. If the key is encrypted, set ssl.key.password.
The principal problem: long DNs
By default the principal is the full distinguished name, so ACLs would read User:CN=billing-service,OU=payments,O=Example,C=IT. That is verbose and fragile (a reissued certificate with a different OU silently loses all permissions). Use ssl.principal.mapping.rules to extract a short name, typically the CN:
ssl.principal.mapping.rules=RULE:^CN=([^,]+),OU=payments,O=Example,C=IT$/$1/,DEFAULT
# CN=billing-service,OU=payments,O=Example,C=IT -> User:billing-service
Rule syntax and more advanced mapping are covered in Section 11.3 (Principal Mapping).
Revocation
A leaked password can be changed; a leaked certificate remains valid until it expires unless verifiers check revocation. Kafka does not check CRLs or OCSP by default (the JVM can be configured to, but it is rarely done in practice). Common strategies:
| Strategy | How |
|---|---|
| Short-lived certificates | Days or hours, with automated renewal; a leaked certificate expires quickly |
| Deny ACL | Add a DENY ACL for the compromised principal immediately (Section 11.3) |
| Re-issue CA | Last resort for a compromised CA; follow the phased CA rotation in Section 11.1 |
Choosing an authentication mechanism
| Situation | Recommended |
|---|---|
| Small team, few services, no identity infrastructure | SCRAM over SASL_SSL |
| Existing Active Directory / Kerberos estate | GSSAPI |
| Central OAuth/OIDC IdP, cloud-native services | OAUTHBEARER |
| Existing PKI, service mesh, or internal broker/controller traffic | mTLS |
| Existing LDAP with passwords | PLAIN with a custom callback handler, over TLS |
Key Takeaways
- mTLS authenticates clients with certificates:
ssl.client.auth=requiredon anSSLlistener, client keystore on the application side. - The default principal is the full certificate DN; map it to a short name with
ssl.principal.mapping.rules. - On
SASL_SSLlisteners, the principal comes from SASL even if a client certificate is required. - Kafka does not check certificate revocation by default; rely on short-lived certificates and deny ACLs.
- Pick the mechanism that matches the identity infrastructure you already operate.
Practice
- (Beginner) Issue a client certificate, configure an
SSLlistener withssl.client.auth=required, and confirm that a client without a keystore fails during the TLS handshake.- (Intermediate) Set
ssl.client.auth=requestedand grant an ACL only to the mapped certificate principal. Connect with and without a certificate and explain which principal each connection gets and what it can do.- Interview question: "A private key for a production client certificate was committed to a public Git repository. Walk through your response." Hint: separate immediate containment on the Kafka side from fixing the root cause; consider what Kafka does and does not check on each handshake.
<a id="113-authorization-and-governance"></a>
11.3 Authorization and Governance
Knowing who a client is does not yet say what it may do. This section covers how Kafka decides whether a principal may perform an operation (ACLs and the pluggable authorizer), how authenticated identities become principals, how quotas stop one tenant from starving others, and how to keep an audit trail of security-relevant activity.
Access Control Lists (ACLs)
Theory
Once authentication is enabled, every client has a principal, but by default Kafka still needs an authorizer to enforce anything. With an authorizer configured and no permissions granted, all non-super-users are denied everything. Permissions are granted with Access Control Lists (ACLs), each of which reads as one sentence:
Principal P is [Allowed | Denied] Operation O from Host H on Resources matching R
| | | | |
User:billing ALLOW READ * (any) Topic, LITERAL, "orders"
Analogy: a building's badge system. The badge (principal) is checked at each door (resource), and the access list says which doors it opens, for which actions (enter, open cabinets), and from which entrance (host).
Resources, operations, and patterns
| Resource type | Examples of operations | Notes |
|---|---|---|
Topic | Read, Write, Create, Delete, Describe, Alter, DescribeConfigs, AlterConfigs | The main data permissions |
Group | Read, Describe, Delete | Consumer group (and share group) membership and offset commits |
Cluster | Create, Alter, AlterConfigs, Describe, ClusterAction, IdempotentWrite | Cluster-wide admin; ClusterAction is for brokers and controllers |
TransactionalId | Write, Describe | Required for transactional producers |
DelegationToken | Describe | Viewing delegation tokens |
User | CreateTokens, DescribeTokens | Creating tokens on behalf of other users |
Resource patterns decide which names an ACL covers:
| Pattern type | Matches | Example |
|---|---|---|
LITERAL | Exactly that name, or every name if the name is * | orders |
PREFIXED | Every name starting with the string | payments. covers payments.refunds, payments.invoices |
Prefixed ACLs pair naturally with topic naming conventions (Chapter 15): give each team ownership of its prefix instead of granting topics one by one.
How a decision is made
request (principal, host, operation, resource)
|
v
super user? ----------------------------- yes --> ALLOW
| no
v
any matching DENY ACL? ------------------ yes --> DENY (deny always wins)
| no
v
any matching ALLOW ACL? ----------------- yes --> ALLOW
| no (Read/Write/Delete/Alter also imply Describe;
v AlterConfigs implies DescribeConfigs)
allow.everyone.if.no.acl.found=true AND
resource has no ACLs at all? ------------ yes --> ALLOW (dangerous default-open setting)
| no
v
DENY
What common clients need
| Client action | Required ACLs |
|---|---|
| Produce | Write on the topic |
| Idempotent produce (the default since 3.0) | Write on the topic is enough (KIP-679); IdempotentWrite on the cluster is no longer required |
| Transactional produce | Write on the topic, Write on the TransactionalId |
| Consume in a group | Read on the topic, Read on the group |
| Commit offsets inside a transaction | Read on the group (in addition to the transactional ACLs) |
| Create topics | Create on the topic (or prefix), or on the cluster |
| Kafka Streams app | Read on input topics, Write on output topics, All on topics prefixed with application.id (internal topics), Read on group application.id, Write on TransactionalId prefixed with application.id if exactly-once is enabled |
Example: managing ACLs with the CLI
ADMIN="--bootstrap-server broker1.kafka.example.com:9093 --command-config admin.properties"
# Consumer: read topic "orders" in group "billing"
kafka-acls.sh $ADMIN --add --allow-principal User:billing \
--operation Read --topic orders --group billing
# Convenience flags generate the usual sets of ACLs
kafka-acls.sh $ADMIN --add --allow-principal User:checkout --producer --topic orders
kafka-acls.sh $ADMIN --add --allow-principal User:fulfilment --consumer --topic orders --group fulfilment
# Transactional producer with a prefixed transactional.id
kafka-acls.sh $ADMIN --add --allow-principal User:ledger --producer --topic ledger. \
--transactional-id ledger- --resource-pattern-type prefixed
# Team ownership of a topic prefix
kafka-acls.sh $ADMIN --add --allow-principal User:payments-team --operation All \
--topic payments. --resource-pattern-type prefixed
# Block a compromised principal immediately (DENY beats every ALLOW)
kafka-acls.sh $ADMIN --add --deny-principal User:billing --operation All --topic '*'
# Inspect and remove
kafka-acls.sh $ADMIN --list --topic orders
kafka-acls.sh $ADMIN --list --principal User:billing
kafka-acls.sh $ADMIN --remove --allow-principal User:billing --operation Read --topic orders --force
Example: managing ACLs with the Admin API
try (Admin admin = Admin.create(adminProps)) {
AclBinding readOrders = new AclBinding(
new ResourcePattern(ResourceType.TOPIC, "orders", PatternType.LITERAL),
new AccessControlEntry("User:billing", "*", AclOperation.READ, AclPermissionType.ALLOW));
AclBinding useGroup = new AclBinding(
new ResourcePattern(ResourceType.GROUP, "billing", PatternType.LITERAL),
new AccessControlEntry("User:billing", "*", AclOperation.READ, AclPermissionType.ALLOW));
admin.createAcls(List.of(readOrders, useGroup)).all().get();
// List every ACL for the principal, whatever the resource
AclBindingFilter byPrincipal = new AclBindingFilter(
ResourcePatternFilter.ANY,
new AccessControlEntryFilter("User:billing", null, AclOperation.ANY, AclPermissionType.ANY));
admin.describeAcls(byPrincipal).values().get().forEach(System.out::println);
}
Managing ACLs by hand does not scale. Most teams keep them as code (a Git repository applied by a pipeline, Terraform's Kafka providers, Strimzi's KafkaUser resources, or a self-service portal) so every change is reviewed and reproducible.
Key Takeaways
- An ACL is a sentence: principal, allow or deny, operation, host, resource pattern.
- DENY always wins over ALLOW; with no matching ACL, the default is DENY (keep
allow.everyone.if.no.acl.found=false). - Consumers need
Readon both the topic and the group; transactional producers also needWriteon theTransactionalId. - Prefixed ACLs combined with naming conventions give teams ownership of namespaces with few rules.
- Treat ACLs as code, reviewed and applied by automation.
Practice
- (Beginner) Write the
kafka-acls.shcommands that letUser:reportingconsume from every topic starting withsales.using consumer groups starting withreporting-, and nothing else.- (Intermediate) Deploy a Kafka Streams application with exactly-once enabled on a secured cluster, starting with no ACLs. Add ACLs one error at a time and compare the final set with the table above.
- Interview question: "A consumer receives
GroupAuthorizationExceptioneven though it hasReadon the topic. Explain, and describe how you would design ACLs for 50 teams." Hint: there are two resources involved in consuming; for scale, think about prefixes and ownership.
Authorizer Implementations
Theory
ACL checks are performed by a pluggable component, the authorizer, configured with authorizer.class.name. Every request that touches a protected resource asks the authorizer a question such as "may User:billing READ Topic:orders from 10.0.4.21?". Without an authorizer, Kafka performs no authorization at all, regardless of how strong authentication is.
| Implementation | Where ACLs live | Status |
|---|---|---|
org.apache.kafka.metadata.authorizer.StandardAuthorizer | KRaft metadata log (__cluster_metadata) | Built-in default choice for Kafka 3.x KRaft and 4.x |
kafka.security.authorizer.AclAuthorizer | ZooKeeper | ZooKeeper-era authorizer, removed with ZooKeeper in Kafka 4.0 |
| Apache Ranger Kafka plugin | Ranger policy server | Centralized policies with UI and auditing across the Hadoop ecosystem |
| Open Policy Agent (OPA) plugins | Rego policies on an OPA server or sidecar | Policy-as-code, attribute-based decisions |
| Vendor RBAC (for example Confluent Platform) | Vendor metadata service | Role-based access with groups and LDAP integration |
| Custom | Anywhere | Implement org.apache.kafka.server.authorizer.Authorizer |
StandardAuthorizer in a KRaft cluster
ACLs are records in the metadata log, just like topics and configs. An admin client sends CreateAcls to any broker, the broker forwards it to the active controller, the controller appends the change, and every broker and controller applies it to its in-memory ACL cache. Authorization checks therefore never leave the node.
# On EVERY broker and controller
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:broker;User:controller;User:admin # semicolon-separated; bypass all ACLs
allow.everyone.if.no.acl.found=false
Super users must include the principals that brokers and controllers use to talk to each other; otherwise brokers cannot register or replicate (they need ClusterAction). Until a node has loaded the ACLs from the metadata log, StandardAuthorizer cannot answer for normal users, so the node only authorizes super users on listeners named in early.start.listeners (typically the controller listener) during startup.
Custom authorizers
The Authorizer interface is small:
public interface Authorizer extends Configurable, Closeable {
// Called at startup; futures tell the broker when each listener may accept requests
Map<Endpoint, ? extends CompletionStage<Void>> start(AuthorizerServerInfo serverInfo);
// Hot path: called for every request, often with several actions at once
List<AuthorizationResult> authorize(AuthorizableRequestContext requestContext, List<Action> actions);
// ACL management, used by CreateAcls / DeleteAcls / DescribeAcls requests
List<? extends CompletionStage<AclCreateResult>> createAcls(AuthorizableRequestContext ctx, List<AclBinding> bindings);
List<? extends CompletionStage<AclDeleteResult>> deleteAcls(AuthorizableRequestContext ctx, List<AclBindingFilter> filters);
Iterable<AclBinding> acls(AclBindingFilter filter);
}
The most practical way to customize is to extend StandardAuthorizer and add rules on top of normal ACLs, keeping ACL storage in KRaft.
Example: an extra policy check for sensitive topics
import org.apache.kafka.common.acl.AclOperation;
import org.apache.kafka.common.resource.ResourceType;
import org.apache.kafka.metadata.authorizer.StandardAuthorizer;
import org.apache.kafka.server.authorizer.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
public class PiiPolicyAuthorizer extends StandardAuthorizer {
private PolicyClient policy; // e.g. a client for an OPA sidecar, with a local cache
@Override
public void configure(Map<String, ?> configs) {
super.configure(configs);
policy = new PolicyClient((String) configs.get("pii.policy.url")); // custom broker property
}
@Override
public List<AuthorizationResult> authorize(AuthorizableRequestContext ctx, List<Action> actions) {
List<AuthorizationResult> aclResults = super.authorize(ctx, actions); // normal ACL decision first
List<AuthorizationResult> results = new ArrayList<>(actions.size());
for (int i = 0; i < actions.size(); i++) {
Action action = actions.get(i);
boolean piiRead = action.operation() == AclOperation.READ
&& action.resourcePattern().resourceType() == ResourceType.TOPIC
&& action.resourcePattern().name().startsWith("pii.");
if (piiRead && aclResults.get(i) == AuthorizationResult.ALLOWED
&& !policy.isAllowed(ctx.principal().getName(), action.resourcePattern().name())) {
results.add(AuthorizationResult.DENIED); // both the ACL AND the policy must allow
} else {
results.add(aclResults.get(i));
}
}
return results;
}
}
authorize runs on the request path of every produce and fetch, so external lookups must be cached and must fail predictably (usually closed: deny) when the policy service is down. Profile any custom authorizer under load before deploying it.
Related governance hooks are not authorizers but are configured alongside them: create.topic.policy.class.name and alter.config.policy.class.name validate topic creation and configuration changes (for example enforcing naming conventions, minimum replication factor, or maximum retention).
Key Takeaways
- Authorization is enforced only when
authorizer.class.nameis set; in KRaft useStandardAuthorizer, which stores ACLs in the metadata log. - Configure the same authorizer and
super.userson brokers and controllers; include the cluster's own principals insuper.users. - Ranger, OPA, and vendor RBAC plug in through the same interface for centralized or attribute-based policies.
- Custom logic is easiest as a subclass of
StandardAuthorizer; keepauthorizefast and cache external decisions. - Use create-topic and alter-config policies to enforce structural rules that ACLs cannot express.
Practice
- (Beginner) Enable
StandardAuthorizeron a local cluster, forget to add the broker principal tosuper.users, and record what fails. Then fix it.- (Intermediate) Implement a
CreateTopicPolicythat rejects topics not matching<domain>.<name>.v<N>or with a replication factor below 3. Test it withkafka-topics.sh --create.- Interview question: "Your organization wants Kafka permissions to follow LDAP group membership. How could you implement that?" Hint: ACLs grant to principals; consider where group information could be resolved: in the principal, in the authorizer, or in an external policy engine, and what each costs at request time.
Principal Mapping
Theory
Each authentication mechanism produces a different raw identity: a certificate DN, a Kerberos principal with host and realm, a SASL username, a JWT claim. ACLs, quotas, and audit logs need a stable, readable name. Principal mapping is the step that converts the raw identity into a KafkaPrincipal (a type, normally User, plus a name):
Authentication result Mapping KafkaPrincipal
--------------------------------------- ---------------------------------------- ----------------------
CN=billing-service,OU=payments,O=Ex,C=IT ssl.principal.mapping.rules User:billing-service
kafka/broker1.kafka.example.com@EX.COM sasl.kerberos.principal.to.local.rules User:broker
billing (SCRAM / PLAIN username) used as-is User:billing
JWT {"sub": "...", "azp": "billing"} sasl.oauthbearer.sub.claim.name=azp User:billing
anything principal.builder.class (custom code) User:<whatever you build>
Analogy: a passport, a driving licence, and a company badge all identify the same person. The front desk copies the name onto one visitor badge format so every door reader understands it.
Getting this right early matters: ACLs reference mapped names, so changing mapping rules later changes every principal and requires migrating every ACL.
SSL principal mapping rules
Format: RULE:<regex>/<replacement>/[L|U], evaluated in order; the first match wins; DEFAULT keeps the full DN. L and U convert the result to lower or upper case.
ssl.principal.mapping.rules=\
RULE:^CN=([a-zA-Z0-9.-]+),OU=ServiceAccounts,O=Example,C=IT$/svc-$1/L,\
RULE:^CN=([^,]+),OU=Brokers,.*$/broker/,\
RULE:^CN=([^,]+),.*$/$1/,\
DEFAULT
# CN=Billing.Service,OU=ServiceAccounts,O=Example,C=IT -> User:svc-billing.service
# CN=broker1.kafka.example.com,OU=Brokers,O=Example -> User:broker (one principal for all brokers)
# CN=alice,OU=Engineering,O=Example -> User:alice
The DN is in RFC 2253 form (comma-separated, most specific first, no spaces after commas), which is what the regular expressions must match.
Kerberos mapping rules
Format: RULE:[n:format](regex)s/pattern/replacement/[g], where n is the number of components in the principal, and format builds a string from $0 (realm), $1, $2 (components):
sasl.kerberos.principal.to.local.rules=\
RULE:[1:$1@$0](.*@EXAMPLE\.COM)s/@.*//,\
RULE:[2:$1@$0](kafka@EXAMPLE\.COM)s/.*/broker/,\
DEFAULT
# billing@EXAMPLE.COM -> [1:$1@$0] = billing@EXAMPLE.COM -> strip realm -> User:billing
# kafka/broker1.kafka.example.com@EXAMPLE.COM -> [2:$1@$0] = kafka@EXAMPLE.COM -> User:broker
Mapping every broker host principal to User:broker lets super.users contain one entry instead of one per host.
Custom principal builders
When rules are not enough (for example, identity encoded in a SAN URI, or a principal that should combine certificate fields), implement KafkaPrincipalBuilder. In KRaft clusters the builder must also implement KafkaPrincipalSerde, because brokers forward some requests to the controller together with the serialized principal.
Example: SPIFFE IDs from the certificate SAN
import org.apache.kafka.common.errors.SerializationException;
import org.apache.kafka.common.security.auth.*;
import javax.net.ssl.SSLPeerUnverifiedException;
import java.nio.charset.StandardCharsets;
import java.security.cert.CertificateParsingException;
import java.security.cert.X509Certificate;
import java.util.Collection;
import java.util.List;
public class SpiffePrincipalBuilder implements KafkaPrincipalBuilder, KafkaPrincipalSerde {
private static final String TRUST_DOMAIN = "spiffe://example.com/";
private static final int SAN_URI = 6; // GeneralName type for URIs
@Override
public KafkaPrincipal build(AuthenticationContext context) {
if (context instanceof SaslAuthenticationContext sasl) {
return new KafkaPrincipal(KafkaPrincipal.USER_TYPE, sasl.server().getAuthorizationID());
}
if (context instanceof SslAuthenticationContext ssl) {
try {
X509Certificate cert = (X509Certificate) ssl.session().getPeerCertificates()[0];
Collection<List<?>> sans = cert.getSubjectAlternativeNames(); // null if none
if (sans != null) {
for (List<?> san : sans) {
if ((Integer) san.get(0) == SAN_URI && ((String) san.get(1)).startsWith(TRUST_DOMAIN)) {
// spiffe://example.com/ns/payments/sa/billing -> User:ns/payments/sa/billing
return new KafkaPrincipal(KafkaPrincipal.USER_TYPE,
((String) san.get(1)).substring(TRUST_DOMAIN.length()));
}
}
}
} catch (SSLPeerUnverifiedException | CertificateParsingException e) {
// no client certificate, or unreadable SAN: fall through
}
}
return KafkaPrincipal.ANONYMOUS; // will match no ACLs
}
@Override
public byte[] serialize(KafkaPrincipal principal) {
return (principal.getPrincipalType() + ":" + principal.getName()).getBytes(StandardCharsets.UTF_8);
}
@Override
public KafkaPrincipal deserialize(byte[] bytes) {
String s = new String(bytes, StandardCharsets.UTF_8);
int sep = s.indexOf(':');
if (sep < 0) throw new SerializationException("Invalid principal: " + s);
return new KafkaPrincipal(s.substring(0, sep), s.substring(sep + 1));
}
}
principal.builder.class=com.example.kafka.SpiffePrincipalBuilder # same class on all brokers AND controllers
Key Takeaways
- Principal mapping turns mechanism-specific identities into
KafkaPrincipalnames used by ACLs, quotas, and logs. ssl.principal.mapping.rulesandsasl.kerberos.principal.to.local.rulescover most needs; rules are evaluated in order and the first match wins.- Map all brokers (and controllers) to one principal to keep
super.usersshort. - Custom
KafkaPrincipalBuilderclasses must also implementKafkaPrincipalSerdein KRaft and must be deployed on every node. - Decide on the naming scheme before creating ACLs; changing it later means migrating all of them.
Practice
- (Beginner) Write
ssl.principal.mapping.rulesthat mapCN=Reporting-Job,OU=Analytics,O=Example,C=ITtoUser:reporting-joband leave every other DN unchanged.- (Intermediate) Write Kerberos rules that map
etl/host1.example.com@EXAMPLE.COMandetl/host2.example.com@EXAMPLE.COMtoUser:etl, users from realmPARTNER.COMtoUser:partner-<name>, and test them by connecting with each principal.- Interview question: "Why does Kafka need to serialize principals in KRaft mode, and what breaks if brokers and controllers use different principal builders?" Hint: follow a
CreateTopicsrequest from the client, through the broker it lands on, to the node that actually authorizes and applies it.
Quotas
Theory
Authorization decides whether a client may do something; it says nothing about how much. On a shared cluster, a single misbehaving client (a backfill job reading from offset zero, a producer stuck in a retry loop, a CI pipeline creating hundreds of topics) can saturate network, disk, or request threads and degrade every other tenant. Quotas cap resource usage per client identity so one tenant cannot starve the others.
Analogy: water pressure regulators on each apartment in a building. Anyone can open every tap, but no single apartment can drop the pressure for everyone else.
Quota types
| Quota | Config key | Limits | Unit |
|---|---|---|---|
| Produce bandwidth | producer_byte_rate | Bytes produced | bytes/second, per broker |
| Fetch bandwidth | consumer_byte_rate | Bytes fetched | bytes/second, per broker |
| Request rate | request_percentage | Time spent on network and I/O threads | percentage of one thread (can exceed 100) |
| Controller mutations | controller_mutation_rate | Partitions created or deleted | partitions/second |
| Connection creation | connection_creation_rate (entity type ips) | New connections per IP | connections/second |
Broker-wide connection limits complement them: max.connections, max.connections.per.ip, and max.connections.per.ip.overrides.
Who a quota applies to
Quotas are set on a user principal, a client-id, or a combination, with defaults at each level. When a request arrives, the most specific matching configuration wins:
1. user=<name>, client-id=<id> most specific
2. user=<name>, client-id=<default>
3. user=<name>
4. user=<default>, client-id=<id>
5. user=<default>, client-id=<default>
6. user=<default>
7. client-id=<id>
8. client-id=<default> least specific
On authenticated clusters, user-based quotas are the reliable choice: anyone can pick any client.id, but not someone else's principal.
How throttling works
Brokers measure usage over a sliding window (quota.window.num samples of quota.window.size.seconds, by default 11 x 1 s). When a client exceeds its quota, the broker does not return an error. It computes a delay, returns the response with throttle_time_ms, and mutes the connection for that long. Well-behaved clients also wait before sending more. The client just slows down, and the throttling is visible in metrics:
| Side | Metric |
|---|---|
| Producer | produce-throttle-time-avg, produce-throttle-time-max |
| Consumer | fetch-throttle-time-avg, fetch-throttle-time-max |
| Broker | `kafka.server:type=Produce |
Quotas are enforced per broker. A producer_byte_rate of 10 MB/s lets a client write up to 10 MB/s to each broker that leads partitions it writes to; across 6 brokers that is up to 60 MB/s cluster-wide. Size quotas with partition placement in mind.
Example: setting quotas
ADMIN="--bootstrap-server broker1.kafka.example.com:9093 --command-config admin.properties"
# A baseline for every authenticated user
kafka-configs.sh $ADMIN --alter --entity-type users --entity-default \
--add-config 'producer_byte_rate=5242880,consumer_byte_rate=10485760,request_percentage=100'
# A bigger allowance for one user
kafka-configs.sh $ADMIN --alter --entity-type users --entity-name analytics \
--add-config 'consumer_byte_rate=52428800'
# A tighter limit for one client-id of that user (e.g. a backfill job)
kafka-configs.sh $ADMIN --alter --entity-type users --entity-name analytics \
--entity-type clients --entity-name backfill \
--add-config 'consumer_byte_rate=10485760'
# Stop a CI pipeline from creating/deleting partitions too fast
kafka-configs.sh $ADMIN --alter --entity-type users --entity-name ci-pipeline \
--add-config 'controller_mutation_rate=10'
# Limit connection storms from one host
kafka-configs.sh $ADMIN --alter --entity-type ips --entity-name 10.0.8.15 \
--add-config 'connection_creation_rate=20'
# Review
kafka-configs.sh $ADMIN --describe --entity-type users
kafka-configs.sh $ADMIN --describe --entity-type users --entity-name analytics --entity-type clients
Example: setting a quota from Java
try (Admin admin = Admin.create(adminProps)) {
ClientQuotaEntity entity = new ClientQuotaEntity(Map.of(ClientQuotaEntity.USER, "analytics"));
ClientQuotaAlteration alteration = new ClientQuotaAlteration(entity, List.of(
new ClientQuotaAlteration.Op("consumer_byte_rate", 52_428_800.0), // 50 MiB/s per broker
new ClientQuotaAlteration.Op("producer_byte_rate", null))); // null removes the quota
admin.alterClientQuotas(List.of(alteration)).all().get();
}
Replication between brokers is not subject to client quotas; it has its own throttles used during partition reassignment (Chapter 12).
Key Takeaways
- Quotas protect shared clusters: byte-rate quotas for bandwidth, request percentage for broker CPU time, controller mutation rate for metadata churn, connection rates for connection storms.
- Quotas attach to users, client-ids, or both, with defaults; the most specific match wins. Prefer user-based quotas on authenticated clusters.
- Exceeding a quota throttles (delays) the client instead of failing it; watch throttle-time metrics on both sides.
- Byte-rate quotas apply per broker, so the effective cluster-wide limit scales with the number of brokers involved.
Practice
- (Beginner) Set
producer_byte_rate=1048576for a user, runkafka-producer-perf-test.shas that user, and compare the achieved throughput andproduce-throttle-time-avgwith and without the quota.- (Intermediate) Define a quota scheme for a cluster shared by 20 teams: defaults, per-team overrides, and a policy for batch jobs. Explain how you would detect teams that are constantly throttled.
- Interview question: "A team complains their consumer is slow, but brokers look idle and there are no errors. What could be happening?" Hint: Kafka throttles silently by delaying responses; which client and broker metrics reveal it?
Audit Logging
Theory
Security controls are only half the story; you must also be able to answer "who did what, when, and was it allowed?" when investigating an incident or proving compliance (SOC 2, PCI DSS, ISO 27001, HIPAA). Audit logging records security-relevant events: authorization decisions, administrative changes (topics, ACLs, quotas, configs), and authentication failures.
Apache Kafka has no dedicated audit log subsystem. Instead, several building blocks provide the raw material:
| Source | What it records | Limits |
|---|---|---|
Authorizer logger (kafka.authorizer.logger) | Authorization decisions: DENIED at INFO, ALLOWED at DEBUG | Unstructured text; DEBUG volume is enormous |
Request logger (kafka.request.logger) | Every request with details at DEBUG/TRACE | Far too expensive to leave on in production |
| Broker server logs and metrics | Authentication failures (failed-authentication-total, log messages) | Mixed with operational logs |
Metadata log (__cluster_metadata) | Every committed change to topics, ACLs, configs, SCRAM credentials, quotas | Records the change, not who made it |
| Custom authorizer or principal builder | Whatever you emit, in any format | You maintain the code |
| Commercial distributions and managed services | Structured audit events (for example in CloudEvents format, or the cloud provider's audit service) | Vendor-specific |
Analogy: a bank vault's access system. The door controller (authorizer) logs every badge swipe; the CCTV (request log) records everything but is too much to review; the ledger (metadata log) shows what changed in the vault, but not whose hand moved it.
Example: routing authorizer decisions to their own file
Kafka 4.x uses Log4j 2 (config/log4j2.yaml). The shipped configuration already defines a separate appender; the relevant parts look like this:
Configuration:
Appenders:
RollingFile:
- name: AuthorizerAppender
fileName: "${sys:kafka.logs.dir}/kafka-authorizer.log"
filePattern: "${sys:kafka.logs.dir}/kafka-authorizer.log.%d{yyyy-MM-dd}"
PatternLayout:
pattern: "[%d] %p %m (%c)%n"
TimeBasedTriggeringPolicy:
modulate: true
interval: 1
Loggers:
Logger:
- name: kafka.authorizer.logger
level: INFO # INFO = denials only; DEBUG adds every allowed decision (very noisy)
additivity: false # keep these lines out of server.log
AppenderRef:
ref: AuthorizerAppender
A denied request produces a line similar to:
[2026-10-23 09:14:02,117] INFO Principal = User:billing is Denied operation = WRITE from host = 10.0.4.21
on resource = Topic:LITERAL:payments.refunds for request = Produce with resourceRefCount = 1
based on rule DefaultDeny (kafka.authorizer.logger)
Ship this file with your log agent (Fluent Bit, Vector, Filebeat) to a SIEM and alert on patterns: repeated denials for one principal, denials on sensitive prefixes, any ClusterAction denial.
Example: structured audit events from a custom authorizer
For JSON events with consistent fields, wrap StandardAuthorizer and log selected decisions to a dedicated logger:
import org.apache.kafka.common.acl.AclOperation;
import org.apache.kafka.common.protocol.ApiKeys;
import org.apache.kafka.metadata.authorizer.StandardAuthorizer;
import org.apache.kafka.server.authorizer.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Instant;
import java.util.List;
import java.util.Set;
public class AuditingAuthorizer extends StandardAuthorizer {
private static final Logger AUDIT = LoggerFactory.getLogger("kafka.audit"); // own appender, async
private static final Set<AclOperation> ALWAYS_AUDIT = Set.of(
AclOperation.CREATE, AclOperation.DELETE, AclOperation.ALTER, AclOperation.ALTER_CONFIGS);
@Override
public List<AuthorizationResult> authorize(AuthorizableRequestContext ctx, List<Action> actions) {
List<AuthorizationResult> results = super.authorize(ctx, actions);
for (int i = 0; i < actions.size(); i++) {
Action a = actions.get(i);
AuthorizationResult r = results.get(i);
// Audit every denial and every administrative operation; skip routine allowed reads/writes
if (r == AuthorizationResult.DENIED || ALWAYS_AUDIT.contains(a.operation())) {
AUDIT.info("{\"ts\":\"{}\",\"principal\":\"{}\",\"clientAddress\":\"{}\",\"listener\":\"{}\","
+ "\"clientId\":\"{}\",\"api\":\"{}\",\"operation\":\"{}\",\"resource\":\"{}:{}:{}\","
+ "\"result\":\"{}\"}",
Instant.now(), ctx.principal(), ctx.clientAddress().getHostAddress(), ctx.listenerName(),
ctx.clientId(), ApiKeys.forId(ctx.requestType()), a.operation(),
a.resourcePattern().resourceType(), a.resourcePattern().patternType(),
a.resourcePattern().name(), r);
}
}
return results;
}
}
Use an asynchronous appender, and do not produce audit events synchronously to the same Kafka cluster from inside authorize: it would add latency to every request and create a dependency loop. Escape values properly before using this pattern in production, since client IDs are client-controlled strings.
Reconstructing administrative history from the metadata log
Every ACL, config, and topic change is a record in __cluster_metadata, which you can inspect after the fact:
# Decode metadata records (ACL, config, topic, quota changes) from a controller's log directory
kafka-dump-log.sh --cluster-metadata-decoder \
--files /var/lib/kafka/meta/__cluster_metadata-0/00000000000000000000.log \
| grep -E 'ACCESS_CONTROL_ENTRY|CONFIG_RECORD|REMOVE_TOPIC'
# Or browse a metadata snapshot interactively
kafka-metadata-shell.sh --snapshot /var/lib/kafka/meta/__cluster_metadata-0/<latest>.checkpoint
This shows what changed and in which order, and the authorizer log (with the auditing authorizer above) shows who attempted it. Correlating both answers most incident questions.
What a useful audit trail captures
| Field | Why |
|---|---|
| Timestamp (UTC, synchronized clocks) | Ordering across brokers |
| Principal and client address | Who and from where |
| Listener and client ID | Which entry point and which application |
| API / operation / resource | What was attempted |
| Decision | Allowed or denied |
| Broker ID | Where it was enforced (each broker logs only what it handled) |
Key Takeaways
- Apache Kafka has no dedicated audit subsystem; combine the authorizer logger, authentication metrics, the metadata log, and optionally a custom auditing authorizer.
kafka.authorizer.loggerlogs denials at INFO and allowed decisions at DEBUG; route it to its own file and ship it to a SIEM.- Audit all denials and administrative operations; sample or skip high-volume allowed reads and writes.
- The metadata log records what changed but not who changed it.
- Keep audit logging asynchronous and outside the cluster it audits.
Practice
- (Beginner) Enable
StandardAuthorizer, configure the authorizer appender, and trigger a denial by producing to a topic withoutWrite. Find the resulting line and identify each field.- (Intermediate) Build the
AuditingAuthorizer, configure an asynchronous appender writing JSON lines, and create a dashboard (for example in Grafana Loki or OpenSearch) showing denials per principal per hour.- Interview question: "An auditor asks you to prove who deleted a production topic last Tuesday. What can you show with a default Apache Kafka installation, and what would you change for next time?" Hint: separate what the metadata log can tell you from what only an authorizer-level log can, and think about log levels and retention.
<a id="12-operations-and-administration"></a>