6. Data Serialization and Schemas
Kafka brokers store and move only bytes. They never look inside a record, so every producer and consumer must agree on how objects become bytes and back, and that agreement has to survive years of independent deployments. This chapter covers the serialization formats used with Kafka, from plain strings to Avro and Protocol Buffers, and the schema management layer (Schema Registry, subjects, compatibility rules, and data contracts) that lets those formats change safely while hundreds of services read the same topics.
<a id="61-serialization-formats"></a>
6.1 Serialization Formats
This section compares the formats you can put inside a Kafka record: the built-in primitive serializers, JSON, the two dominant binary schema formats (Avro and Protobuf), and your own serializers when nothing standard fits.
String and Byte Serializers
Theory
A Kafka record is a key and a value, and to the broker both are just byte arrays (Section 2.1). Something has to turn the objects in your program into those bytes. That something is a serializer, configured on the producer as key.serializer and value.serializer (Section 4.1). The consumer does the reverse with a matching deserializer (Section 5.1).
Why does Kafka not choose a format for you? Because the broker never needs to understand the data. Leaving the format to clients keeps the broker fast (it copies bytes without parsing them) and lets each team pick the format that fits. The cost is that the format becomes an unwritten agreement between producers and consumers, and nothing on the broker enforces it.
Analogy: Kafka is a postal service for sealed envelopes. It weighs and routes them but never opens them. Whether the letter inside is in English or Japanese is between sender and recipient.
In Java, every serializer implements one small interface:
public interface Serializer<T> extends Closeable {
default void configure(Map<String, ?> configs, boolean isKey) {} // called once with the producer config
byte[] serialize(String topic, T data); // object -> bytes
default byte[] serialize(String topic, Headers headers, T data) { // header-aware variant
return serialize(topic, data);
}
default void close() {}
}
Built-in serializers
The org.apache.kafka.common.serialization package ships serializers for primitive types:
| Serializer | Input type | Bytes produced |
|---|---|---|
StringSerializer | String | UTF-8 text by default (key.serializer.encoding, value.serializer.encoding, or serializer.encoding change it) |
ByteArraySerializer | byte[] | The array, unchanged |
ByteBufferSerializer, BytesSerializer | ByteBuffer, Bytes | The buffer contents |
ShortSerializer, IntegerSerializer, LongSerializer | Short, Integer, Long | 2, 4, 8 bytes, big-endian |
FloatSerializer, DoubleSerializer | Float, Double | 4, 8 bytes, IEEE 754 |
UUIDSerializer | UUID | The 36-character string form, as UTF-8 |
VoidSerializer | Void | Always null (for records without a key) |
ListSerializer | List<T> | Elements encoded with an inner serializer (Kafka 3.0+) |
All of them turn null into null, so tombstones (Section 7.2) pass through.
Why the key format matters more than it looks
The default partitioner hashes the serialized key bytes with murmur2 (Section 4.2). The same logical key encoded two ways produces two different byte arrays, and therefore usually two different partitions:
key = 42 with IntegerSerializer -> 00 00 00 2A -> murmur2 -> partition 5
key = 42 with StringSerializer -> 34 32 -> murmur2 -> partition 1 (example values)
If one producer sends customer 42 as an integer and another as the string "42", events for the same customer land in different partitions, and per-key ordering is lost without any error being raised. This is common in multi-language systems, where a Python producer writes str(customer_id) and a Java producer uses IntegerSerializer.
Example: primitive serializers in Java and Python
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, LongSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
try (Producer<Long, byte[]> producer = new KafkaProducer<>(props)) {
byte[] thumbnail = Files.readAllBytes(Path.of("thumb.jpg"));
producer.send(new ProducerRecord<>("thumbnails", 1001L, thumbnail)); // key: 8 bytes, value: raw JPEG
}
from confluent_kafka import Producer
from confluent_kafka.serialization import IntegerSerializer, StringSerializer
producer = Producer({"bootstrap.servers": "localhost:9092"})
key_ser = IntegerSerializer() # 4-byte big-endian, same bytes as Java's IntegerSerializer
value_ser = StringSerializer("utf_8")
# confluent-kafka itself accepts only str/bytes; the serializer objects make the format explicit
producer.produce("customers", key=key_ser(42), value=value_ser("Ada Lovelace"))
producer.flush()
Primitive serializers are the right choice for keys (IDs, UUIDs) and for opaque payloads such as images or already-encoded blobs. For structured values, they leave you to invent a format, which is what the rest of this section is about.
Key Takeaways
- Kafka stores bytes. Serializers on the producer and deserializers on the consumer define the format, and the broker never checks it.
- Built-in serializers cover strings, raw bytes, numbers, and UUIDs. Numbers are fixed-width big-endian.
- Partitioning hashes the serialized key bytes, so every producer of a topic must serialize keys identically, across languages too.
- Use primitive serializers for keys and opaque payloads, and a structured format for record values.
Practice
- (Beginner) Produce the key
42once withIntegerSerializerand once withStringSerializerto a topic with 6 partitions. Usekafka-console-consumer.sh --property print.partition=trueto confirm which partition each record landed in.- (Intermediate) A Python service and a Java service both produce order events keyed by a numeric customer ID. Write the Python producer code that guarantees its keys land in the same partitions as the Java producer that uses
LongSerializer.- Interview question: "Why doesn't Kafka validate the format of the data it stores?" Hint: think about what the broker does with records on the write and read paths, and what parsing would cost in throughput and flexibility.
JSON
Theory
JSON is usually the first structured format teams reach for, and for good reasons: every language can read and write it, it is human-readable in a console consumer, and there is nothing to set up. A JSON value serializer is a few lines of code.
The problems appear as the system grows:
| Problem | What goes wrong |
|---|---|
| No enforced structure | A producer renames customerId to customer_id; consumers get null and fail days later, or silently compute wrong results |
| Field names in every record | "customerId": is repeated in millions of records, so payloads are 2 to 5 times larger than binary formats |
| Weak types | All numbers are "number". Values above 2^53 lose precision in JavaScript, and there is no date, decimal, or binary type |
| No evolution rules | Nothing tells you whether a change is safe for existing consumers |
Analogy: plain JSON is a handwritten note. Anyone can read it, but nobody checks that the writer used the agreed form, and the next writer may use a different one.
Example: plain JSON with Jackson (Java)
public record Order(String orderId, long customerId, long amountCents, String currency, String status) {}
public class JsonSerializer<T> implements Serializer<T> {
private final ObjectMapper mapper = new ObjectMapper(); // thread-safe once configured
@Override
public byte[] serialize(String topic, T data) {
if (data == null) return null; // keep tombstones as null
try {
return mapper.writeValueAsBytes(data);
} catch (JsonProcessingException e) {
throw new SerializationException("cannot serialize to JSON for " + topic, e);
}
}
}
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName());
Example: plain JSON in Python
import json
from confluent_kafka import Producer
producer = Producer({"bootstrap.servers": "localhost:9092"})
order = {"orderId": "o-1001", "customerId": 42, "amountCents": 1999, "currency": "EUR", "status": "PAID"}
producer.produce(
"orders",
key=order["orderId"],
value=json.dumps(order, separators=(",", ":")).encode("utf-8"), # compact JSON, 88 bytes
)
producer.flush()
JSON Schema: JSON with a contract
You can keep JSON on the wire and still get enforcement by describing it with JSON Schema and using a schema-aware serializer (KafkaJsonSchemaSerializer in Java, JSONSerializer in Python) with a schema registry (Section 6.2). The serializer validates each object against the schema before sending, and the registry checks that schema changes are compatible.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Order",
"type": "object",
"properties": {
"orderId": { "type": "string" },
"customerId": { "type": "integer" },
"amountCents": { "type": "integer", "minimum": 0 },
"currency": { "type": "string", "pattern": "^[A-Z]{3}$" },
"status": { "enum": ["PENDING", "PAID", "SHIPPED"] }
},
"required": ["orderId", "customerId", "amountCents", "currency", "status"],
"additionalProperties": false
}
The size problem remains, because field names are still sent in every record. Compatibility rules are also harder to reason about than for Avro or Protobuf, especially around additionalProperties: whether an "open" or "closed" content model is compatible depends on the direction of the change.
Size comparison for the same order
| Format | Bytes for the example order | Notes |
|---|---|---|
| JSON (compact) | 88 | Field names included |
| JSON Schema + registry framing | 93 | Same JSON plus a 5-byte schema ID prefix |
| Avro + registry framing | 20 | 15 bytes of data (topic "Apache Avro") |
| Protobuf + registry framing | 26 | 20 bytes of data (topic "Protocol Buffers") |
Compression (Section 4.4) narrows the gap for batches of similar records, because repeated field names compress well, but it costs CPU on both sides and does not fix the lack of types.
Key Takeaways
- JSON is universal and readable, which makes it good for prototypes, low-volume topics, and data meant for humans.
- Plain JSON has no enforced structure, weak types, and large payloads. Producers can break consumers without anyone noticing.
- JSON Schema with a schema registry adds validation and compatibility checks while keeping JSON on the wire.
- For high-volume or long-lived topics, a binary schema format is usually a better default.
Practice
- (Beginner) Produce 10,000 JSON order events with and without
compression.type=lz4. Compare the topic size on disk usingkafka-log-dirs.sh --describe.- (Intermediate) Write a JSON Schema for a
Paymentevent that has an optionalrefundOffield and a required ISO 4217 currency. Explain what happens to existing consumers if you later setadditionalPropertiestofalse.- Interview question: "Your team uses plain JSON on Kafka and a producer change broke three downstream services. What would you change, and how would you roll it out without stopping production?" Hint: think about schema enforcement at write time, compatibility checks before deployment, and how consumers can be migrated gradually.
Apache Avro
Theory
Avro, created in the Hadoop ecosystem, solves JSON's problems with one idea: the schema is required to read the data, so the data does not need to describe itself. The schema is written in JSON, but records are encoded as compact binary with no field names, no tags, and no type markers. Fields appear one after another in the order the schema lists them.
That design has two consequences:
- Records are very small, because only values are encoded.
- A reader cannot decode a record without the exact schema it was written with (the writer schema). Avro solves this with schema resolution: the reader supplies its own reader schema, and Avro maps writer fields to reader fields by name, filling in defaults and ignoring extra fields. This is what makes Avro well suited to evolving data.
Analogy: an Avro record is a form with the labels removed, leaving only the answers in the boxes. To read it, you need a copy of the blank form it was filled in on. If your copy of the form is a newer edition, Avro knows how to match the old answers to the new boxes.
A schema
{
"type": "record",
"name": "Order",
"namespace": "com.example.orders",
"fields": [
{ "name": "orderId", "type": "string" },
{ "name": "customerId", "type": "long" },
{ "name": "amountCents", "type": "long" },
{ "name": "currency", "type": "string" },
{ "name": "status",
"type": { "type": "enum", "name": "OrderStatus",
"symbols": ["PENDING", "PAID", "SHIPPED"], "default": "PENDING" } },
{ "name": "couponCode", "type": ["null", "string"], "default": null },
{ "name": "createdAt",
"type": { "type": "long", "logicalType": "timestamp-millis" }, "default": 0 }
]
}
| Avro type | Meaning |
|---|---|
null, boolean, int, long, float, double, bytes, string | Primitive types |
record, enum, array, map, fixed | Complex types |
Union, e.g. ["null", "string"] | One of several types. This is how optional fields are written; the default must match the first branch |
Logical types (decimal, uuid, date, timestamp-millis, ...) | Annotations on primitive types that give them meaning |
The binary encoding
Integers and longs are written as zigzag variable-length integers, so small numbers take one byte. Strings are a length followed by UTF-8 bytes. Enums are the symbol's index. Here is the order from the JSON topic, with only the first five fields for brevity:
field value bytes
orderId "o-1001" 0C 6F 2D 31 30 30 31 length 6 (zigzag 12 = 0x0C), then UTF-8
customerId 42 54 zigzag(42) = 84 = 0x54
amountCents 1999 9E 1F zigzag(1999) = 3998, as a 2-byte varint
currency "EUR" 06 45 55 52 length 3, then UTF-8
status PAID 02 enum index 1, zigzag = 2
-------------------
15 bytes total (JSON: 88 bytes)
There is nothing in these bytes that says "customerId". Without the schema, 54 could be anything.
Generic and specific records
| Style | How you use it | When to use |
|---|---|---|
GenericRecord | Access fields by name at runtime: record.get("customerId") | Tools, routers, and pipelines that handle many schemas |
| Specific record | Classes generated from .avsc by the Avro Maven or Gradle plugin: order.getCustomerId() | Application code with fixed types and compile-time checking |
Example: Avro producer and consumer in Java
// Producer: Order is a class generated from Order.avsc
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
p.put("schema.registry.url", "http://localhost:8081");
p.put("auto.register.schemas", false); // schemas are registered by CI, not by applications
p.put("use.latest.version", true); // serialize with the latest registered version
try (Producer<String, Order> producer = new KafkaProducer<>(p)) {
Order order = Order.newBuilder()
.setOrderId("o-1001").setCustomerId(42L).setAmountCents(1999L)
.setCurrency("EUR").setStatus(OrderStatus.PAID)
.setCreatedAt(Instant.now()) // timestamp-millis maps to Instant
.build(); // couponCode defaults to null
producer.send(new ProducerRecord<>("orders", order.getOrderId(), order));
}
// Consumer
Properties c = new Properties();
c.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
c.put(ConsumerConfig.GROUP_ID_CONFIG, "billing");
c.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
c.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
c.put("schema.registry.url", "http://localhost:8081");
c.put("specific.avro.reader", true); // return Order instead of GenericRecord
Example: Avro in Python
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import MessageField, SerializationContext
schema_str = open("Order.avsc").read()
registry = SchemaRegistryClient({"url": "http://localhost:8081"})
avro_ser = AvroSerializer(registry, schema_str) # dicts are serialized directly
producer = Producer({"bootstrap.servers": "localhost:9092"})
order = {"orderId": "o-1001", "customerId": 42, "amountCents": 1999,
"currency": "EUR", "status": "PAID", "couponCode": None, "createdAt": 1761210942120}
producer.produce(
"orders",
key=order["orderId"],
value=avro_ser(order, SerializationContext("orders", MessageField.VALUE)), # registers or looks up the schema
)
producer.flush()
The Confluent serializers do not write the schema into each record. They write a 5-byte prefix holding the schema's registry ID (topic "Schema Registry Architecture"), and the deserializer fetches the writer schema by that ID.
Key Takeaways
- Avro schemas are JSON documents. Records are compact binary containing only values, in schema field order.
- Reading always needs the writer schema. Schema resolution maps it onto the reader's schema by field name, using defaults and ignoring unknown fields.
- Optional fields are unions with
nullfirst and"default": null. - In Kafka, Avro is almost always used with a schema registry, which stores the writer schemas and puts only a schema ID in each record.
Practice
- (Beginner) Write an Avro schema for a
UserSignedUpevent with a requireduserId, an optionalreferrer, and asignedUpAttimestamp. Generate Java classes or use a Python dict, and produce one record.- (Intermediate) Encode
{"orderId": "a", "customerId": -1, "amountCents": 64, "currency": "", "status": "SHIPPED"}by hand using the rules above, then check your answer withfastavro.schemaless_writerin Python.- Interview question: "Avro records contain no field names. How can a consumer built against a newer schema read older records?" Hint: think about writer schema versus reader schema, how fields are matched, and the role of defaults.
Protocol Buffers
Theory
Protocol Buffers (Protobuf), created at Google and used by gRPC, solves the same problem as Avro with a different trade-off. Instead of relying on field order and a writer schema, each encoded field carries a small tag: the field's number plus a wire type. Field names are still not on the wire, but a reader can skip fields it does not know about and can decode a record with any compatible version of the schema.
You describe messages in a .proto file and generate code for each language with protoc (or Buf). The field numbers are the contract: names can change freely, numbers must never be reused for something else.
Analogy: if Avro is a form with the labels removed, Protobuf is a form where each answer carries a small box number. A reader with an older form still knows which answers go in which boxes and simply sets aside answers whose box numbers it does not recognize.
A schema
syntax = "proto3";
package com.example.orders;
import "google/protobuf/timestamp.proto";
option java_multiple_files = true;
enum OrderStatus {
ORDER_STATUS_UNSPECIFIED = 0; // proto3 enums must start at 0; reserve 0 for "not set"
ORDER_STATUS_PENDING = 1;
ORDER_STATUS_PAID = 2;
ORDER_STATUS_SHIPPED = 3;
}
message Order {
string order_id = 1;
int64 customer_id = 2;
int64 amount_cents = 3;
string currency = 4;
OrderStatus status = 5;
optional string coupon_code = 6; // "optional" tracks presence (set vs default)
google.protobuf.Timestamp created_at = 7;
reserved 8; // number of a deleted field: never reuse it
reserved "legacy_channel";
}
The binary encoding
Each field is written as tag = (field_number << 3) | wire_type, followed by the value. In proto3, fields holding their default value (0, empty string, first enum value) are not written at all.
field tag bytes explanation
order_id = 1 0A 0A 06 6F 2D 31 30 30 31 (1<<3)|2 = 0x0A, length-delimited, length 6
customer_id 10 10 2A (2<<3)|0 = 0x10, varint 42
amount_cents 18 18 CF 0F (3<<3)|0 = 0x18, varint 1999
currency 22 22 03 45 55 52 (4<<3)|2 = 0x22, length 3
status 28 28 02 (5<<3)|0 = 0x28, enum value 2 (PAID)
----------------------
20 bytes total
Protobuf is a few bytes larger than Avro because of the tags, but a reader without the exact writer schema can still parse it.
Example: Protobuf with Schema Registry (Java)
// Producer
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaProtobufSerializer.class);
p.put("schema.registry.url", "http://localhost:8081");
// Imported schemas such as google/protobuf/timestamp.proto are registered as references
Order order = Order.newBuilder()
.setOrderId("o-1001").setCustomerId(42).setAmountCents(1999)
.setCurrency("EUR").setStatus(OrderStatus.ORDER_STATUS_PAID)
.build();
producer.send(new ProducerRecord<>("orders", order.getOrderId(), order));
// Consumer: without a specific type, values come back as DynamicMessage
c.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaProtobufDeserializer.class);
c.put("schema.registry.url", "http://localhost:8081");
c.put("specific.protobuf.value.type", Order.class.getName());
Example: Protobuf in Python
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.protobuf import ProtobufDeserializer, ProtobufSerializer
from confluent_kafka.serialization import MessageField, SerializationContext
import order_pb2 # generated with: protoc --python_out=. order.proto
registry = SchemaRegistryClient({"url": "http://localhost:8081"})
proto_ser = ProtobufSerializer(order_pb2.Order, registry)
proto_de = ProtobufDeserializer(order_pb2.Order)
ctx = SerializationContext("orders", MessageField.VALUE)
order = order_pb2.Order(order_id="o-1001", customer_id=42, amount_cents=1999,
currency="EUR", status=order_pb2.ORDER_STATUS_PAID)
data = proto_ser(order, ctx) # 5-byte prefix + message index + Protobuf bytes
assert proto_de(data, ctx) == order
Because one .proto file can define several messages, the Confluent wire format for Protobuf adds a message index after the schema ID, identifying which message in the file was encoded. For the first message it is a single 0x00 byte.
Avro, Protobuf, and JSON Schema compared
| Aspect | Avro | Protobuf | JSON Schema |
|---|---|---|---|
| Wire format | Binary, no tags, schema order | Binary, tagged fields | JSON text |
| Size | Smallest | Small | Large |
| Reading without writer schema | Not possible | Possible (unknown fields skipped) | Possible |
| Code generation | Optional (generic records work well) | Expected | Optional |
| Field identity | Name (aliases for renames) | Number | Name |
| Optional fields | Union with null and a default | optional keyword, or wrapper types | Leave out of required |
| Typical home | Data platforms, Kafka Connect, Hadoop and lakehouse tools | Microservices already using gRPC | Teams already using JSON, web clients |
| Main pitfall | Forgetting defaults breaks compatibility | Reusing field numbers corrupts data | Open/closed content model compatibility |
Key Takeaways
- Protobuf tags each field with its number and wire type, so readers can skip unknown fields and decode without the exact writer schema.
- Field numbers are the contract. Renaming a field is safe; changing or reusing its number is not. Use
reservedfor deleted fields. - In proto3, default values are not encoded. Use
optionalwhen you need to distinguish "not set" from zero or empty. - Choose Avro for the smallest records and data-platform tooling, Protobuf when your services already use it, and JSON Schema when readable JSON on the wire matters.
Practice
- (Beginner) Write a
.protofile for aShipmentmessage with an enumcarrier, a repeatedparcel_idsfield, and an optionaldelivered_attimestamp. Generate code in Java or Python and produce one record with the Confluent Protobuf serializer.- (Intermediate) Decode the bytes
08 96 01 12 03 61 62 63by hand (field numbers, wire types, values). Then explain what an old reader does when it meets field number 9, which it has never seen.- Interview question: "Why is it dangerous to delete a Protobuf field and later reuse its number for a new field?" Hint: think about old records still in the topic and old consumers still running, and what each would do with the bytes under that tag.
Custom Serializers
Theory
Sometimes no standard format fits: a legacy binary protocol, a domain type you want written as a compact fixed layout, a payload that must be encrypted or signed, or a value that wraps another format with an envelope. Kafka lets you plug in any class implementing Serializer<T> and Deserializer<T>.
Before writing one, ask whether a standard format would do. A custom format means you own the cross-language implementations, the documentation, and the evolution rules, and every consumer team inherits that burden. Custom serializers work best as thin wrappers around a standard format (adding encryption, compression, or headers) rather than as new formats.
Analogy: a custom serializer is a private shorthand. It can be very efficient for the person who invented it, but everyone else who needs to read it has to learn it, and it is hard to change once others depend on it.
What a good serializer must handle
| Concern | Rule |
|---|---|
null | Return null for null input so tombstones work (and accept null in the deserializer) |
| Thread safety | One serializer instance per producer is called from every thread that calls send(). Keep it stateless or use thread-safe objects |
| Errors | Throw SerializationException. On the producer it is thrown from send() and is never retried |
| Versioning | Put a version byte or header in the payload so the format can change later |
| Configuration | Read settings in configure(configs, isKey), which receives the full client config |
| Language parity | Every consumer language needs an identical implementation, with tests against shared sample bytes |
Example: a versioned binary serializer that sets a header
public record Temperature(String sensorId, long timestampMs, double celsius) {}
public class TemperatureSerializer implements Serializer<Temperature> {
private static final byte VERSION = 1;
@Override
public byte[] serialize(String topic, Temperature t) {
return serialize(topic, null, t);
}
@Override
public byte[] serialize(String topic, Headers headers, Temperature t) {
if (t == null) return null;
byte[] id = t.sensorId().getBytes(StandardCharsets.UTF_8);
if (id.length > 255) throw new SerializationException("sensorId too long: " + id.length);
ByteBuffer buf = ByteBuffer.allocate(1 + 1 + id.length + 8 + 8);
buf.put(VERSION); // format version, checked by the deserializer
buf.put((byte) id.length); // 1-byte length prefix
buf.put(id);
buf.putLong(t.timestampMs()); // big-endian, matching the built-in serializers
buf.putDouble(t.celsius());
if (headers != null) { // headers are still writable during serialization
headers.add("content-type", "application/x-temperature-v1".getBytes(StandardCharsets.UTF_8));
}
return buf.array();
}
}
public class TemperatureDeserializer implements Deserializer<Temperature> {
@Override
public Temperature deserialize(String topic, byte[] data) {
if (data == null) return null;
ByteBuffer buf = ByteBuffer.wrap(data);
byte version = buf.get();
if (version != 1) throw new SerializationException("unsupported version " + version);
byte[] id = new byte[Byte.toUnsignedInt(buf.get())];
buf.get(id);
return new Temperature(new String(id, StandardCharsets.UTF_8), buf.getLong(), buf.getDouble());
}
}
For Kafka Streams, which needs both directions together, wrap the pair in a Serde:
Serde<Temperature> temperatureSerde =
Serdes.serdeFrom(new TemperatureSerializer(), new TemperatureDeserializer());
Example: the same format in Python
import struct
from confluent_kafka.serialization import Deserializer, SerializationError, Serializer
class TemperatureSerializer(Serializer):
def __call__(self, obj, ctx=None):
if obj is None:
return None
sensor_id = obj["sensorId"].encode("utf-8")
# > big-endian | B version | B length | Ns id | q int64 | d float64
return struct.pack(f">BB{len(sensor_id)}sqd", 1, len(sensor_id), sensor_id,
obj["timestampMs"], obj["celsius"])
class TemperatureDeserializer(Deserializer):
def __call__(self, value, ctx=None):
if value is None:
return None
version, length = struct.unpack_from(">BB", value)
if version != 1:
raise SerializationError(f"unsupported version {version}")
sensor_id, ts, celsius = struct.unpack_from(f">{length}sqd", value, 2)
return {"sensorId": sensor_id.decode("utf-8"), "timestampMs": ts, "celsius": celsius}
Wrapping instead of inventing
A common, healthy use of custom serializers is delegation: an EncryptingSerializer<T> that calls an inner KafkaAvroSerializer, encrypts the result, and adds a header naming the key ID. The data format stays standard, and the wrapper adds one concern. Field-level encryption in a schema registry's data contracts (topic "Data Contracts") is an alternative that keeps the payload readable to authorized consumers without custom code.
Key Takeaways
- Implement
Serializer<T>andDeserializer<T>(and aSerde<T>for Kafka Streams) when no standard format fits. - Handle
null, be thread-safe, throwSerializationException, and include a version marker from day one. - The header-aware
serializemethod can attach metadata such as content type or encryption key IDs. - Prefer thin wrappers around standard formats over new formats. Every custom format must be reimplemented and maintained in every consumer language.
Practice
- (Beginner) Implement and unit-test a
MoneySerializer/MoneyDeserializerpair for aMoney(long cents, String currency)type, including thenullcase.- (Intermediate) Extend
TemperatureSerializerto version 2 by adding ahumidityfield. Make the deserializer read both versions, and write a test that decodes bytes written by version 1.- Interview question: "A colleague proposes a custom binary format for a new high-volume topic because it is 20% smaller than Avro. How would you evaluate the proposal?" Hint: compare the size saving with compression, and consider tooling (Connect, ksqlDB, console tools), other consumer languages, and how the format will evolve.
<a id="62-schema-management"></a>
6.2 Schema Management
A format alone does not stop a producer from shipping a breaking change. This section covers the schema registry that stores and versions schemas, how schemas are grouped into subjects, the compatibility rules the registry enforces, how to evolve schemas safely, and data contracts that extend schemas with ownership and rules.
Schema Registry Architecture
Theory
Avro needs the writer schema to read a record, and every schema-based format needs some agreement on which schema is current. Two naive solutions both fail:
- Embedding the schema in every record works, but a 1 KB schema attached to a 50-byte record makes the format pointless.
- Sharing schema files by copying them between repositories works until two teams have different copies, and nothing stops a breaking change from reaching production.
A schema registry is a separate service that stores every version of every schema, gives each one a numeric ID, and checks new versions against compatibility rules. Producers put only the ID in each record. Consumers look the ID up once and cache the schema.
Analogy: the registry is a library catalog. Instead of mailing a whole book with every letter, you write "see catalog number 17". The reader looks it up once, keeps a copy, and the librarian refuses to file a new edition that contradicts the old one.
The registry is not part of Apache Kafka. Common implementations:
| Implementation | License | Notes |
|---|---|---|
| Confluent Schema Registry | Confluent Community License | The reference implementation; stores schemas in a Kafka topic |
| Apicurio Registry | Apache 2.0 | Red Hat; supports Confluent-compatible APIs and several storage backends |
| Karapace | Apache 2.0 | Aiven; drop-in replacement for the Confluent API |
| AWS Glue Schema Registry | Managed service | Its own serializers and wire format, not Confluent-compatible |
The rest of this chapter uses the Confluent API, which most tools and alternatives implement.
How it works
+--------------------------------------------+
| Schema Registry cluster (stateless) |
| +---------+ +---------+ +---------+ |
REST: register, | | primary | | replica | | replica | | every instance holds all
look up, check -->| | (writes)| | | | | | schemas in memory and
| +----+----+ +----+----+ +----+----+ | serves reads
+--------|-------------|-------------|--------+
| write | read | read
v v v
+---------------------------------------------+
| Kafka topic _schemas (1 partition, compacted,|
| replication factor 3): the source of truth |
+---------------------------------------------+
- Schemas are stored as records in the compacted topic
_schemas(kafkastore.topic). Every registry instance consumes it from the beginning at startup and keeps the full state in memory. - One instance is the primary, elected through Kafka's group membership protocol among instances with
leader.eligibility=true. Only the primary writes, which keeps ID assignment consistent. Replicas forward writes to it. - Schema IDs are global across the registry. Registering a schema that already exists returns its existing ID.
The produce and consume path
Producer (KafkaAvroSerializer) Schema Registry
1. serialize(order) |
2. schema in local cache? no ---- lookup/register --> | returns id = 17
3. cache schema -> 17 |
4. bytes = 0x00 | 00 00 00 11 | avro payload |
5. send to Kafka |
|
Consumer (KafkaAvroDeserializer) |
6. read byte 0 = 0x00 (magic), bytes 1-4 = 17 |
7. schema 17 in cache? no ------ GET /schemas/ids/17 >|
8. decode payload with writer schema 17 (+ reader schema if specific)
After the first record, both sides use their cache, so the registry is not on the hot path. If it goes down, producers and consumers keep working with schemas they already know. Only new schemas (a new producer version, or a consumer meeting an unknown ID) fail.
The wire format
| Bytes | Content |
|---|---|
| 0 | Magic byte, always 0x00 |
| 1 to 4 | Schema ID, 4-byte big-endian integer |
| 5 onward | Avro or JSON payload. For Protobuf: message indexes, then the Protobuf payload |
Recent Confluent clients (Confluent Platform 8.0 and later) can alternatively put a schema identifier in a record header and leave the payload unprefixed. Either way, a consumer that uses a plain ByteArrayDeserializer sees these extra bytes, which is why a console consumer shows a few unreadable characters before the data.
Example: running a registry and using its REST API
# docker-compose.yml (added next to the Kafka broker from Section 3.1)
schema-registry:
image: confluentinc/cp-schema-registry:8.0.0
depends_on: [kafka]
ports: ["8081:8081"]
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092
# Register a schema under subject orders-value; the response is the global ID
curl -s -X POST http://localhost:8081/subjects/orders-value/versions \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' \
-d "{\"schemaType\": \"AVRO\", \"schema\": $(jq -Rs . < Order.avsc)}"
# {"id":17}
# Fetch a schema by ID (what deserializers do)
curl -s http://localhost:8081/schemas/ids/17
# List subjects and the versions of one subject
curl -s http://localhost:8081/subjects
curl -s http://localhost:8081/subjects/orders-value/versions
# [1,2,3]
Production practices
| Practice | Why |
|---|---|
auto.register.schemas=false in applications | Stops any service from registering a schema at startup; schemas go through review and CI |
| Two or more registry instances behind a load balancer | Reads stay available during restarts and primary changes |
Protect _schemas like any critical topic | Losing it loses every schema, so old records become unreadable |
| Authentication and authorization on the REST API | Only CI pipelines should be able to register or delete schemas |
Key Takeaways
- A schema registry stores versioned schemas, assigns global IDs, and enforces compatibility. It is a separate service, not part of Apache Kafka.
- Records carry a 5-byte prefix (magic byte plus schema ID) instead of the schema itself.
- Confluent Schema Registry stores state in the compacted
_schemastopic. A single primary handles writes, and every instance serves reads from memory. - Clients cache schemas by ID, so a registry outage only affects schemas a client has not seen yet.
Practice
- (Beginner) Start a registry with Docker Compose, register an Avro schema with
curl, produce one record with the Avro serializer, and read it withkafka-console-consumer.shand--property print.value=true. Identify the magic byte and schema ID in the raw output (for example, by piping toxxd).- (Intermediate) Stop the registry while an Avro producer and consumer are running. Which operations keep working and which fail? Then restart the producer with a new schema version and explain the result.
- Interview question: "Why does Confluent Schema Registry store its data in a Kafka topic instead of a database?" Hint: think about durability, ordering of writes, how instances rebuild state, and what infrastructure a Kafka user already runs.
Subjects and Versions
Theory
A registry needs to know which schemas are successive versions of the same thing, so that it can check a new version against the old ones. That grouping is the subject: a named scope under which schema versions evolve. Each subject has its own version numbers (1, 2, 3, ...) and its own compatibility setting. Schema IDs, in contrast, are global across all subjects.
Analogy: a subject is a document's edit history. Version numbers count the edits to that one document. The schema ID is like a content fingerprint: if two documents end up with identical text, they share the same ID.
subject: orders-value global schema IDs
version 1 ------------------------------> id 17
version 2 (added couponCode) ----------> id 23
version 3 (added createdAt) ----------> id 31
subject: orders-key
version 1 ("string") ------------------> id 2
subject: invoices-value
version 1 (same Customer schema as -> id 23 reused if the schema text is identical
another subject)
Subject naming strategies
The serializer derives the subject name from the topic and the record type, using key.subject.name.strategy and value.subject.name.strategy:
| Strategy | Subject name for topic orders, record com.example.OrderCreated | Use when |
|---|---|---|
TopicNameStrategy (default) | orders-value | One record type per topic. Compatibility is checked per topic |
RecordNameStrategy | com.example.OrderCreated | The same type appears in many topics and must evolve consistently everywhere |
TopicRecordNameStrategy | orders-com.example.OrderCreated | Several types share one topic, each evolving independently |
Several event types in one topic
Ordering in Kafka is per partition. If OrderCreated, OrderPaid, and OrderShipped for the same order must be read in order, they belong in one topic with the same key. Two ways to model this:
| Approach | How | Trade-off |
|---|---|---|
TopicRecordNameStrategy | Each type gets its own subject | Simple, but the topic accepts any type, and nothing lists which types are allowed |
TopicNameStrategy with a union schema and schema references | orders-value is a union of references to each event type | The allowed types are explicit and checked, but needs auto.register.schemas=false and use.latest.version=true on producers |
A schema reference lets one schema use a type registered under another subject:
# orders-value is a union of three types that are registered separately
curl -s -X POST http://localhost:8081/subjects/orders-value/versions \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' \
-d '{
"schemaType": "AVRO",
"schema": "[\"com.example.OrderCreated\", \"com.example.OrderPaid\", \"com.example.OrderShipped\"]",
"references": [
{"name": "com.example.OrderCreated", "subject": "com.example.OrderCreated", "version": 1},
{"name": "com.example.OrderPaid", "subject": "com.example.OrderPaid", "version": 1},
{"name": "com.example.OrderShipped", "subject": "com.example.OrderShipped", "version": 1}
]
}'
// Producer for the union subject
props.put("auto.register.schemas", false); // never register the concrete type as orders-value
props.put("use.latest.version", true); // serialize with the union, the latest version of orders-value
For Protobuf, the equivalent is a wrapper message with a oneof, and references are created automatically for imported .proto files.
Deleting versions
# Soft delete: hidden from listings and compatibility checks, still readable by ID
curl -s -X DELETE http://localhost:8081/subjects/orders-value/versions/3
# Hard delete (only after a soft delete): gone permanently
curl -s -X DELETE "http://localhost:8081/subjects/orders-value/versions/3?permanent=true"
Records written with a hard-deleted schema can no longer be deserialized. Use hard deletes only for schemas that were never used in production, and only after checking that no topic still contains records written with them.
Key Takeaways
- A subject is the scope in which schema versions evolve and compatibility is checked. Versions are numbered per subject, and IDs are global.
TopicNameStrategy(one type per topic,<topic>-keyand<topic>-value) is the default. Record-based strategies support several types per topic or shared types across topics.- When several event types must stay ordered together, put them in one topic, using either
TopicRecordNameStrategyor a union schema with references. - Soft-deleted schemas still resolve by ID. Hard deletes can make existing records unreadable.
Practice
- (Beginner) Register the same Avro schema under two subjects and confirm that both return the same ID. Then register a modified version in one subject and list versions and IDs for both.
- (Intermediate) Model an
orderstopic carryingOrderCreated,OrderPaid, andOrderShippedusing a union with schema references. Produce one of each, and consume them with a specific reader that dispatches on the record type.- Interview question: "When would you choose
RecordNameStrategyover the defaultTopicNameStrategy, and what risk does it introduce?" Hint: think about where compatibility is checked, how many topics share a subject, and what a single topic is then allowed to contain.
Compatibility Modes
Theory
Producers and consumers are deployed at different times, by different teams. During a rollout, and for as long as old records stay in a topic, data written with one schema version is read with another. Compatibility describes which direction of that mismatch is safe, and the registry rejects any new schema version that violates the subject's configured mode.
Two questions define the modes:
- Backward compatible: can consumers using the new schema read data written with the old one?
- Forward compatible: can consumers using the old schema read data written with the new one?
Analogy: a new version of a word processor that opens old files is backward compatible. An old version that can open files saved by the new version (perhaps ignoring new features) is forward compatible.
BACKWARD: old data ---> new reader upgrade consumers first
FORWARD: new data ---> old reader upgrade producers first
FULL: both directions upgrade in any order
The modes
| Mode | New schema is checked against | Guarantee | Upgrade order |
|---|---|---|---|
BACKWARD (default) | The latest version | New readers read data from the previous version | Consumers first |
BACKWARD_TRANSITIVE | All versions | New readers read all historical data | Consumers first |
FORWARD | The latest version | Previous readers read new data | Producers first |
FORWARD_TRANSITIVE | All versions | All older readers read new data | Producers first |
FULL | The latest version | Both directions with the previous version | Any order |
FULL_TRANSITIVE | All versions | Both directions with every version | Any order |
NONE | Nothing | No check | Coordinate manually |
Why do the transitive modes matter? Non-transitive modes check only adjacent versions. Version 3 can be compatible with version 2, which was compatible with version 1, while version 3 cannot read version 1 records. If a topic keeps data for a long time, or is replayed from the beginning, use a _TRANSITIVE mode.
Which changes are allowed (Avro)
| Change | Backward | Forward | Full |
|---|---|---|---|
| Add a field with a default | Yes | Yes | Yes |
| Add a field without a default | No: new readers have no value for old records | Yes | No |
| Remove a field with a default | Yes | Yes | Yes |
| Remove a field without a default | Yes | No: old readers require it | No |
| Rename a field | No (Yes with an aliases entry in the reader schema) | No | No |
Widen int to long | Yes | No | No |
| Add an enum symbol | Yes | Only if the enum has a default | Only with a default |
For Protobuf, adding and removing fields are usually compatible in both directions, because unknown fields are skipped and missing fields read as defaults. Changing a field's number or switching it to an incompatible type breaks compatibility. Confluent recommends BACKWARD_TRANSITIVE for Protobuf, because adding new message types to a schema is not forward compatible.
Example: setting and testing compatibility
# Global default for the registry
curl -s -X PUT http://localhost:8081/config \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' -d '{"compatibility": "BACKWARD"}'
# Stricter mode for one long-lived subject
curl -s -X PUT http://localhost:8081/config/orders-value \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' -d '{"compatibility": "FULL_TRANSITIVE"}'
# Test a candidate schema without registering it
curl -s -X POST "http://localhost:8081/compatibility/subjects/orders-value/versions/latest?verbose=true" \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' \
-d "{\"schema\": $(jq -Rs . < Order-v4.avsc)}"
# {"is_compatible":false,"messages":["READER_FIELD_MISSING_DEFAULT_VALUE: discount ..."]}
The same check can run in CI before a schema reaches the registry, for example with the Confluent Maven plugin:
<plugin>
<groupId>io.confluent</groupId>
<artifactId>kafka-schema-registry-maven-plugin</artifactId>
<version>8.0.0</version>
<configuration>
<schemaRegistryUrls><param>http://schema-registry:8081</param></schemaRegistryUrls>
<subjects>
<orders-value>src/main/avro/Order.avsc</orders-value>
</subjects>
</configuration>
<!-- mvn schema-registry:test-compatibility in PR builds, schema-registry:register on merge -->
</plugin>
Choosing a mode
| Situation | Suggested mode |
|---|---|
| Typical event topic, consumers can be upgraded before producers | BACKWARD |
| Topic replayed from the start, compacted topics, long retention | BACKWARD_TRANSITIVE or FULL_TRANSITIVE |
| Many consumers you do not control (other teams, other companies) | FORWARD or FULL, so producers can ship first without breaking them |
| Producers and consumers deployed independently with no coordination | FULL_TRANSITIVE |
| Development environments | NONE is acceptable, never in production |
Key Takeaways
- Backward compatibility means new readers handle old data (upgrade consumers first). Forward compatibility means old readers handle new data (upgrade producers first). Full means both.
- Non-transitive modes check only the latest version. Use transitive modes for topics that are replayed or retain data for a long time.
- In Avro, defaults decide compatibility: add and remove only fields that have defaults to stay fully compatible.
- Test compatibility in CI with the registry's compatibility endpoint or build plugins before registering.
Practice
- (Beginner) With subject
lab-valueinBACKWARDmode, try to register a version that adds a field without a default, then one that adds a field with a default. Record the registry's responses.- (Intermediate) Construct three Avro schema versions where each is
BACKWARDcompatible with the previous one, but version 3 is not compatible with version 1. Show thatBACKWARD_TRANSITIVErejects version 3.- Interview question: "A topic's subject is set to
FORWARD. In what order do you deploy a schema change, and why?" Hint: work out which side must understand the other's schema at each step of the rollout, and what happens to records written while the rollout is in progress.
Schema Evolution
Theory
Compatibility modes tell you which changes the registry accepts. Schema evolution is the practice of making real changes through those rules: adding fields, deprecating others, renaming, and, occasionally, handling changes that cannot be made compatible.
At its core is schema resolution. When a consumer reads a record, two schemas are involved: the writer schema (looked up by the record's schema ID) and the reader schema (the one the consumer was compiled with, or configured to use). Fields are matched by name in Avro and by number in Protobuf.
writer schema v1 reader schema v2
+------------------+ +----------------------------+
| orderId | ---------> | orderId |
| customerId | ---------> | customerId |
| amountCents | ---------> | amountCents |
| channel | ignored | | field only in writer: skipped
| | | couponCode (default null) | field only in reader: default used
+------------------+ +----------------------------+
Analogy: schema evolution is like renovating a house while people live in it. You can add rooms and close doors, but you cannot remove the staircase until everyone has moved to the new floor plan.
Example: evolving an Avro schema through three versions
// v1
{ "type": "record", "name": "Order", "namespace": "com.example.orders",
"fields": [
{ "name": "orderId", "type": "string" },
{ "name": "customerId", "type": "long" },
{ "name": "amount", "type": "long" },
{ "name": "channel", "type": "string", "default": "web" }
] }
// v2: add an optional field (fully compatible)
{ "name": "couponCode", "type": ["null", "string"], "default": null }
// v3: rename amount to amountCents, remove channel, add an enum symbol with a default
{ "type": "record", "name": "Order", "namespace": "com.example.orders",
"fields": [
{ "name": "orderId", "type": "string" },
{ "name": "customerId", "type": "long" },
{ "name": "amountCents", "type": "long", "aliases": ["amount"] }, // reader still finds "amount" in old data
{ "name": "couponCode", "type": ["null", "string"], "default": null },
{ "name": "status",
"type": { "type": "enum", "name": "OrderStatus",
"symbols": ["PENDING", "PAID", "SHIPPED", "REFUNDED"],
"default": "PENDING" }, // unknown symbols map to PENDING
"default": "PENDING" }
] }
What a v3 consumer sees when it reads each version:
| Record written with | amountCents | couponCode | status | channel |
|---|---|---|---|---|
| v1 | From amount (alias) | null (default) | PENDING (default) | Ignored |
| v2 | From amount (alias) | Written value | PENDING (default) | Ignored |
| v3 | Written value | Written value | Written value | Not present |
Removing channel is safe only because it had a default. A v1 consumer reading v3 data fills it with "web". The alias makes the rename backward compatible but not forward compatible: v1 consumers still look for amount. So the rename requires BACKWARD with consumers upgraded first. Under FULL, you would add amountCents, write both fields for a while, and remove amount later.
Rules of thumb
| Do | Avoid |
|---|---|
Give every new field a default (Avro) or make it optional (Protobuf) | Required fields added after the first version |
| Deprecate fields first, remove them after all consumers have stopped reading them | Removing a field the same day producers stop writing it |
Add reserved entries for removed Protobuf field numbers and names | Reusing a field number |
| Give enums an "unknown" or default symbol from the first version | Treating enums as closed sets |
Use logical types (decimal, timestamp-millis) from the start | Changing the meaning of an existing field, such as euros to cents, without renaming it |
The last row is a trap that no compatibility checker catches. Changing amount from euros to cents keeps the type long, so the schema is "compatible", and every consumer silently computes wrong totals. A change in meaning needs a new field name.
The safe rollout sequence (BACKWARD mode)
1. Open a PR with the new schema -> CI runs test-compatibility against the registry
2. Merge -> CI registers the new version (auto.register=false in apps)
3. Deploy consumers built with v(n+1) -> they read both v(n) and v(n+1) records
4. Deploy producers with v(n+1) -> new records use the new schema ID
5. Later: remove deprecated fields in v(n+2), following the same steps
When a change cannot be made compatible
Sometimes a change truly breaks compatibility, such as a new key structure, a split entity, or a different meaning. Options, from simplest to most involved:
| Strategy | How it works | Cost |
|---|---|---|
| New topic | Producers write to orders.v2; consumers migrate at their own pace; a stream processor can translate v2 to v1 (or the reverse) during the transition | Two topics during migration, consumers must switch topics |
| Dual write | Producers write both formats to both topics for a period | Extra load and risk of the two topics diverging |
| Compatibility groups with migration rules | The registry checks compatibility only within a "major version", and clients transform records between versions on the fly (topic "Data Contracts") | Requires a registry and clients that support rules |
NONE and a coordinated stop | Stop all producers and consumers, deploy, restart | Downtime, and old records in the topic remain in the old format |
Key Takeaways
- Readers combine the writer schema with their own reader schema: extra writer fields are ignored, and missing fields take the reader's defaults.
- Add fields with defaults, deprecate before removing, never reuse Protobuf field numbers, and give enums a default symbol.
- Avro aliases support renames for new readers, but old readers still need the old name.
- A change in a field's meaning is a breaking change even if the checker accepts it. Use a new field or a new topic.
Practice
- (Beginner) Produce records with schema v1 above, register v2, and produce more. Read the whole topic with a consumer generated from v2 and confirm that old records show
couponCode = null.- (Intermediate) Plan the rename of
customerIdtobuyerIdunderFULL_TRANSITIVEcompatibility. List each schema version, what producers write, and when each version can be registered and retired.- Interview question: "A schema change passed the registry's compatibility check, but downstream reports were wrong the next day. How could that happen, and how would you prevent it?" Hint: compatibility checks look at structure; think about units, meaning, and value ranges, and where those could be checked.
Data Contracts
Theory
A schema guarantees structure: amountCents is a long, status is one of four symbols. It says nothing about whether amountCents may be negative, who to call when the topic breaks, which fields contain personal data, or how fresh the data should be. Consumers rely on all of those things, so a producer can break them without ever changing the schema.
A data contract is a formal, versioned agreement between the producer of a data stream and its consumers. It covers:
| Element | Question it answers | Example |
|---|---|---|
| Structure | What does a record look like? | The Avro, Protobuf, or JSON schema |
| Integrity and quality rules | Which values are valid? | amountCents > 0, currency is 3 uppercase letters |
| Metadata | Who owns it, and how sensitive is it? | Owner team, contact, customerEmail tagged as PII |
| Service levels | What can consumers expect? | Freshness under 5 minutes, retention of 7 days |
| Evolution policy | How can it change? | Compatibility mode, deprecation notice period, migration rules |
Analogy: a schema is the specification of a part's shape. A data contract is the full supplier agreement: the shape, the tolerances, the quality inspection, the warranty, and who to call when a delivery is wrong.
The idea is independent of any tool. Contracts can be YAML files in Git following a standard such as the Open Data Contract Standard (ODCS), reviewed in pull requests like code. Confluent Schema Registry implements data contracts directly, by attaching metadata and rules to a registered schema so that serializers can enforce them.
Data contracts in Confluent Schema Registry
A registered schema can carry three extra parts:
metadata.properties: free-form key-value pairs (owner, contact, SLAs,major_version).metadata.tags: tags applied to fields, such asPIIoncustomerEmail, used by rules and by data catalogs.ruleSet: rules executed by the client serializer and deserializer:- Domain rules run on every record. A
CONDITIONrule validates a record with a CEL expression. ATRANSFORMrule changes it, for example by encrypting fields taggedPII(client-side field-level encryption). - Migration rules transform records between incompatible major versions, with JSONata expressions that
UPGRADEold records for new readers orDOWNGRADEnew records for old readers.
- Domain rules run on every record. A
Data contract rules are commercial features (Confluent Platform Enterprise and Confluent Cloud's Stream Governance Advanced), and the client applications need the matching rule executor libraries.
Example: registering a schema with a contract
curl -s -X POST http://localhost:8081/subjects/orders-value/versions \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' \
-d @- <<'EOF'
{
"schemaType": "AVRO",
"schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"com.example.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"customerEmail\",\"type\":\"string\"},{\"name\":\"amountCents\",\"type\":\"long\"},{\"name\":\"currency\",\"type\":\"string\"}]}",
"metadata": {
"properties": {
"owner": "payments-team",
"owner_email": "payments@example.com",
"slo_freshness": "5m",
"major_version": "1"
},
"tags": { "Order.customerEmail": ["PII"] }
},
"ruleSet": {
"domainRules": [
{
"name": "validAmountAndCurrency",
"kind": "CONDITION",
"type": "CEL",
"mode": "WRITE",
"expr": "message.amountCents > 0 && message.currency.matches('^[A-Z]{3}$')",
"onFailure": "DLQ",
"params": { "dlq.topic": "orders.dlq" }
},
{
"name": "encryptPii",
"kind": "TRANSFORM",
"type": "ENCRYPT",
"mode": "WRITEREAD",
"tags": ["PII"],
"params": { "encrypt.kek.name": "orders-kek" }
}
]
}
}
EOF
With this contract in place:
producer.send(order)
-> serializer runs CEL condition
amountCents = -500 -> record sent to orders.dlq instead of orders (onFailure: DLQ)
amountCents = 1999 -> continue
-> serializer encrypts customerEmail (tagged PII) with a data key protected by orders-kek
-> record written to orders
consumer.poll()
-> deserializer decrypts customerEmail only if this client has access to orders-kek
Example: migrating across a breaking change
Compatibility groups let a subject hold incompatible major versions side by side. The registry checks compatibility only between versions with the same major_version, and migration rules translate records for clients that use a different major version:
# Check compatibility only within the same major_version
curl -s -X PUT http://localhost:8081/config/orders-value \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' \
-d '{"compatibilityGroup": "major_version"}'
"metadata": { "properties": { "major_version": "2" } },
"ruleSet": {
"migrationRules": [
{ "name": "stateToStatus", "kind": "TRANSFORM", "type": "JSONATA", "mode": "UPGRADE",
"expr": "$merge([$sift($, function($v, $k) {$k != 'state'}), {'status': $.'state'}])" },
{ "name": "statusToState", "kind": "TRANSFORM", "type": "JSONATA", "mode": "DOWNGRADE",
"expr": "$merge([$sift($, function($v, $k) {$k != 'status'}), {'state': $.'status'}])" }
]
}
# A consumer that has not migrated yet asks for major version 1;
# records written with version 2 are downgraded on read
use.latest.with.metadata=major_version=1
latest.compatibility.strict=false
Governance workflow
contract (schema + metadata + rules) in Git
|
v
pull request ---> review by producer team and key consumers
|
v
CI: compatibility test, rule lint, register on merge (only CI can write to the registry)
|
v
applications: auto.register.schemas=false, use.latest.version=true,
rule executors on the classpath
Key Takeaways
- A data contract extends a schema with quality rules, ownership and classification metadata, service levels, and an evolution policy.
- It is a practice first. Contracts can live in Git in a standard format such as ODCS, independent of any registry.
- Confluent Schema Registry enforces contracts in the client: CEL condition rules (with dead letter topics), field encryption based on tags, and JSONata migration rules between major versions.
- Treat contracts like code: reviewed in pull requests, checked in CI, and registered only by automation.
Practice
- (Beginner) Write a data contract for a
PaymentAuthorizedevent as a YAML document: schema, owner, two quality rules, the fields that are PII, and the compatibility mode. Explain who should approve changes to it.- (Intermediate) Using a registry that supports rules, attach a CEL condition requiring
amountCents > 0withonFailure: DLQ. Produce one valid and one invalid record and show where each ends up.- Interview question: "What is the difference between a schema and a data contract, and why would a producer team agree to be bound by one?" Hint: think about what consumers depend on beyond structure, who pays when an implicit agreement breaks, and how explicit contracts let producers change things safely.
<a id="7-storage-internals"></a>