Kafka Stores Bytes
Kafka does not know what is inside a record. Producers serialize objects to bytes, and consumers deserialize bytes back to objects. Both sides must agree on the format.
| Format | Pros | Cons |
| String / JSON | Readable, easy to start | Larger, no enforced structure, field typos and type changes break consumers silently |
| Avro | Compact binary, schema evolution rules, great with Schema Registry | Not human readable without tools |
| Protobuf | Compact, strong typing, many languages, good evolution rules | Needs .proto management |
| JSON Schema | JSON with validation | Larger than binary formats |
JSON with Jackson (Simple Start)
java
public record Order(String orderId, String customerId, double amount, String currency) {}
public class JsonSerializer<T> implements org.apache.kafka.common.serialization.Serializer<T> {
private final com.fasterxml.jackson.databind.ObjectMapper mapper = new com.fasterxml.jackson.databind.ObjectMapper();
@Override public byte[] serialize(String topic, T data) {
try { return data == null ? null : mapper.writeValueAsBytes(data); }
catch (Exception e) { throw new org.apache.kafka.common.errors.SerializationException(e); }
}
}
public class JsonDeserializer<T> implements org.apache.kafka.common.serialization.Deserializer<T> {
private final com.fasterxml.jackson.databind.ObjectMapper mapper = new com.fasterxml.jackson.databind.ObjectMapper();
private final Class<T> type;
public JsonDeserializer(Class<T> type) { this.type = type; }
@Override public T deserialize(String topic, byte[] bytes) {
try { return bytes == null ? null : mapper.readValue(bytes, type); }
catch (Exception e) { throw new org.apache.kafka.common.errors.SerializationException(e); }
}
}
The silent breakage problem: If the producer team renames amount to total, JSON consumers may start getting null or crashing. In a large company, you need a contract that checks changes before they reach production. That is the job of a schema registry.
What is a Schema Registry?
A Schema Registry is a service that stores schemas (Avro, Protobuf, JSON Schema) and gives each a numeric id. The producer sends only the id (5 bytes) plus the compact data, and the consumer fetches the schema by id. The registry also checks compatibility when a new schema version is registered.
text
Producer: serialize with schema -> register or find schema -> id 42 -> record = [magic byte][id 42][avro bytes]
Consumer: read id 42 -> fetch schema from registry (cached) -> deserialize
Avro Schema Example
json
{
"type": "record",
"name": "Order",
"namespace": "com.shop.events",
"fields": [
{"name": "orderId", "type": "string"},
{"name": "customerId", "type": "string"},
{"name": "amount", "type": "double"},
{"name": "currency", "type": "string", "default": "INR"},
{"name": "coupon", "type": ["null", "string"], "default": null}
]
}
java
// Avro producer with Schema Registry (Confluent-compatible serializer)
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");
Schema schema = new Schema.Parser().parse(new File("order.avsc"));
GenericRecord order = new GenericData.Record(schema);
order.put("orderId", "1001");
order.put("customerId", "c1");
order.put("amount", 250.0);
order.put("currency", "INR");
order.put("coupon", null);
try (Producer<String, GenericRecord> p = new KafkaProducer<>(props)) {
p.send(new ProducerRecord<>("orders-avro", "c1", order));
}
Compatibility Modes
| Mode | Meaning | Safe changes |
| BACKWARD (default) | New schema can read data written with the old schema. Upgrade consumers first | Add field with a default, remove a field |
| FORWARD | Old schema can read data written with the new schema. Upgrade producers first | Add a field, remove a field that had a default |
| FULL | Both directions | Add or remove only fields with defaults |
*_TRANSITIVE | Check against all earlier versions, not just the previous one | Same rules, stricter |
| NONE | No checks | Anything (dangerous) |
bash
# register / check with the REST API
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\":\"record\",\"name\":\"Order\",...}"}' http://localhost:8081/subjects/orders-value/versions
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data @new-schema.json http://localhost:8081/compatibility/subjects/orders-value/versions/latest
curl http://localhost:8081/subjects
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility":"FULL"}' http://localhost:8081/config/orders-value
Subjects and Naming
The registry groups schema versions in a subject. The default strategy is TopicNameStrategy: subject orders-value for the value of topic orders, and orders-key for the key. Other strategies (RecordNameStrategy) allow several event types in one topic.
Event Design Tips
- Include an event id, event type, timestamp and a version in every event
- Use ids and facts, not whole database rows. Include enough data so consumers do not need to call back
- Add new fields as optional with defaults. Never change the meaning or type of an existing field
- Prefer one topic per event type or aggregate with a clear name:
orders.created, payments.completed
- Use UTC timestamps (epoch millis or ISO 8601) and decimal types or integer minor units for money, never floating point for amounts that must be exact
- Keep personal data minimal and consider field-level encryption
Other registries: Besides the Confluent Schema Registry, compatible registries exist (for example Apicurio and the AWS Glue Schema Registry). The ideas are the same.
Conclusion
Kafka transports bytes, so agree on a format. JSON is easy but fragile. Avro or Protobuf with a Schema Registry gives compact data and compatibility checks that stop breaking changes before they reach consumers.
Quick Summary: bytes in Kafka; Avro/Protobuf + registry; BACKWARD default; add optional fields with defaults.