Serialization, JSON, Avro and Schema Registry for Kafka Events

Apache Kafka 18 min min read Updated: Nov 17, 2026 Intermediate
Serialization, JSON, Avro and Schema Registry for Kafka Events
Intermediate Topic 4 of 4

Kafka Stores Bytes

Kafka does not know what is inside a record. Producers serialize objects to bytes, and consumers deserialize bytes back to objects. Both sides must agree on the format.

FormatProsCons
String / JSONReadable, easy to startLarger, no enforced structure, field typos and type changes break consumers silently
AvroCompact binary, schema evolution rules, great with Schema RegistryNot human readable without tools
ProtobufCompact, strong typing, many languages, good evolution rulesNeeds .proto management
JSON SchemaJSON with validationLarger than binary formats

JSON with Jackson (Simple Start)

java public record Order(String orderId, String customerId, double amount, String currency) {} public class JsonSerializer<T> implements org.apache.kafka.common.serialization.Serializer<T> { private final com.fasterxml.jackson.databind.ObjectMapper mapper = new com.fasterxml.jackson.databind.ObjectMapper(); @Override public byte[] serialize(String topic, T data) { try { return data == null ? null : mapper.writeValueAsBytes(data); } catch (Exception e) { throw new org.apache.kafka.common.errors.SerializationException(e); } } } public class JsonDeserializer<T> implements org.apache.kafka.common.serialization.Deserializer<T> { private final com.fasterxml.jackson.databind.ObjectMapper mapper = new com.fasterxml.jackson.databind.ObjectMapper(); private final Class<T> type; public JsonDeserializer(Class<T> type) { this.type = type; } @Override public T deserialize(String topic, byte[] bytes) { try { return bytes == null ? null : mapper.readValue(bytes, type); } catch (Exception e) { throw new org.apache.kafka.common.errors.SerializationException(e); } } }
The silent breakage problem: If the producer team renames amount to total, JSON consumers may start getting null or crashing. In a large company, you need a contract that checks changes before they reach production. That is the job of a schema registry.

What is a Schema Registry?

A Schema Registry is a service that stores schemas (Avro, Protobuf, JSON Schema) and gives each a numeric id. The producer sends only the id (5 bytes) plus the compact data, and the consumer fetches the schema by id. The registry also checks compatibility when a new schema version is registered.

text Producer: serialize with schema -> register or find schema -> id 42 -> record = [magic byte][id 42][avro bytes] Consumer: read id 42 -> fetch schema from registry (cached) -> deserialize

Avro Schema Example

json { "type": "record", "name": "Order", "namespace": "com.shop.events", "fields": [ {"name": "orderId", "type": "string"}, {"name": "customerId", "type": "string"}, {"name": "amount", "type": "double"}, {"name": "currency", "type": "string", "default": "INR"}, {"name": "coupon", "type": ["null", "string"], "default": null} ] }
java // Avro producer with Schema Registry (Confluent-compatible serializer) Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); props.put("schema.registry.url", "http://localhost:8081"); Schema schema = new Schema.Parser().parse(new File("order.avsc")); GenericRecord order = new GenericData.Record(schema); order.put("orderId", "1001"); order.put("customerId", "c1"); order.put("amount", 250.0); order.put("currency", "INR"); order.put("coupon", null); try (Producer<String, GenericRecord> p = new KafkaProducer<>(props)) { p.send(new ProducerRecord<>("orders-avro", "c1", order)); }

Compatibility Modes

ModeMeaningSafe changes
BACKWARD (default)New schema can read data written with the old schema. Upgrade consumers firstAdd field with a default, remove a field
FORWARDOld schema can read data written with the new schema. Upgrade producers firstAdd a field, remove a field that had a default
FULLBoth directionsAdd or remove only fields with defaults
*_TRANSITIVECheck against all earlier versions, not just the previous oneSame rules, stricter
NONENo checksAnything (dangerous)
bash # register / check with the REST API curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"schema": "{\"type\":\"record\",\"name\":\"Order\",...}"}' http://localhost:8081/subjects/orders-value/versions curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data @new-schema.json http://localhost:8081/compatibility/subjects/orders-value/versions/latest curl http://localhost:8081/subjects curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility":"FULL"}' http://localhost:8081/config/orders-value

Subjects and Naming

The registry groups schema versions in a subject. The default strategy is TopicNameStrategy: subject orders-value for the value of topic orders, and orders-key for the key. Other strategies (RecordNameStrategy) allow several event types in one topic.

Event Design Tips

  • Include an event id, event type, timestamp and a version in every event
  • Use ids and facts, not whole database rows. Include enough data so consumers do not need to call back
  • Add new fields as optional with defaults. Never change the meaning or type of an existing field
  • Prefer one topic per event type or aggregate with a clear name: orders.created, payments.completed
  • Use UTC timestamps (epoch millis or ISO 8601) and decimal types or integer minor units for money, never floating point for amounts that must be exact
  • Keep personal data minimal and consider field-level encryption
Other registries: Besides the Confluent Schema Registry, compatible registries exist (for example Apicurio and the AWS Glue Schema Registry). The ideas are the same.

Conclusion

Kafka transports bytes, so agree on a format. JSON is easy but fragile. Avro or Protobuf with a Schema Registry gives compact data and compatibility checks that stop breaking changes before they reach consumers.

Quick Summary: bytes in Kafka; Avro/Protobuf + registry; BACKWARD default; add optional fields with defaults.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators