Why KRaft?
| ZooKeeper era (Kafka 3.x and earlier) | KRaft (Kafka 4.x) |
| Separate ZooKeeper ensemble to run, secure and monitor | One system: Kafka nodes manage their own metadata |
| Metadata changes pushed through ZooKeeper, controller reloads state on failover (slow for big clusters) | Metadata is a replicated log. A new active controller already has the state, so failover is fast |
| Limited partition scale per cluster | Supports far more partitions per cluster |
| Two security models | One security model |
Version status: Kafka 4.0 removed ZooKeeper completely. Kafka 3.9 was the last release that supported it. A ZooKeeper-based cluster must be migrated to KRaft before upgrading to 4.x.
How It Works
- A set of controller nodes forms a quorum using the Raft consensus protocol. One is the active controller (leader), the others are followers
- All cluster metadata (topics, partitions, configs, ACLs, broker registrations) is stored as events in an internal log, the metadata log (topic
__cluster_metadata)
- Brokers fetch the metadata log from the controllers and keep a local copy, so they know the cluster state without asking for every request
- A change (create topic, new leader) is a record committed by a majority of the quorum
text
Controller quorum (3 nodes): C1 (leader) C2 C3 -> tolerates 1 failure (majority = 2)
Controller quorum (5 nodes): ... -> tolerates 2 failures (majority = 3)
Brokers: B1 B2 B3 B4 ... fetch metadata from the controllers
Node Roles
| Mode | process.roles | Use |
| Combined | broker,controller | Development and very small clusters. Simple, but controller and broker share resources |
| Dedicated controllers | controller on separate nodes, brokers use broker | Production. Isolates metadata work from heavy data traffic |
text
# controller node (config/controller.properties)
process.roles=controller
node.id=1
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
log.dirs=/var/lib/kafka/metadata
# broker node (config/broker.properties)
process.roles=broker
node.id=101
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker1:9092
log.dirs=/var/lib/kafka/data
bash
# format every node with the SAME cluster id (once)
CID=$(bin/kafka-storage.sh random-uuid)
bin/kafka-storage.sh format -t $CID -c config/controller.properties # on controllers
bin/kafka-storage.sh format -t $CID -c config/broker.properties # on brokers
Sizing the Quorum
| Controllers | Tolerated failures | Advice |
| 1 | 0 | Development only |
| 3 | 1 | Standard for most production clusters |
| 5 | 2 | Large or critical clusters, or when you want maintenance during a failure |
More than 5 controllers slows down commits without much benefit. Use an odd number. Spread controllers over failure domains (racks or zones).
Inspecting the Quorum
bash
bin/kafka-metadata-quorum.sh --bootstrap-server $BS describe --status
# ClusterId: ... LeaderId: 1 LeaderEpoch: 7 HighWatermark: 24531
# CurrentVoters: [1,2,3] CurrentObservers: [101,102,103]
bin/kafka-metadata-quorum.sh --bootstrap-server $BS describe --replication # lag of each voter and observer
bin/kafka-features.sh --bootstrap-server $BS describe # metadata.version and feature levels
Dynamic Quorum
Earlier KRaft versions fixed the voter list in a static file. Newer versions (3.9 and later) support a dynamic quorum, where controllers can be added or removed with tools such as kafka-metadata-quorum.sh add-controller and remove-controller, without restarting the whole cluster. Check the documentation of your version for the exact procedure.
Migrating from ZooKeeper (Outline)
- Upgrade the cluster to 3.9.x (the bridge release)
- Deploy a KRaft controller quorum in migration mode, configured to connect to ZooKeeper
- Enable migration on the brokers (
zookeeper.metadata.migration.enable=true and the controller quorum settings) and do a rolling restart
- The controllers copy all metadata from ZooKeeper into the KRaft metadata log
- Roll brokers to KRaft mode (remove the ZooKeeper settings, set roles and quorum), then finalize the migration
- Shut down ZooKeeper, then upgrade to 4.x
Test first and keep a rollback plan: Migration steps changed between 3.x versions and have a point of no return when you finalize. Follow the official migration guide of your exact version, rehearse in a staging cluster and back up topic data and configurations.
Operational Differences
| Topic | KRaft behaviour |
| Topic and config tools | Use --bootstrap-server everywhere. Tools that needed --zookeeper were removed |
| Broker registration | Brokers register with the controller quorum |
| Monitoring | Watch ActiveControllerCount (exactly one), LastAppliedRecordLagMs, metadata log size and quorum lag |
| Security | ACLs are stored in the metadata log. Secure the controller listener with TLS or SASL too |
| Disaster recovery | Back up configs and use cluster mirroring or snapshots of the controller metadata folder |
Conclusion
KRaft stores metadata in a replicated log managed by a Raft controller quorum, so Kafka no longer needs ZooKeeper. Use three dedicated controllers in production, format nodes with one cluster id, and migrate old clusters through 3.9 before moving to 4.x.
Quick Summary: KRaft = Raft controllers + metadata log; 3 dedicated controllers; migrate via 3.9; ZooKeeper gone in 4.x.