Kafka Interview Questions for Administrators and Architects: Cluster, Operations and Design

Apache Kafka 22 min min read Updated: Nov 17, 2026 Advanced
Kafka Interview Questions for Administrators and Architects: Cluster, Operations and Design
Advanced Topic 2 of 3

Cluster and Architecture

1. Describe the architecture of a production Kafka cluster.

Three or more brokers (often across three availability zones, with broker.rack set), a KRaft controller quorum (3 or 5 controllers, either dedicated or combined with brokers in small clusters), replication factor 3 for important topics, min.insync.replicas=2, monitored by Prometheus and Grafana, secured with TLS plus SASL and ACLs, and integrated with Schema Registry, Connect and a mirroring setup for DR.

2. How does the KRaft quorum work?

Controller nodes form a Raft quorum that stores cluster metadata in an internal log (__cluster_metadata). One is the active controller, the others replicate its log. A majority must be available: 3 controllers tolerate 1 failure, 5 tolerate 2. Brokers fetch metadata from the quorum.

3. Dedicated or combined controllers?

Combined (broker and controller on the same node) is fine for small and development clusters. Production clusters of meaningful size use dedicated controllers so heavy broker load cannot starve metadata operations.

4. How did you migrate from ZooKeeper to KRaft?

Upgrade to the bridge release (Kafka 3.9), verify metadata version, deploy KRaft controllers in migration mode, enable migration on brokers, let metadata copy, move brokers one by one to KRaft mode, finalize, then remove ZooKeeper. Since Kafka 4.0 cannot run with ZooKeeper, migrate before upgrading to 4.x.

5. How are replicas placed across brokers?

The controller spreads replicas over brokers and, with broker.rack, tries to put replicas in different racks or zones. Reassignment tools or Cruise Control can rebalance later.

Storage and Retention

6. How does retention work?

Logs are split into segments. Old closed segments are deleted when they exceed retention.ms or retention.bytes (checked by a background thread). The active segment is never deleted, so a low-traffic topic may keep data longer than expected unless segment.ms is also small.

7. Explain log compaction.

With cleanup.policy=compact the log keeps at least the latest record per key. A cleaner thread removes older records for the same key in closed segments. A record with a null value (tombstone) marks deletion and is removed after delete.retention.ms. Used for changelog and state topics such as __consumer_offsets.

8. What is tiered storage?

A feature that keeps recent segments on broker disks and moves older ones to remote object storage. It lowers disk cost and speeds up scaling, because less data lives on brokers. Check the support level in your Kafka version and platform.

9. A disk is 95 percent full. What do you do?

  1. Reduce the growth: throttle or pause big producers, lower retention for non-critical large topics (dynamic config, effective after the next cleanup)
  2. Find the biggest topics with kafka-log-dirs.sh
  3. Add disk or move partitions with a throttled reassignment
  4. Never delete segment files manually while the broker is running
  5. Add alerts at 70 and 85 percent and review capacity planning

Operations

10. How do you do a rolling upgrade?

  1. Read the release notes and upgrade guide, check client compatibility
  2. Upgrade brokers one at a time: controlled shutdown, install, start, wait until under-replicated partitions are 0
  3. Keep the metadata version (and inter-broker settings if needed) at the old value until all brokers run the new binary
  4. Finalize the upgrade by raising the metadata version (kafka-features.sh upgrade)
  5. Keep a rollback plan until finalization

11. How do you move partitions between brokers?

Generate a plan with kafka-reassign-partitions.sh --generate (or use Cruise Control), execute with --execute using a throttle, verify with --verify and remove the throttle. Moving data creates network and disk load, so run it in quiet hours.

12. How do you add a broker to a running cluster?

Start the broker with a unique node id, it joins the cluster but holds no data. Move partitions to it with reassignment or Cruise Control. New topics will use it automatically.

13. What is a controlled shutdown?

When a broker stops gracefully it asks the controller to move partition leadership away first, so clients see minimal disruption. Always stop brokers gracefully (kafka-server-stop.sh or SIGTERM), not kill -9.

14. How do you change topic configuration without downtime?

kafka-configs.sh --alter --add-config ... --entity-type topics. Many settings (retention, compression, min.insync.replicas) apply dynamically. Broker settings can be updated dynamically per broker or cluster-wide for some properties, others need a rolling restart.

Performance

15. How do you tune Kafka for high throughput?

AreaSettings or actions
ProducerLarger batch.size, linger.ms 5 to 50, compression.type=lz4 or zstd, enough buffer.memory
BrokerEnough num.network.threads, num.io.threads, fast disks (SSD or NVMe), JBOD or RAID design, XFS file system, sensible log.segment.bytes
ConsumerLarger fetch.min.bytes and max.poll.records, enough partitions and consumers
OSLarge file descriptor limit, vm.swappiness=1, vm.max_map_count, noatime mounts, leave RAM for page cache
JVMHeap 6 to 8 GB, G1GC, avoid large heaps

16. How do you decide on the number of partitions?

From the target throughput divided by the measured throughput per partition (both for producers and consumers), and from the maximum consumer parallelism you will want. Also consider key ordering needs, broker partition limits, replication overhead and recovery time. Start moderate (for example 6 to 12) and measure.

17. What is the effect of too many partitions?

  • More open files and memory
  • Longer leader election and recovery after failures
  • More replication and metadata traffic
  • Higher end-to-end latency in some cases
  • KRaft raises limits compared to ZooKeeper, but design with care

18. How do you benchmark a cluster?

Use kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh with your real message size, replication factor, acks and compression. Measure throughput, p99 latency, broker CPU, disk and network.

Security

19. How do you secure a Kafka cluster?

TLS for all listeners, SASL (SCRAM, OAUTHBEARER or Kerberos) or mutual TLS for authentication, an authorizer with default deny and ACLs per principal, secrets in a secret manager, network isolation, disk encryption, auditing, quotas and regular patching.

20. What ACLs does a producer and a consumer need?

Producer: Write and Describe on the topic, Create if it creates topics, IdempotentWrite on the cluster (when needed), and Write on the transactional.id for transactions. Consumer: Read and Describe on the topic, Read on the consumer group.

21. How do you roll out security on a running cluster without downtime?

Add a new secured listener beside the old one, switch clients one by one, switch inter-broker communication, then remove the plaintext listener. Do each step as a rolling restart.

Reliability and DR

22. What replication settings do you use for critical data?

replication.factor=3, min.insync.replicas=2, producers with acks=all and idempotence, unclean.leader.election.enable=false, rack awareness across zones.

23. How do you design disaster recovery?

One cluster per region with MirrorMaker 2 (or a vendor cluster linking feature) replicating topics, configs and translated consumer offsets. Define RPO and RTO, document and rehearse failover and failback. Make consumers idempotent because of possible duplicates and replication lag.

24. Stretch cluster or mirrored clusters?

A stretch cluster across 3 zones in one region works well (low latency). Across distant regions, latency hurts acks=all, so prefer separate clusters with asynchronous mirroring.

25. What do you monitor?

  • Under-replicated and offline partitions, active controller count, ISR shrinks
  • Request latency, handler and network idle percent
  • Disk usage and I/O, network, GC
  • Consumer lag and rebalances
  • Certificate expiry and authorization failures
  • Connect task states and DLQ growth

Architecture Scenarios

26. Design a pipeline that streams database changes to a data lake and a search index.

Debezium source on the database (log-based CDC) into Kafka with Avro and Schema Registry. Sink connectors write Parquet to object storage partitioned by time and index documents into OpenSearch. Use compacted topics for latest state, DLQs on sinks, monitoring of connector state and lag, and a schema compatibility policy.

27. Multi-tenant Kafka for many teams: what do you put in place?

Naming conventions and ownership per topic prefix, ACLs by prefix, quotas per user, topic creation through self-service (Git and operator) with policies (partition and retention limits), per-team dashboards and cost visibility, schema governance, and a platform team on-call.

28. Kafka or RabbitMQ or a cloud queue?

Kafka suits high throughput streams, replay, multiple independent consumers and stream processing. RabbitMQ or cloud queues suit task queues, complex routing, per-message acknowledgement and lower volumes. Many systems use both.

29. A broker crashed and restarts slowly. Why?

After an unclean shutdown the broker recovers log segments and rebuilds indexes, which takes long with many or large partitions. Mitigate with graceful shutdowns, num.recovery.threads.per.data.dir higher, fewer partitions per broker, and proper disk sizing.

30. How do you estimate capacity for 100 MB/s ingress with retention of 7 days and replication factor 3?

Storage: 100 MB/s x 86,400 s x 7 = about 60 TB raw, times replication 3 = about 181 TB, plus 30 percent headroom, about 235 TB. Network: ingress is multiplied by replication on the cluster, and egress by number of consumer groups. Divide by broker count and keep under 50 to 60 percent of NIC and disk limits. Then validate with a performance test.

Conclusion

Administrator and architect interviews focus on KRaft and replication settings, retention and compaction, rolling upgrades and reassignments, performance tuning, security rollout, monitoring, disaster recovery, capacity maths and clear scenario thinking.

Quick Summary: RF3 + minISR2; KRaft quorum; rolling upgrade; throttled reassignment; MM2 DR; capacity = rate x time x RF.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators