Practice Challenges
Try each one before reading the hint. Use a local Docker Compose cluster.
| # | Challenge | Hint |
|---|---|---|
| 1 | Create a topic with 3 partitions. Produce 9 messages with keys a, b, c repeated. Verify the same key always lands in the same partition | --property print.key=true print.partition=true |
| 2 | Start two consumers in one group and one in another group on the same topic. Explain who gets which records | Partitions split inside a group, every group gets everything |
| 3 | Stop a consumer, produce records, restart it. Show it continues from its committed offset. Then reset offsets to the beginning | kafka-consumer-groups.sh --reset-offsets --to-earliest --execute |
| 4 | Write a producer with acks=all and idempotence. Stop one broker in a 3-broker cluster and show writes still work. Stop a second and show what happens | min.insync.replicas=2, NotEnoughReplicasException |
| 5 | Build a consumer with manual commits that sends records failing 3 times to a dead letter topic | Retry counter, separate producer for DLT |
| 6 | Create a compacted topic. Write several updates for the same key and a tombstone. Read it from the beginning and explain the result | Small segment.bytes and min.cleanable.dirty.ratio to see compaction quickly |
| 7 | Make a Streams app that counts events per key per minute and expose the counts with an HTTP endpoint | Windowed count plus interactive queries |
| 8 | Use Kafka Connect to copy rows of a database table into a topic and then into another table | JDBC source and sink or Debezium plus JDBC sink |
| 9 | Enable SASL_SSL on a test cluster and add an ACL so only one user can produce to a topic | kafka-acls.sh --producer |
| 10 | Measure producer throughput with and without compression and with different linger.ms | kafka-producer-perf-test.sh |
Sample Solutions
Challenge 1 commands:
Challenge 4 idea: with replication factor 3 and min.insync.replicas=2 and acks=all, writes work with 2 brokers up (ISR size 2). With only 1 broker up the ISR is 1, below the minimum, so producers get NotEnoughReplicasException and reads of existing data still work. This is the intended trade: refuse writes rather than risk loss.
Challenge 6 idea:
Cheat Sheet: CLI Commands
| Task | Command |
|---|---|
| Create topic | kafka-topics.sh --bootstrap-server $BS --create --topic T --partitions 6 --replication-factor 3 |
| List and describe | kafka-topics.sh --bootstrap-server $BS --list / --describe --topic T |
| Increase partitions | kafka-topics.sh --bootstrap-server $BS --alter --topic T --partitions 12 |
| Change topic config | kafka-configs.sh --bootstrap-server $BS --alter --entity-type topics --entity-name T --add-config retention.ms=86400000 |
| Console producer | kafka-console-producer.sh --bootstrap-server $BS --topic T --property parse.key=true --property key.separator=: |
| Console consumer | kafka-console-consumer.sh --bootstrap-server $BS --topic T --from-beginning --group G --property print.key=true |
| Groups and lag | kafka-consumer-groups.sh --bootstrap-server $BS --describe --group G |
| Reset offsets | kafka-consumer-groups.sh --bootstrap-server $BS --group G --topic T --reset-offsets --to-datetime 2026-05-01T00:00:00.000 --execute |
| Delete records before offset | kafka-delete-records.sh --bootstrap-server $BS --offset-json-file del.json |
| Under-replicated partitions | kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions |
| Controller quorum | kafka-metadata-quorum.sh --bootstrap-server $BS describe --status |
| Reassign partitions | kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --execute --throttle 50000000 |
| Log directories | kafka-log-dirs.sh --bootstrap-server $BS --describe |
| Dump a log segment | kafka-dump-log.sh --files /data/T-0/00000000000000000000.log --print-data-log |
| ACLs | kafka-acls.sh --bootstrap-server $BS --command-config admin.properties --list |
| Perf test | kafka-producer-perf-test.sh --topic T --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=$BS acks=all |
Cheat Sheet: Key Configurations
| Component | Setting | Typical value and meaning |
|---|---|---|
| Producer | acks | all for durability |
| Producer | enable.idempotence | true (no duplicates from retries) |
| Producer | linger.ms / batch.size | 5 to 50 ms / 64 to 256 KB for throughput |
| Producer | compression.type | lz4 or zstd |
| Producer | delivery.timeout.ms | 120000 (total time to deliver, including retries) |
| Consumer | group.id, auto.offset.reset | Group name, earliest or latest |
| Consumer | enable.auto.commit | false with commit after processing |
| Consumer | max.poll.records, max.poll.interval.ms | Batch size per poll and allowed processing time between polls (300000 default) |
| Consumer | isolation.level | read_committed with transactions |
| Consumer | partition.assignment.strategy | CooperativeStickyAssignor |
| Broker | num.partitions, default.replication.factor | Defaults for auto-created topics. Prefer explicit topics, auto.create.topics.enable=false |
| Broker | min.insync.replicas | 2 with replication factor 3 |
| Broker | unclean.leader.election.enable | false |
| Broker | log.retention.hours, log.segment.bytes | 168 (7 days) / 1 GB defaults |
| Broker | num.network.threads, num.io.threads | Tune with load (idle percent metrics) |
| Topic | cleanup.policy | delete (default), compact or both |
| Topic | retention.ms, retention.bytes, max.message.bytes | Time and size limits, maximum message size |
Common Mistakes to Avoid
- Relying on
auto.create.topics.enableand default partition counts - Replication factor 1 or
acks=1for important data - Auto-committing offsets while doing slow or fallible processing
- Ignoring duplicates: no idempotent handling
- Very large messages instead of storing blobs elsewhere and sending a reference
- Changing the key or partition count without thinking about ordering
- No monitoring of lag, under-replicated partitions and disk
- Unbounded retries with no dead letter topic
- One giant topic with unrelated events
- Skipping schema management until the first breaking change
Learning Roadmap
| Stage | Focus | Outcome |
|---|---|---|
| 1. Foundations | Topics, partitions, offsets, brokers, consumer groups, CLI tools | You can explain and demo the core model |
| 2. Developer skills | Producers and consumers in your language, serialization, error handling, idempotence | You can build reliable services |
| 3. Data integration | Kafka Connect, Debezium, Schema Registry | You can move data between systems without custom code |
| 4. Stream processing | Kafka Streams, windows, joins, ksqlDB or Flink | You can build real-time analytics |
| 5. Operations | Cluster setup, KRaft, upgrades, security, monitoring, DR | You can run production clusters |
| 6. Architecture | Event-driven patterns, topic design, governance, cloud and Kubernetes | You can design platforms and review designs |
Portfolio Project Ideas
- Order and payment microservices with outbox, retries and DLT (Project 1)
- Database to data lake CDC pipeline with Debezium (Project 2)
- Live analytics dashboard with Streams (Project 3)
- Fraud alert detector: sliding windows on transactions and a rule engine
- IoT sensor pipeline: MQTT to Kafka, anomaly detection, time-series sink
- Audit log platform with compacted topics and search
- Chaos test: kill brokers and consumers while measuring data loss and duplicates
What to Study Next
- The official Apache Kafka documentation and the KIP list for the version you run
- Source books and talks about distributed logs and stream processing
- Your cloud provider's managed Kafka documentation
- Apache Flink and Apache Iceberg for lakehouse streaming
- Observability: OpenTelemetry with Kafka clients
Conclusion
Practise by building: run the ten challenges on a local cluster, keep the cheat sheet nearby, avoid the common mistakes, and follow the roadmap from foundations to architecture, adding portfolio projects as you go.

