Kafka Practice Challenges, Cheat Sheet and Learning Roadmap

Apache Kafka 17 min min read Updated: Nov 17, 2026 Beginner
Kafka Practice Challenges, Cheat Sheet and Learning Roadmap
Beginner Topic 3 of 3

Practice Challenges

Try each one before reading the hint. Use a local Docker Compose cluster.

#ChallengeHint
1Create a topic with 3 partitions. Produce 9 messages with keys a, b, c repeated. Verify the same key always lands in the same partition--property print.key=true print.partition=true
2Start two consumers in one group and one in another group on the same topic. Explain who gets which recordsPartitions split inside a group, every group gets everything
3Stop a consumer, produce records, restart it. Show it continues from its committed offset. Then reset offsets to the beginningkafka-consumer-groups.sh --reset-offsets --to-earliest --execute
4Write a producer with acks=all and idempotence. Stop one broker in a 3-broker cluster and show writes still work. Stop a second and show what happensmin.insync.replicas=2, NotEnoughReplicasException
5Build a consumer with manual commits that sends records failing 3 times to a dead letter topicRetry counter, separate producer for DLT
6Create a compacted topic. Write several updates for the same key and a tombstone. Read it from the beginning and explain the resultSmall segment.bytes and min.cleanable.dirty.ratio to see compaction quickly
7Make a Streams app that counts events per key per minute and expose the counts with an HTTP endpointWindowed count plus interactive queries
8Use Kafka Connect to copy rows of a database table into a topic and then into another tableJDBC source and sink or Debezium plus JDBC sink
9Enable SASL_SSL on a test cluster and add an ACL so only one user can produce to a topickafka-acls.sh --producer
10Measure producer throughput with and without compression and with different linger.mskafka-producer-perf-test.sh

Sample Solutions

Challenge 1 commands:

bash bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic keys-demo --partitions 3 printf 'a:1\nb:1\nc:1\na:2\nb:2\nc:2\na:3\nb:3\nc:3\n' | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic keys-demo \ --property parse.key=true --property key.separator=: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic keys-demo --from-beginning \ --property print.key=true --property print.partition=true --timeout-ms 5000 # every record with key a shows the same partition number

Challenge 4 idea: with replication factor 3 and min.insync.replicas=2 and acks=all, writes work with 2 brokers up (ISR size 2). With only 1 broker up the ISR is 1, below the minimum, so producers get NotEnoughReplicasException and reads of existing data still work. This is the intended trade: refuse writes rather than risk loss.

Challenge 6 idea:

bash bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic latest-state \ --config cleanup.policy=compact --config segment.bytes=1048576 --config min.cleanable.dirty.ratio=0.01 --config delete.retention.ms=1000 # produce user1:v1, user1:v2, user2:x, user1:v3, user2:<null> (tombstone) # after compaction a full read shows only user1:v3 (and later user2 disappears)

Cheat Sheet: CLI Commands

TaskCommand
Create topickafka-topics.sh --bootstrap-server $BS --create --topic T --partitions 6 --replication-factor 3
List and describekafka-topics.sh --bootstrap-server $BS --list / --describe --topic T
Increase partitionskafka-topics.sh --bootstrap-server $BS --alter --topic T --partitions 12
Change topic configkafka-configs.sh --bootstrap-server $BS --alter --entity-type topics --entity-name T --add-config retention.ms=86400000
Console producerkafka-console-producer.sh --bootstrap-server $BS --topic T --property parse.key=true --property key.separator=:
Console consumerkafka-console-consumer.sh --bootstrap-server $BS --topic T --from-beginning --group G --property print.key=true
Groups and lagkafka-consumer-groups.sh --bootstrap-server $BS --describe --group G
Reset offsetskafka-consumer-groups.sh --bootstrap-server $BS --group G --topic T --reset-offsets --to-datetime 2026-05-01T00:00:00.000 --execute
Delete records before offsetkafka-delete-records.sh --bootstrap-server $BS --offset-json-file del.json
Under-replicated partitionskafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions
Controller quorumkafka-metadata-quorum.sh --bootstrap-server $BS describe --status
Reassign partitionskafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --execute --throttle 50000000
Log directorieskafka-log-dirs.sh --bootstrap-server $BS --describe
Dump a log segmentkafka-dump-log.sh --files /data/T-0/00000000000000000000.log --print-data-log
ACLskafka-acls.sh --bootstrap-server $BS --command-config admin.properties --list
Perf testkafka-producer-perf-test.sh --topic T --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=$BS acks=all

Cheat Sheet: Key Configurations

ComponentSettingTypical value and meaning
Produceracksall for durability
Producerenable.idempotencetrue (no duplicates from retries)
Producerlinger.ms / batch.size5 to 50 ms / 64 to 256 KB for throughput
Producercompression.typelz4 or zstd
Producerdelivery.timeout.ms120000 (total time to deliver, including retries)
Consumergroup.id, auto.offset.resetGroup name, earliest or latest
Consumerenable.auto.commitfalse with commit after processing
Consumermax.poll.records, max.poll.interval.msBatch size per poll and allowed processing time between polls (300000 default)
Consumerisolation.levelread_committed with transactions
Consumerpartition.assignment.strategyCooperativeStickyAssignor
Brokernum.partitions, default.replication.factorDefaults for auto-created topics. Prefer explicit topics, auto.create.topics.enable=false
Brokermin.insync.replicas2 with replication factor 3
Brokerunclean.leader.election.enablefalse
Brokerlog.retention.hours, log.segment.bytes168 (7 days) / 1 GB defaults
Brokernum.network.threads, num.io.threadsTune with load (idle percent metrics)
Topiccleanup.policydelete (default), compact or both
Topicretention.ms, retention.bytes, max.message.bytesTime and size limits, maximum message size

Common Mistakes to Avoid

  • Relying on auto.create.topics.enable and default partition counts
  • Replication factor 1 or acks=1 for important data
  • Auto-committing offsets while doing slow or fallible processing
  • Ignoring duplicates: no idempotent handling
  • Very large messages instead of storing blobs elsewhere and sending a reference
  • Changing the key or partition count without thinking about ordering
  • No monitoring of lag, under-replicated partitions and disk
  • Unbounded retries with no dead letter topic
  • One giant topic with unrelated events
  • Skipping schema management until the first breaking change

Learning Roadmap

StageFocusOutcome
1. FoundationsTopics, partitions, offsets, brokers, consumer groups, CLI toolsYou can explain and demo the core model
2. Developer skillsProducers and consumers in your language, serialization, error handling, idempotenceYou can build reliable services
3. Data integrationKafka Connect, Debezium, Schema RegistryYou can move data between systems without custom code
4. Stream processingKafka Streams, windows, joins, ksqlDB or FlinkYou can build real-time analytics
5. OperationsCluster setup, KRaft, upgrades, security, monitoring, DRYou can run production clusters
6. ArchitectureEvent-driven patterns, topic design, governance, cloud and KubernetesYou can design platforms and review designs

Portfolio Project Ideas

  • Order and payment microservices with outbox, retries and DLT (Project 1)
  • Database to data lake CDC pipeline with Debezium (Project 2)
  • Live analytics dashboard with Streams (Project 3)
  • Fraud alert detector: sliding windows on transactions and a rule engine
  • IoT sensor pipeline: MQTT to Kafka, anomaly detection, time-series sink
  • Audit log platform with compacted topics and search
  • Chaos test: kill brokers and consumers while measuring data loss and duplicates

What to Study Next

  • The official Apache Kafka documentation and the KIP list for the version you run
  • Source books and talks about distributed logs and stream processing
  • Your cloud provider's managed Kafka documentation
  • Apache Flink and Apache Iceberg for lakehouse streaming
  • Observability: OpenTelemetry with Kafka clients

Conclusion

Practise by building: run the ten challenges on a local cluster, keep the cheat sheet nearby, avoid the common mistakes, and follow the roadmap from foundations to architecture, adding portfolio projects as you go.

Quick Summary: 10 challenges; CLI + config cheat sheet; avoid mistakes; roadmap foundations -> operations -> architecture.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators