What You'll Learn
- Topics, partitions and offsets
- Producers and consumer groups
- Replication and exactly-once
- Kafka Connect and Debezium CDC
- Kafka Streams and ksqlDB
- Security, monitoring and projects
Prerequisites
- No prior programming experience needed
- Basic computer knowledge
- A laptop with internet access
- Enthusiasm to learn!
Tools & Setup
Tools: Apache Kafka (KRaft), Docker, Java, Spring BootExtras: Kafka Connect, Debezium, Schema Registry, Prometheus & Grafana
Level: Beginner to Advanced
About the Apache Kafka Tutorial
Apache Kafka is a distributed event streaming platform. Companies use it to move events between services, build real-time pipelines and keep systems loosely coupled. This tutorial covers Apache Kafka in 42 lessons, and the lessons follow a fixed order, so each one builds on the one before it.
What you will learn
The curriculum has 42 lessons in 13 chapters, with around 13 hours of reading time before you add practice:
- Chapter 1 (4 lessons): What is Apache Kafka? Events, Topics, Brokers and Why Companies Use It, Install Kafka, Kafka Command Line Tools, Core Concepts in Depth.
- Chapter 2 (4 lessons): Kafka Producer API in Java, Producer Reliability and Performance, Partitioning and Message Keys, Serialization, JSON, Avro and Schema Registry for Kafka Events.
- Chapter 3 (4 lessons): Kafka Consumer API in Java, Offset Commit Strategies, Rebalancing and Static Membership, Consumer Error Handling, Consumer Scaling and Lag.
- Chapter 4 (4 lessons): Replication in Depth, Delivery Semantics and Transactions, Log Storage Internals, KRaft Explained.
- Chapter 5 (4 lessons): Set Up a Multi-Broker Kafka Cluster, Topic and Cluster Administration, Broker Tuning and Performance Testing, Rolling Upgrades, Maintenance and Capacity Planning for Kafka.
- Chapter 6 (2 lessons): Kafka Security Part 1, Kafka Security Part 2.
- Chapter 7 (3 lessons): Monitoring Kafka, Troubleshooting Kafka, Multi-Cluster Kafka.
- Chapter 8 (3 lessons): Kafka Connect Basics, Kafka Connect in Practice, Change Data Capture with Debezium.
- Chapter 9 (4 lessons): Kafka Streams Basics, Kafka Streams Stateful Processing, Kafka Streams Interactive Queries, Testing, Deployment and Monitoring, ksqlDB and Stream Processing Options.
- Chapter 10 (2 lessons): Spring Boot with Kafka, Kafka Clients for Python, Node.js and Go with Schema Registry and Avro.
- Chapter 11 (2 lessons): Event-Driven Architecture Patterns with Kafka, Running Kafka on Kubernetes and in the Cloud.
- Chapter 12 (3 lessons): Event-Driven Order System, Real-Time Database Replication Pipeline, Real-Time Analytics.
- Chapter 13 (3 lessons): Kafka Interview Questions for Developers, Kafka Interview Questions for Administrators and Architects, Kafka Practice Challenges, Cheat Sheet and Learning Roadmap.
Who this tutorial is for
It suits back-end developers, data engineers and architects who work with event-driven systems. Prerequisites: Basic programming, preferably Java, and a general idea of how back-end services work.
How to get the most from it
Type out every example instead of copying it, then change it and see what breaks. Finish each chapter before moving on, build the projects and labs without looking at the solution first, and use the Apache Kafka interview questions to check what you remember. You will need Java and Maven or Gradle, Kafka (local or in Docker), and an IDE such as IntelliJ.
Apache Kafka Tutorial Curriculum: 42 Lessons in 13 Chapters
Learn Apache Kafka step by step , from beginner to advanced level. Start with What is Apache Kafka? Events, Topics, Brokers and Why Companies Use It and work up to Kafka Practice Challenges, Cheat Sheet and Learning Roadmap (about 13 hours in total).
Getting Started with Kafka
-
What is Apache Kafka? Events, Topics, Brokers and Why Companies Use It Understand Kafka in plain words: what problem it solves, the main parts and common use cases14 min
-
Install Kafka: KRaft Mode, Docker and Your First Broker Run a single-node Kafka on your computer without ZooKeeper and check that it works14 min
-
Kafka Command Line Tools: Create Topics, Produce, Consume and Inspect Practise with the built-in scripts to create topics, send messages and read them back16 min
-
Core Concepts in Depth: Partitions, Offsets, Segments, Retention and Replication Learn how Kafka stores data on disk and why the design is fast and safe16 min
Producers
-
Kafka Producer API in Java: Send Messages, Callbacks and Keys Write your first Java producer, send keyed events and handle the result of each send17 min
-
Producer Reliability and Performance: acks, retries, Idempotence, Batching and Compression Choose settings so messages are not lost or duplicated, and tune for throughput or latency18 min
-
Partitioning and Message Keys: Ordering, Custom Partitioners and Hot Partitions Decide which partition each event goes to, keep order where needed and avoid uneven load16 min
-
Serialization, JSON, Avro and Schema Registry for Kafka Events Send structured events safely, evolve them over time and share schemas between teams18 min
Consumers
-
Kafka Consumer API in Java: poll Loop, Consumer Groups and Offsets Write a consumer, understand group membership and read from a chosen position17 min
-
Offset Commit Strategies, Rebalancing and Static Membership Choose how to commit offsets and keep consumer groups stable during deployments17 min
-
Consumer Error Handling: Retries, Dead Letter Topics, Poison Messages and Idempotent Processing Keep pipelines running when messages are bad or downstream systems fail18 min
-
Consumer Scaling and Lag: Parallel Processing, Pause and Resume, and Multi-Threading Patterns Process more events per second safely by scaling consumers and partitions16 min
Internals & Reliability
-
Replication in Depth: Leaders, ISR, High Watermark and Failure Scenarios See exactly what happens to your data when brokers fail, and how to configure for zero data loss18 min
-
Delivery Semantics and Transactions: At-Least-Once, At-Most-Once and Exactly-Once Understand what Kafka can guarantee, and how transactions make read-process-write pipelines exactly-once19 min
-
Log Storage Internals: Segments, Indexes, Compaction, Tiered Storage and Disk Planning Look inside the log directory, configure cleanup policies and plan disk capacity17 min
-
KRaft Explained: Controllers, Metadata Log, Quorum and Migrating from ZooKeeper Understand how Kafka manages cluster metadata without ZooKeeper and how to run controllers in production17 min
Cluster Administration
-
Set Up a Multi-Broker Kafka Cluster: Dedicated Controllers, Brokers and Production Layout Build a three-broker cluster with a three-node controller quorum and verify it19 min
-
Topic and Cluster Administration: Config Changes, Partition Reassignment and Rebalancing Data Change settings online, move partitions between brokers and add or remove brokers safely18 min
-
Broker Tuning and Performance Testing: Threads, Memory, Disks, Network and Benchmarks Measure your cluster, find the bottleneck and tune the right settings18 min
-
Rolling Upgrades, Maintenance and Capacity Planning for Kafka Upgrade brokers without downtime, perform routine maintenance and plan for growth17 min
Security
-
Kafka Security Part 1: TLS Encryption and SASL Authentication Encrypt traffic and make clients prove who they are, with working configuration examples19 min
-
Kafka Security Part 2: ACLs, Authorization, Quotas and Hardening Checklist Give each application only the permissions it needs, and harden the cluster17 min
Monitoring & Operations
-
Monitoring Kafka: JMX Metrics, Prometheus, Grafana and Consumer Lag Alerts Know which metrics matter, collect them and build alerts that catch problems early18 min
-
Troubleshooting Kafka: Common Problems, Diagnosis Steps and Fixes A practical guide to find and fix the issues you will meet in real Kafka systems19 min
-
Multi-Cluster Kafka: MirrorMaker 2, Disaster Recovery and Data Migration Replicate topics between clusters for DR, geo distribution and migrations17 min
Kafka Connect
-
Kafka Connect Basics: Source and Sink Connectors, Workers and Tasks Move data in and out of Kafka without writing producer or consumer code17 min
-
Kafka Connect in Practice: JDBC, S3 Sink, Converters, SMTs and Dead Letter Queues Real connector configurations, data formats, transformations and error handling19 min
-
Change Data Capture with Debezium: Stream Database Changes into Kafka Capture every insert, update and delete from your database as Kafka events18 min
Kafka Streams & ksqlDB
-
Kafka Streams Basics: KStream, KTable, Topology and Your First Streams App Process data in Kafka with a plain Java library, no separate cluster needed18 min
-
Kafka Streams Stateful Processing: Aggregations, Windows, Joins and Exactly-Once Count, sum, window and join streams, and understand state stores and late data21 min
-
Kafka Streams Interactive Queries, Testing, Deployment and Monitoring Query state directly, test topologies without a broker and run Streams in production17 min
-
ksqlDB and Stream Processing Options: SQL on Streams, Flink and Choosing a Tool Write streaming queries in SQL and decide between ksqlDB, Kafka Streams and Flink16 min
Clients & Integration
-
Spring Boot with Kafka: KafkaTemplate, @KafkaListener, Retry Topics and Testing Build Kafka producers and consumers the Spring way, with error handling and tests20 min
-
Kafka Clients for Python, Node.js and Go with Schema Registry and Avro Use Kafka from other languages: producers, consumers and schemas that stay compatible19 min
Patterns & Cloud
-
Event-Driven Architecture Patterns with Kafka: Event Sourcing, CQRS, Saga and Outbox Design patterns that make microservices reliable when they talk through Kafka20 min
-
Running Kafka on Kubernetes and in the Cloud: Strimzi, Confluent Cloud, MSK and Sizing Choose between self-managed and managed Kafka and deploy with an operator18 min
Hands-On Projects
-
Project 1: Event-Driven Order System with Spring Boot, Kafka and Docker Compose Build three small services that talk through Kafka with retries, DLT and idempotency25 min
-
Project 2: Real-Time Database Replication Pipeline with Debezium, Kafka Connect and a Data Lake Stream changes from MySQL to Kafka, then to PostgreSQL and files, without writing code24 min
-
Project 3: Real-Time Analytics with Kafka Streams, Windowed Metrics and a Live Dashboard Process a click stream into live metrics and serve them to a web dashboard23 min
Interview Questions
-
Kafka Interview Questions for Developers: Producers, Consumers, Streams and Design Common developer interview questions with clear, short model answers22 min
-
Kafka Interview Questions for Administrators and Architects: Cluster, Operations and Design Operations, performance, security and architecture questions with practical answers22 min
-
Kafka Practice Challenges, Cheat Sheet and Learning Roadmap Hands-on exercises, a command and config cheat sheet, and where to go next17 min
🎯 Apache Kafka Interview Preparation
Top Apache Kafka interview questions organized by difficulty
Freshers / Entry Level
- Explain Broker vs Cluster in Apache Kafka with examples and production considerations. (Q1)
- Explain Topics and Partitions in Apache Kafka with examples and production considerations. (Q2)
- Explain ISR (In-Sync Replicas) in Apache Kafka with examples and production considerations. (Q3)
- Explain Leader Election in Apache Kafka with examples and production considerations. (Q4)
- Explain Producer Acknowledgment Levels in Apache Kafka with examples and production considerations. (Q5)
Experienced / Mid-Level
- Explain Batch Size and Linger.ms in Apache Kafka with examples and production considerations. (Q61)
- Explain Compression Types in Apache Kafka with examples and production considerations. (Q62)
- Explain Zookeeper Role in Apache Kafka with examples and production considerations. (Q63)
- Explain KRaft Mode in Apache Kafka with examples and production considerations. (Q64)
- Explain Kafka Monitoring in Apache Kafka with examples and production considerations. (Q65)
Senior / Lead Level
- Explain Log Compaction in Apache Kafka with examples and production considerations. (Q131)
- Explain Partition Rebalancing in Apache Kafka with examples and production considerations. (Q132)
- Explain Kafka Streams API in Apache Kafka with examples and production considerations. (Q133)
- Explain Kafka Connect in Apache Kafka with examples and production considerations. (Q134)
- Explain Schema Registry in Apache Kafka with examples and production considerations. (Q135)
Apache Kafka Tutorial FAQs
What is Apache Kafka?
Apache Kafka is a distributed event streaming platform. Companies use it to move events between services, build real-time pipelines and keep systems loosely coupled.
Is this Apache Kafka tutorial suitable for beginners?
Yes. The tutorial starts at beginner level and moves up to advanced topics, and the lessons follow a fixed order so that each one builds on the last. Prerequisites: basic programming, preferably Java, and a general idea of how back-end services work.
What should I know before starting Apache Kafka?
Basic programming, preferably Java, and a general idea of how back-end services work. Anything else you need is explained in the lessons.
How long does it take to learn Apache Kafka with this tutorial?
The tutorial has 42 lessons with around 13 hours of reading time. At one hour a day, the reading alone takes about 2 weeks. Plan extra time for practice and projects, because you learn Apache Kafka by using it.
Which tools or software do I need for Apache Kafka?
You will need Java and Maven or Gradle, Kafka (local or in Docker), and an IDE such as IntelliJ.
In what order should I learn Apache Kafka topics?
Follow the chapters in the order shown on this page. The path runs from What is Apache Kafka? Events, Topics, Brokers and Why Companies Use It, through Kafka Consumer API in Java, Kafka Security Part 1, Kafka Connect Basics, Event-Driven Architecture Patterns with Kafka, to Kafka Interview Questions for Developers. Skipping ahead is fine for revision, but the later lessons assume the earlier ones.
What is the difference between Kafka and a message queue like RabbitMQ?
A queue usually deletes a message once it is consumed. Kafka keeps events in a durable log for a set time, so many consumers can read them independently and replay them later. That makes Kafka a good fit for event streaming and data pipelines.
What projects or labs are included in the Apache Kafka tutorial?
The tutorial includes hands-on work such as Event-Driven Order System; Real-Time Database Replication Pipeline; Real-Time Analytics. Build them yourself before reading the solution.
How should I prepare for Apache Kafka interviews?
Finish the core chapters first, then practise explaining each concept out loud and writing small examples. The Apache Kafka interview questions page groups questions by difficulty, from freshers to experienced candidates.
What should I learn after finishing this Apache Kafka tutorial?
Good next steps are Kafka Streams and ksqlDB in depth, Schema Registry, Spark or Flink streaming, and cloud-managed Kafka. Choose the one that matches the job or project you are aiming for.

