Commit Strategies
| Strategy | How | Semantics | Risk |
|---|---|---|---|
| Auto commit | enable.auto.commit=true, every auto.commit.interval.ms (5 s) during poll | At-least-once (mostly) | May commit records you have not finished processing if a crash or error happens, giving loss in some code paths. Avoid for important data |
| commitSync after processing | Call consumer.commitSync() after the batch | At-least-once | Slower (blocks), duplicates after a crash between processing and commit |
| commitAsync | consumer.commitAsync(callback) | At-least-once, faster | A failed async commit is not retried automatically. Use sync commit in finally / before shutdown |
| Per-record or per-partition commit | commitSync(Map<TopicPartition, OffsetAndMetadata>) | At-least-once, smaller duplicate window | More commits means more overhead |
| Commit before processing | Commit then process | At-most-once | Data loss if processing fails. Rare |
| Store offsets with the result | Save the offset in your database in the same transaction as the result | Effectively exactly-once into that database | More code, you manage offsets (seek on startup) |
offset + 1? The committed offset is the offset of the next record to read. If you processed offset 9, commit 10.
What is a Rebalance?
A rebalance redistributes partitions among the consumers of a group. It happens when a consumer joins, leaves or crashes, when it fails to poll() within max.poll.interval.ms, when it misses heartbeats beyond session.timeout.ms, or when topic partitions change.
Reducing Rebalance Pain
| Technique | How it helps |
|---|---|
Cooperative rebalancing (CooperativeStickyAssignor) | Only partitions that must move are revoked. Others keep processing (incremental rebalance) |
| New consumer group protocol (KIP-848) | A newer, server-side protocol with much faster and smoother rebalances, generally available since Kafka 4.0 (enable on the client with group.protocol=consumer and check the documentation of your version) |
Static membership (group.instance.id) | A restarting consumer with the same id gets the same partitions back without a rebalance if it returns within session.timeout.ms |
| Keep processing fast | Lower max.poll.records so one poll batch finishes well within max.poll.interval.ms |
| Tune timeouts | session.timeout.ms 30-45 s, heartbeat.interval.ms about a third of it |
| Graceful shutdown | Call close() so the consumer leaves the group immediately instead of waiting for a timeout |
ConsumerRebalanceListener
When partitions are taken away, commit what you have processed so the next owner does not repeat it. When partitions are given, load any state you need.
Diagnose a Rebalance Storm
| Signal | Likely reason |
|---|---|
Logs: Member ... has failed ... poll interval | Processing between polls takes longer than max.poll.interval.ms |
Logs: session timeout expired | Long GC pauses, CPU starvation, network issues |
| Rebalance after every rolling deployment | Use cooperative assignor, static membership or the new protocol, and slower rollouts |
| Autoscaler adds and removes pods constantly | Smooth the scaling rules, scale on lag with cooldowns |
consumer.pause(partitions), process in a worker thread while still calling poll() regularly (so you stay in the group), then resume.
Conclusion
Commit offsets after processing for at-least-once delivery and make handlers idempotent. Rebalances pause consumption, so use cooperative assignors, static membership or the new group protocol, keep processing time under the poll interval and close consumers gracefully.

