Offset Commit Strategies, Rebalancing and Static Membership

Apache Kafka 17 min min read Updated: Nov 17, 2026 Intermediate
Offset Commit Strategies, Rebalancing and Static Membership
Intermediate Topic 2 of 4

Commit Strategies

StrategyHowSemanticsRisk
Auto commitenable.auto.commit=true, every auto.commit.interval.ms (5 s) during pollAt-least-once (mostly)May commit records you have not finished processing if a crash or error happens, giving loss in some code paths. Avoid for important data
commitSync after processingCall consumer.commitSync() after the batchAt-least-onceSlower (blocks), duplicates after a crash between processing and commit
commitAsyncconsumer.commitAsync(callback)At-least-once, fasterA failed async commit is not retried automatically. Use sync commit in finally / before shutdown
Per-record or per-partition commitcommitSync(Map<TopicPartition, OffsetAndMetadata>)At-least-once, smaller duplicate windowMore commits means more overhead
Commit before processingCommit then processAt-most-onceData loss if processing fails. Rare
Store offsets with the resultSave the offset in your database in the same transaction as the resultEffectively exactly-once into that databaseMore code, you manage offsets (seek on startup)
java // commit exactly what you processed, per partition, committing offset + 1 Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>(); for (ConsumerRecord<String, String> r : records) { process(r); toCommit.put(new TopicPartition(r.topic(), r.partition()), new OffsetAndMetadata(r.offset() + 1)); } consumer.commitSync(toCommit);
Why offset + 1? The committed offset is the offset of the next record to read. If you processed offset 9, commit 10.
java // fast async commits during the loop, one safe sync commit at shutdown try { while (running) { ConsumerRecords<String, String> recs = consumer.poll(Duration.ofMillis(500)); handle(recs); consumer.commitAsync((offsets, ex) -> { if (ex != null) log.warn("async commit failed {}", offsets, ex); }); } } finally { try { consumer.commitSync(); } finally { consumer.close(); } }

What is a Rebalance?

A rebalance redistributes partitions among the consumers of a group. It happens when a consumer joins, leaves or crashes, when it fails to poll() within max.poll.interval.ms, when it misses heartbeats beyond session.timeout.ms, or when topic partitions change.

text Classic (eager) protocol: 1. Something changes (a new pod starts) 2. ALL consumers give up ALL partitions ("stop the world") 3. Group leader computes a new assignment 4. Consumers get new partitions and resume (from committed offsets) Result: pause in processing for the whole group during every deployment or scale event.

Reducing Rebalance Pain

TechniqueHow it helps
Cooperative rebalancing (CooperativeStickyAssignor)Only partitions that must move are revoked. Others keep processing (incremental rebalance)
New consumer group protocol (KIP-848)A newer, server-side protocol with much faster and smoother rebalances, generally available since Kafka 4.0 (enable on the client with group.protocol=consumer and check the documentation of your version)
Static membership (group.instance.id)A restarting consumer with the same id gets the same partitions back without a rebalance if it returns within session.timeout.ms
Keep processing fastLower max.poll.records so one poll batch finishes well within max.poll.interval.ms
Tune timeoutssession.timeout.ms 30-45 s, heartbeat.interval.ms about a third of it
Graceful shutdownCall close() so the consumer leaves the group immediately instead of waiting for a timeout
java props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, "org.apache.kafka.clients.consumer.CooperativeStickyAssignor"); props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "email-service-pod-0"); // unique and stable per instance (for example StatefulSet name) props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "45000"); props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "200"); // newer clients and brokers: use the new group protocol (verify support in your client version) // props.put("group.protocol", "consumer");

ConsumerRebalanceListener

When partitions are taken away, commit what you have processed so the next owner does not repeat it. When partitions are given, load any state you need.

java consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) { log.info("revoked {}", partitions); consumer.commitSync(currentOffsets); // flush progress for partitions we lose } @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { log.info("assigned {}", partitions); } });

Diagnose a Rebalance Storm

SignalLikely reason
Logs: Member ... has failed ... poll intervalProcessing between polls takes longer than max.poll.interval.ms
Logs: session timeout expiredLong GC pauses, CPU starvation, network issues
Rebalance after every rolling deploymentUse cooperative assignor, static membership or the new protocol, and slower rollouts
Autoscaler adds and removes pods constantlySmooth the scaling rules, scale on lag with cooldowns
bash bin/kafka-consumer-groups.sh --bootstrap-server $BS --describe --group email-service --state bin/kafka-consumer-groups.sh --bootstrap-server $BS --describe --group email-service --members --verbose
Slow handler pattern: If one record can legitimately take minutes (for example a report), call consumer.pause(partitions), process in a worker thread while still calling poll() regularly (so you stay in the group), then resume.

Conclusion

Commit offsets after processing for at-least-once delivery and make handlers idempotent. Rebalances pause consumption, so use cooperative assignors, static membership or the new group protocol, keep processing time under the poll interval and close consumers gracefully.

Quick Summary: commit after processing (offset+1); cooperative/static/new protocol; keep poll interval safe.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators