Producer Reliability and Performance: acks, retries, Idempotence, Batching and Compression

Apache Kafka 18 min min read Updated: Nov 17, 2026 Intermediate
Producer Reliability and Performance: acks, retries, Idempotence, Batching and Compression
Intermediate Topic 2 of 4

acks: How Sure Do You Want to Be?

acksProducer waits forRiskUse
0NothingMessage can be lost without noticeRare, very loss-tolerant metrics
1Leader has written itLost if the leader fails before followers copy itModerate importance, lower latency
all (or -1)Leader and all in-sync replicas have itLowest risk (together with min.insync.replicas=2)Default recommendation for important data
Defaults changed: Since Kafka 3.0 the Java producer defaults to acks=all and idempotence enabled. Older clients and other language clients may differ, so set them explicitly.

Retries and Timeouts

Network glitches and leader changes cause temporary errors. The producer retries automatically. You normally control the total time you are willing to spend with delivery.timeout.ms (default 2 minutes) rather than the number of retries.

SettingMeaningDefault
retriesMaximum retry attemptsVery large (limited by the delivery timeout)
delivery.timeout.msUpper bound for send() to finish (success or failure), including retries120000
request.timeout.msTime to wait for one broker response30000
retry.backoff.msWait between retries100
max.block.msHow long send() blocks if the buffer is full or metadata is missing60000

The Duplicate and Ordering Problem

Suppose the broker writes a record but the acknowledgement is lost on the network. The producer retries, and now the record exists twice. If several batches are in flight, a retry can also change the order.

text Producer -> send batch A (seq 5) -> broker writes it -> ack lost Producer -> retries batch A (seq 5) Without idempotence: duplicate A With idempotence: broker sees producer id + partition + sequence 5 already exists -> ignores the duplicate

Idempotent Producer

java props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // no duplicates from retries, keeps order per partition props.put(ProducerConfig.ACKS_CONFIG, "all"); // required with idempotence props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"); // up to 5 allowed with idempotence, order is still preserved
Always on for important data: Idempotence is free in practice and removes duplicates caused by the producer's own retries. It does not stop duplicates caused by your application sending the same business event twice (for that, see transactions and consumer-side deduplication).

Batching: Throughput vs Latency

The producer groups records for the same partition into a batch before sending. Bigger batches mean fewer network requests, better compression and higher throughput.

SettingMeaningDefault
batch.sizeMaximum size of a batch in bytes16384 (16 KB)
linger.msHow long to wait for more records before sending a non-full batch0 (in newer versions often 5)
buffer.memoryTotal memory for records waiting to be sent32 MB
compression.typenone, gzip, snappy, lz4, zstdnone
java // high throughput profile props.put(ProducerConfig.LINGER_MS_CONFIG, "20"); props.put(ProducerConfig.BATCH_SIZE_CONFIG, String.valueOf(128 * 1024)); props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // or zstd for better ratio // low latency profile props.put(ProducerConfig.LINGER_MS_CONFIG, "0"); props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "none");

Compression

CodecSpeedRatioNote
lz4Very fastMediumGood default for throughput
snappyFastMediumPopular, similar to lz4
zstdFastBestGood balance, recommended when CPU allows
gzipSlowGoodHigh CPU, rarely best

The producer compresses each batch. The broker stores it compressed and consumers decompress it. Compression helps most with text formats such as JSON, and with larger batches.

Recommended Starting Configurations

GoalSettings
Safe (default for business data)acks=all, enable.idempotence=true, topic replication.factor=3, min.insync.replicas=2, delivery.timeout.ms=120000
High throughputSafe settings plus linger.ms=10-50, batch.size=64-256KB, compression.type=lz4 or zstd, bigger buffer.memory
Low latencylinger.ms=0, acks=1 only if loss is acceptable, no compression, small batches
Strict ordering of all events of one keyA single key per partition ordering is automatic with idempotence. Avoid changing partition counts

Handling Failures in the Application

java producer.send(record, (md, ex) -> { if (ex == null) return; if (ex instanceof org.apache.kafka.common.errors.RetriableException) { // the client already retried until delivery.timeout.ms. Decide: store in an outbox table / retry queue } else { // non-retriable (serialization, record too large, authorization): log, alert and store the event for review } });
Do not swallow errors: If send fails after all retries and you ignore the callback, the event is lost. For critical events use the outbox pattern (write the event to your database in the same transaction, and a separate process publishes it).

Conclusion

Use acks=all with idempotence and a topic replication factor of 3 with min.insync.replicas=2 for safe producers. Tune throughput with linger.ms, batch.size and compression, and handle send failures explicitly.

Quick Summary: acks=all + idempotence; delivery.timeout.ms; linger.ms + batch.size + lz4/zstd; handle callback errors.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators