acks: How Sure Do You Want to Be?
| acks | Producer waits for | Risk | Use |
|---|---|---|---|
0 | Nothing | Message can be lost without notice | Rare, very loss-tolerant metrics |
1 | Leader has written it | Lost if the leader fails before followers copy it | Moderate importance, lower latency |
all (or -1) | Leader and all in-sync replicas have it | Lowest risk (together with min.insync.replicas=2) | Default recommendation for important data |
acks=all and idempotence enabled. Older clients and other language clients may differ, so set them explicitly.
Retries and Timeouts
Network glitches and leader changes cause temporary errors. The producer retries automatically. You normally control the total time you are willing to spend with delivery.timeout.ms (default 2 minutes) rather than the number of retries.
| Setting | Meaning | Default |
|---|---|---|
retries | Maximum retry attempts | Very large (limited by the delivery timeout) |
delivery.timeout.ms | Upper bound for send() to finish (success or failure), including retries | 120000 |
request.timeout.ms | Time to wait for one broker response | 30000 |
retry.backoff.ms | Wait between retries | 100 |
max.block.ms | How long send() blocks if the buffer is full or metadata is missing | 60000 |
The Duplicate and Ordering Problem
Suppose the broker writes a record but the acknowledgement is lost on the network. The producer retries, and now the record exists twice. If several batches are in flight, a retry can also change the order.
Idempotent Producer
Batching: Throughput vs Latency
The producer groups records for the same partition into a batch before sending. Bigger batches mean fewer network requests, better compression and higher throughput.
| Setting | Meaning | Default |
|---|---|---|
batch.size | Maximum size of a batch in bytes | 16384 (16 KB) |
linger.ms | How long to wait for more records before sending a non-full batch | 0 (in newer versions often 5) |
buffer.memory | Total memory for records waiting to be sent | 32 MB |
compression.type | none, gzip, snappy, lz4, zstd | none |
Compression
| Codec | Speed | Ratio | Note |
|---|---|---|---|
lz4 | Very fast | Medium | Good default for throughput |
snappy | Fast | Medium | Popular, similar to lz4 |
zstd | Fast | Best | Good balance, recommended when CPU allows |
gzip | Slow | Good | High CPU, rarely best |
The producer compresses each batch. The broker stores it compressed and consumers decompress it. Compression helps most with text formats such as JSON, and with larger batches.
Recommended Starting Configurations
| Goal | Settings |
|---|---|
| Safe (default for business data) | acks=all, enable.idempotence=true, topic replication.factor=3, min.insync.replicas=2, delivery.timeout.ms=120000 |
| High throughput | Safe settings plus linger.ms=10-50, batch.size=64-256KB, compression.type=lz4 or zstd, bigger buffer.memory |
| Low latency | linger.ms=0, acks=1 only if loss is acceptable, no compression, small batches |
| Strict ordering of all events of one key | A single key per partition ordering is automatic with idempotence. Avoid changing partition counts |
Handling Failures in the Application
send fails after all retries and you ignore the callback, the event is lost. For critical events use the outbox pattern (write the event to your database in the same transaction, and a separate process publishes it).
Conclusion
Use acks=all with idempotence and a topic replication factor of 3 with min.insync.replicas=2 for safe producers. Tune throughput with linger.ms, batch.size and compression, and handle send failures explicitly.

