Inside a Partition Directory
The file name is the base offset of the segment (the first offset it contains). To find offset 915000, Kafka picks the segment with the largest base offset not above it, uses the sparse index to jump near the position, and scans a little.
When Segments Roll and Are Deleted
| Setting | Meaning |
|---|---|
segment.bytes | Roll a new segment at this size (default 1 GB) |
segment.ms | Roll after this time even if small (default 7 days) |
retention.ms, retention.bytes | A segment is deleted when all its records are older than retention, or the partition is over the size limit. The active segment is never deleted, so very quiet topics may keep old data longer |
log.retention.check.interval.ms | How often the cleaner checks (5 minutes) |
segment.ms (for example 1 hour) so retention and compaction can act on closed segments.
Log Compaction Details
A compacted topic keeps the latest record per key. The log cleaner thread works on closed segments in the background:
| Setting | Meaning | Default |
|---|---|---|
cleanup.policy=compact (or compact,delete) | Enable compaction (and also time-based deletion if both) | delete |
min.cleanable.dirty.ratio | Start cleaning when dirty part is this fraction of the log | 0.5 |
min.compaction.lag.ms | Do not compact records younger than this (lets consumers see recent history) | 0 |
max.compaction.lag.ms | Force compaction after this time | unlimited |
delete.retention.ms | How long tombstones (null values) are kept so slow consumers see the delete | 1 day |
Tiered Storage
Tiered storage keeps recent data on fast local disks and moves older closed segments to cheap remote storage (object storage). Consumers can still read old data through the broker. It lets you keep very long histories without huge local disks. It is available in recent Kafka versions with a pluggable remote storage manager; check the documentation of your version and your distribution for maturity and limits.
| Setting | Meaning |
|---|---|
remote.log.storage.system.enable (broker) | Enable the feature |
remote.storage.enable (topic) | Tier this topic |
local.retention.ms / local.retention.bytes | How much stays on local disk |
retention.ms | Total retention including remote storage |
Disk Layout and JBOD
log.dirs=/data1,/data2,/data3spreads partitions over several disks (JBOD: just a bunch of disks). Each partition lives on one disk- If a disk fails, the broker can continue with the others and re-replicate from other brokers (check your version's JBOD failure handling)
- Alternative: one RAID volume per broker. Simpler failure handling but different cost and performance
- Use XFS or ext4, mount with
noatime. Prefer local SSD or NVMe for busy clusters, and avoid network file systems
Capacity Planning Formula
Conclusion
Each partition is a folder of segment files with sparse indexes. Retention deletes whole segments, compaction keeps the latest value per key and uses tombstones for deletes, and tiered storage can move old segments to object storage. Plan disk from ingest rate, retention and replication.

