Log Storage Internals: Segments, Indexes, Compaction, Tiered Storage and Disk Planning

Apache Kafka 17 min min read Updated: Nov 17, 2026 Advanced
Log Storage Internals: Segments, Indexes, Compaction, Tiered Storage and Disk Planning
Advanced Topic 3 of 4

Inside a Partition Directory

bash ls -lh /var/lib/kafka/data/orders-0/ # 00000000000000000000.log 1.0G records of the first segment # 00000000000000000000.index 10M sparse offset index (offset -> position in .log) # 00000000000000000000.timeindex 10M sparse timestamp index (time -> offset) # 00000000000000912345.log 312M the active segment (still being written) # 00000000000000912345.snapshot producer state for idempotence and transactions # leader-epoch-checkpoint leader epoch history # partition.metadata

The file name is the base offset of the segment (the first offset it contains). To find offset 915000, Kafka picks the segment with the largest base offset not above it, uses the sparse index to jump near the position, and scans a little.

bash bin/kafka-dump-log.sh --files /var/lib/kafka/data/orders-0/00000000000000000000.log --print-data-log | head # baseOffset: 0 lastOffset: 4 count: 5 ... compresscodec: lz4 ... # | offset: 0 CreateTime: 1731750001000 keySize: 2 valueSize: 31 key: c1 payload: {"order":1001}

When Segments Roll and Are Deleted

SettingMeaning
segment.bytesRoll a new segment at this size (default 1 GB)
segment.msRoll after this time even if small (default 7 days)
retention.ms, retention.bytesA segment is deleted when all its records are older than retention, or the partition is over the size limit. The active segment is never deleted, so very quiet topics may keep old data longer
log.retention.check.interval.msHow often the cleaner checks (5 minutes)
Small segments help compaction and retention accuracy: On low-volume topics lower segment.ms (for example 1 hour) so retention and compaction can act on closed segments.

Log Compaction Details

A compacted topic keeps the latest record per key. The log cleaner thread works on closed segments in the background:

text Log = [ clean part (already compacted) ][ dirty part (new records) ][ active segment (never touched) ] Cleaner builds a map key -> latest offset in the dirty part, then rewrites segments keeping only the latest record per key.
SettingMeaningDefault
cleanup.policy=compact (or compact,delete)Enable compaction (and also time-based deletion if both)delete
min.cleanable.dirty.ratioStart cleaning when dirty part is this fraction of the log0.5
min.compaction.lag.msDo not compact records younger than this (lets consumers see recent history)0
max.compaction.lag.msForce compaction after this timeunlimited
delete.retention.msHow long tombstones (null values) are kept so slow consumers see the delete1 day
bash bin/kafka-topics.sh --bootstrap-server $BS --create --topic customer-profile --partitions 6 --replication-factor 3 \ --config cleanup.policy=compact --config min.cleanable.dirty.ratio=0.1 --config segment.ms=3600000 --config delete.retention.ms=86400000
java // update and delete in a compacted topic producer.send(new ProducerRecord<>("customer-profile", "c1", "{\"name\":\"Asha\",\"city\":\"Pune\"}")); producer.send(new ProducerRecord<>("customer-profile", "c1", "{\"name\":\"Asha\",\"city\":\"Mumbai\"}")); // replaces the old value (after compaction) producer.send(new ProducerRecord<>("customer-profile", "c1", null)); // TOMBSTONE: delete key c1 (value is null)
Compaction is eventual: Before the cleaner runs, old values still exist and consumers will read them. Treat the topic as a changelog: apply every record in order, and treat a null value as a delete.

Tiered Storage

Tiered storage keeps recent data on fast local disks and moves older closed segments to cheap remote storage (object storage). Consumers can still read old data through the broker. It lets you keep very long histories without huge local disks. It is available in recent Kafka versions with a pluggable remote storage manager; check the documentation of your version and your distribution for maturity and limits.

SettingMeaning
remote.log.storage.system.enable (broker)Enable the feature
remote.storage.enable (topic)Tier this topic
local.retention.ms / local.retention.bytesHow much stays on local disk
retention.msTotal retention including remote storage

Disk Layout and JBOD

  • log.dirs=/data1,/data2,/data3 spreads partitions over several disks (JBOD: just a bunch of disks). Each partition lives on one disk
  • If a disk fails, the broker can continue with the others and re-replicate from other brokers (check your version's JBOD failure handling)
  • Alternative: one RAID volume per broker. Simpler failure handling but different cost and performance
  • Use XFS or ext4, mount with noatime. Prefer local SSD or NVMe for busy clusters, and avoid network file systems

Capacity Planning Formula

text Disk needed per cluster = (average ingest in MB/s) x (retention in seconds) x (replication factor) x (1 + overhead) / (compression ratio) Example: 20 MB/s ingest, 7 days retention, RF 3, compression 2.5x, 30% headroom 20 x 604,800 = 12,096,000 MB = 12.1 TB raw x 3 (replicas) = 36.3 TB, / 2.5 (compression, already counted if measured after compression) = 14.5 TB x 1.3 headroom = 18.9 TB total across the cluster -> with 6 brokers about 3.2 TB each, so choose 4 TB disks per broker
Keep disks under 60 to 70 percent full: Kafka needs room for new segments, compaction and for rebalancing partitions after a failure.

Conclusion

Each partition is a folder of segment files with sparse indexes. Retention deletes whole segments, compaction keeps the latest value per key and uses tombstones for deletes, and tiered storage can move old segments to object storage. Plan disk from ingest rate, retention and replication.

Quick Summary: segments + sparse indexes; retention by whole segment; compact = latest per key + tombstones; plan disk with RF.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators