Topic and Cluster Administration: Config Changes, Partition Reassignment and Rebalancing Data
Topic Creation Policy
- Disable
auto.create.topics.enable in production. A typo in a topic name should not silently create a new topic
- Create topics through code (Terraform, GitOps, AdminClient) with reviewed settings: partitions, replication factor, retention,
min.insync.replicas
- Use a naming convention, for example
domain.entity.event.v1 such as shop.orders.created.v1
- Set a
create.topic.policy.class.name to enforce minimum replication in large organisations
Dynamic Configuration (No Restart)
bash
# topic level
bin/kafka-configs.sh --bootstrap-server $BS --entity-type topics --entity-name orders --alter \
--add-config retention.ms=259200000,min.insync.replicas=2
bin/kafka-configs.sh --bootstrap-server $BS --entity-type topics --entity-name orders --describe
bin/kafka-configs.sh --bootstrap-server $BS --entity-type topics --entity-name orders --alter --delete-config retention.ms # back to default
# one broker or all brokers (cluster-wide default)
bin/kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-name 101 --alter --add-config log.cleaner.threads=2
bin/kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-default --alter --add-config log.retention.hours=72
bin/kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-name 101 --describe --all
Dynamic vs static: Some broker settings can be changed at runtime (listed as dynamic in the documentation). Others need a restart (read-only). Dynamic values are stored in the cluster metadata and survive restarts.
Partition Reassignment
Reassignment moves replicas between brokers. Typical reasons: you added brokers, a broker is being retired, disks are unbalanced, or you want to change the replication factor.
bash
# 1. list topics to move
cat > topics.json <<EOF
{"topics": [{"topic": "orders"}, {"topic": "payments"}], "version": 1}
EOF
# 2. let Kafka propose a plan for the target brokers
bin/kafka-reassign-partitions.sh --bootstrap-server $BS --topics-to-move-json-file topics.json --broker-list "101,102,103,104" --generate
# copy the "Proposed partition reassignment configuration" into plan.json (keep the "Current" output as a rollback file)
# 3. execute with a THROTTLE (bytes/sec per broker) so production traffic is not hurt
bin/kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --execute --throttle 50000000
# 4. check progress (also removes the throttle when complete)
bin/kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --verify
json
{"version":1,"partitions":[
{"topic":"orders","partition":0,"replicas":[104,101,102],"log_dirs":["any","any","any"]},
{"topic":"orders","partition":1,"replicas":[101,102,104],"log_dirs":["any","any","any"]}
]}
Always throttle and always verify: Copying terabytes without a throttle can saturate disks and the network and hurt producers and consumers. Run --verify to remove the throttle after completion, otherwise it stays active and slows normal replication catch-up.
Change the Replication Factor
Edit the replicas list in the plan: add a broker id to increase the factor, remove one to decrease. Execute with the same tool.
Add a Broker
- Install Kafka, configure a **new unique
node.id and the same controller quorum, format with the cluster id** and start
- The broker joins but holds no partitions. Existing partitions do not move by themselves
- Run reassignment (or Cruise Control rebalance) to move some partitions to the new broker, throttled
- Run preferred leader election so that the new broker serves its share of leadership
Decommission a Broker
- Generate and execute a reassignment that moves all replicas away from the broker
- Wait until it holds no partitions (
kafka-log-dirs.sh or --describe shows it nowhere)
- Stop the broker (controlled shutdown moves leaders first)
- Unregister it from the cluster (
kafka-cluster.sh unregister --id <id> where applicable) and remove it from monitoring and clients
Preferred Leader Election
bash
bin/kafka-leader-election.sh --bootstrap-server $BS --election-type preferred --all-topic-partitions
# after a broker returns, bring leaders back to their preferred replicas to balance load
Automatic Rebalancing: Cruise Control
Cruise Control is an open-source tool (from LinkedIn) that monitors broker load and computes and executes rebalance plans that respect goals (disk, network, CPU, rack awareness, replica counts). It is widely used for adding or removing brokers and for self-healing, and is included in operators such as Strimzi.
AdminClient API
java
import org.apache.kafka.clients.admin.*;
Properties p = new Properties();
p.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
try (Admin admin = Admin.create(p)) {
NewTopic t = new NewTopic("shop.orders.created.v1", 12, (short) 3)
.configs(Map.of("min.insync.replicas", "2", "retention.ms", "604800000", "compression.type", "producer"));
admin.createTopics(List.of(t)).all().get();
admin.listTopics().names().get().forEach(System.out::println);
Map<String, TopicDescription> d = admin.describeTopics(List.of("shop.orders.created.v1")).allTopicNames().get();
d.values().forEach(td -> System.out.println(td.name() + " partitions=" + td.partitions().size()));
// consumer group lag
Map<TopicPartition, OffsetAndMetadata> committed = admin.listConsumerGroupOffsets("email-service").partitionsToOffsetAndMetadata().get();
}
Infrastructure as code: Keep topics, ACLs and quotas in Git (Terraform provider, Kubernetes KafkaTopic resources, or scripts using AdminClient) so that environments stay identical and changes are reviewed.
Conclusion
Administer Kafka online: change configs dynamically, create topics through reviewed code, reassign partitions with a throttle when brokers change, run preferred leader election and consider Cruise Control for automatic balancing.
Quick Summary: kafka-configs dynamic; reassign with --throttle then --verify; add broker needs reassignment; AdminClient for automation.