Topic and Cluster Administration: Config Changes, Partition Reassignment and Rebalancing Data

Apache Kafka 18 min min read Updated: Nov 17, 2026 Advanced
Topic and Cluster Administration: Config Changes, Partition Reassignment and Rebalancing Data
Advanced Topic 2 of 4

Topic Creation Policy

  • Disable auto.create.topics.enable in production. A typo in a topic name should not silently create a new topic
  • Create topics through code (Terraform, GitOps, AdminClient) with reviewed settings: partitions, replication factor, retention, min.insync.replicas
  • Use a naming convention, for example domain.entity.event.v1 such as shop.orders.created.v1
  • Set a create.topic.policy.class.name to enforce minimum replication in large organisations

Dynamic Configuration (No Restart)

bash # topic level bin/kafka-configs.sh --bootstrap-server $BS --entity-type topics --entity-name orders --alter \ --add-config retention.ms=259200000,min.insync.replicas=2 bin/kafka-configs.sh --bootstrap-server $BS --entity-type topics --entity-name orders --describe bin/kafka-configs.sh --bootstrap-server $BS --entity-type topics --entity-name orders --alter --delete-config retention.ms # back to default # one broker or all brokers (cluster-wide default) bin/kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-name 101 --alter --add-config log.cleaner.threads=2 bin/kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-default --alter --add-config log.retention.hours=72 bin/kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-name 101 --describe --all
Dynamic vs static: Some broker settings can be changed at runtime (listed as dynamic in the documentation). Others need a restart (read-only). Dynamic values are stored in the cluster metadata and survive restarts.

Partition Reassignment

Reassignment moves replicas between brokers. Typical reasons: you added brokers, a broker is being retired, disks are unbalanced, or you want to change the replication factor.

bash # 1. list topics to move cat > topics.json <<EOF {"topics": [{"topic": "orders"}, {"topic": "payments"}], "version": 1} EOF # 2. let Kafka propose a plan for the target brokers bin/kafka-reassign-partitions.sh --bootstrap-server $BS --topics-to-move-json-file topics.json --broker-list "101,102,103,104" --generate # copy the "Proposed partition reassignment configuration" into plan.json (keep the "Current" output as a rollback file) # 3. execute with a THROTTLE (bytes/sec per broker) so production traffic is not hurt bin/kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --execute --throttle 50000000 # 4. check progress (also removes the throttle when complete) bin/kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --verify
json {"version":1,"partitions":[ {"topic":"orders","partition":0,"replicas":[104,101,102],"log_dirs":["any","any","any"]}, {"topic":"orders","partition":1,"replicas":[101,102,104],"log_dirs":["any","any","any"]} ]}
Always throttle and always verify: Copying terabytes without a throttle can saturate disks and the network and hurt producers and consumers. Run --verify to remove the throttle after completion, otherwise it stays active and slows normal replication catch-up.

Change the Replication Factor

Edit the replicas list in the plan: add a broker id to increase the factor, remove one to decrease. Execute with the same tool.

Add a Broker

  1. Install Kafka, configure a **new unique node.id and the same controller quorum, format with the cluster id** and start
  2. The broker joins but holds no partitions. Existing partitions do not move by themselves
  3. Run reassignment (or Cruise Control rebalance) to move some partitions to the new broker, throttled
  4. Run preferred leader election so that the new broker serves its share of leadership

Decommission a Broker

  1. Generate and execute a reassignment that moves all replicas away from the broker
  2. Wait until it holds no partitions (kafka-log-dirs.sh or --describe shows it nowhere)
  3. Stop the broker (controlled shutdown moves leaders first)
  4. Unregister it from the cluster (kafka-cluster.sh unregister --id <id> where applicable) and remove it from monitoring and clients

Preferred Leader Election

bash bin/kafka-leader-election.sh --bootstrap-server $BS --election-type preferred --all-topic-partitions # after a broker returns, bring leaders back to their preferred replicas to balance load

Automatic Rebalancing: Cruise Control

Cruise Control is an open-source tool (from LinkedIn) that monitors broker load and computes and executes rebalance plans that respect goals (disk, network, CPU, rack awareness, replica counts). It is widely used for adding or removing brokers and for self-healing, and is included in operators such as Strimzi.

AdminClient API

java import org.apache.kafka.clients.admin.*; Properties p = new Properties(); p.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); try (Admin admin = Admin.create(p)) { NewTopic t = new NewTopic("shop.orders.created.v1", 12, (short) 3) .configs(Map.of("min.insync.replicas", "2", "retention.ms", "604800000", "compression.type", "producer")); admin.createTopics(List.of(t)).all().get(); admin.listTopics().names().get().forEach(System.out::println); Map<String, TopicDescription> d = admin.describeTopics(List.of("shop.orders.created.v1")).allTopicNames().get(); d.values().forEach(td -> System.out.println(td.name() + " partitions=" + td.partitions().size())); // consumer group lag Map<TopicPartition, OffsetAndMetadata> committed = admin.listConsumerGroupOffsets("email-service").partitionsToOffsetAndMetadata().get(); }
Infrastructure as code: Keep topics, ACLs and quotas in Git (Terraform provider, Kubernetes KafkaTopic resources, or scripts using AdminClient) so that environments stay identical and changes are reviewed.

Conclusion

Administer Kafka online: change configs dynamically, create topics through reviewed code, reassign partitions with a throttle when brokers change, run preferred leader election and consider Cruise Control for automatic balancing.

Quick Summary: kafka-configs dynamic; reassign with --throttle then --verify; add broker needs reassignment; AdminClient for automation.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators