How to prevent demoted/decommissioned kafka broker from becoming leader during rebalance in cruise control ?
Issue
- How to prevent demoted/decommissioned kafka broker from becoming leader during rebalance in cruise control ?
- when we perform a rebalance with Cruise Control we have leaders movements on the fourth broker despite the configuration, it's not good for us because the fourth broker isn't reachable by the clients.
- We've found a workaround to remove leaders from the broker 4, at first we perform the rebalance, once it has done, we demote the broker 4, so the number of leaders decrease to 1 but the replicas remains the same.
- We wonder if there's a better way to do this because for a certain period of time during the operation broker 4 has leaders and we want to avoid it.
- checked the cruisecontrol.properties file, the parameter self.healing.exclude.recently.demoted.brokers was already configured to True
$ grep self config/cruisecontrol.properties
self.healing.disk.failure.enabled=false
self.healing.enabled=false
self.healing.exclude.recently.demoted.brokers=true
self.healing.exclude.recently.removed.brokers=true
self.healing.goal.violation.enabled=false
self.healing.maintenance.event.enabled=false
self.healing.metric.anomaly.enabled=false
self.healing.target.topic.replication.factor=3
self.healing.topic.anomaly.enabled=false
- I have tried again to demote broker 4 then I have triggered a rebalance and I have seen leaders movements on broker 4.
- I've also tried to trigger a rebalance with the parameter exclude_recently_demoted_brokers=true but I've got the same result
curl --location --request POST 'http://localhost:9090/kafkacruisecontrol/rebalance?dryrun=true&concurrent_leader_movements=100&replication_throttle=12500000&concurrent_partition_movements_per_broker=100&exclude_recently_demoted_brokers=true'
Environment
- Streams for Apache Kafka
- 2.9.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.