How to prevent demoted/decommissioned kafka broker from becoming leader during rebalance in cruise control ?

Solution Verified - Updated -

Issue

  • How to prevent demoted/decommissioned kafka broker from becoming leader during rebalance in cruise control ?
  • when we perform a rebalance with Cruise Control we have leaders movements on the fourth broker despite the configuration, it's not good for us because the fourth broker isn't reachable by the clients.
  • We've found a workaround to remove leaders from the broker 4, at first we perform the rebalance, once it has done, we demote the broker 4, so the number of leaders decrease to 1 but the replicas remains the same.
  • We wonder if there's a better way to do this because for a certain period of time during the operation broker 4 has leaders and we want to avoid it.
  • checked the cruisecontrol.properties file, the parameter self.healing.exclude.recently.demoted.brokers was already configured to True
$ grep self config/cruisecontrol.properties 
self.healing.disk.failure.enabled=false
self.healing.enabled=false
self.healing.exclude.recently.demoted.brokers=true
self.healing.exclude.recently.removed.brokers=true
self.healing.goal.violation.enabled=false
self.healing.maintenance.event.enabled=false
self.healing.metric.anomaly.enabled=false
self.healing.target.topic.replication.factor=3
self.healing.topic.anomaly.enabled=false
  • I have tried again to demote broker 4 then I have triggered a rebalance and I have seen leaders movements on broker 4.
  • I've also tried to trigger a rebalance with the parameter exclude_recently_demoted_brokers=true but I've got the same result
curl --location --request POST 'http://localhost:9090/kafkacruisecontrol/rebalance?dryrun=true&concurrent_leader_movements=100&replication_throttle=12500000&concurrent_partition_movements_per_broker=100&exclude_recently_demoted_brokers=true'

Environment

  • Streams for Apache Kafka
    • 2.9.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content