Issue

• When adding new failure domains to a CRUSH map in a cluster running prior to RHCS 1.2.3 (firefly), data movement can be seen when the CRUSH map is updated even if the newly added failure domains have no weight associated with them.
• In the example below we are adding racks 4 and 5 to an existing CRUSH map where racks 1, 2 and 3 already exist and have CRUSH weights associated with them. Racks 4 and 5 correctly have 0 weight set so updating the CRUSH map should not trigger any type of data movement at this time.
  Example from an exported CRUSH map where racks 4 and 5 were added:

root default {
    id -1       # do not change unnecessarily
    # weight 1155.999
    alg straw
    hash 0  # rjenkins1
    item rack1 weight 405.280
    item rack2 weight 375.360
    item rack3 weight 375.360
    item rack4 weight 0
    item rack5 weight 0

• After editing the CRUSH map and injecting the map with the new failure domains configured the cluster can be seen re-balancing the data. You can use 'ceph -s' to track the data movement occurring between placement groups.
• Since the newly configured racks have 0 weight adding them to the cluster should not have triggered any data movement to the cluster.
• Details on de-compiling and editing CRUSH maps can be found here: How to edit a CRUSH map