8.5. Replacing Ceph Storage Nodes

A situation might occur when a Ceph Storage node fails. In this situation, you must ensure to disable and rebalance the faulty node before removing it from the Overcloud to ensure no data loss. This procedure explains the process for replacing a Ceph Storage node.

Note

This procedure uses steps from the Red Hat Ceph Storage Administration Guide to manually remove Ceph Storage nodes. For more in-depth information about manual removal of Ceph Storage nodes, see Chapter 15. Removing OSDs (Manual) from the Red Hat Ceph Storage Administration Guide.
  1. Log into either a Controller node or a Ceph Storage node as the heat-admin user. The director's stack user has an SSH key to access the heat-admin user.
  2. List the OSD tree and find the OSDs for your node. For example, your node to remove might contain the following OSDs: 
    -2 0.09998     host overcloud-cephstorage-0
0 0.04999         osd.0                         up  1.00000          1.00000
1 0.04999         osd.1                         up  1.00000          1.00000
  3. Disable the OSDs on the Ceph Storage node. In this case, the OSD IDs are 0 and 1. 
    [heat-admin@overcloud-controller-0 ~]$ sudo ceph osd out 0
[heat-admin@overcloud-controller-0 ~]$ sudo ceph osd out 1
    The Ceph Storage cluster begins rebalancing. Wait for this process to complete. You can follow the status using the following command: 
    [heat-admin@overcloud-controller-0 ~]$ sudo ceph -w
  4. Once the Ceph cluster completes rebalancing, log into the faulty Ceph Storage node as the heat-admin user and stop the node. 
    [heat-admin@overcloud-cephstorage-0 ~]$ sudo /etc/init.d/ceph stop osd.0
[heat-admin@overcloud-cephstorage-0 ~]$ sudo /etc/init.d/ceph stop osd.1
  5. Remove the Ceph Storage node from the CRUSH map so that it no longer receives data. 
    [heat-admin@overcloud-cephstorage-0 ~]$ sudo ceph osd crush remove osd.0
[heat-admin@overcloud-cephstorage-0 ~]$ sudo ceph osd crush remove osd.1
  6. Remove the OSD authentication key. 
    [heat-admin@overcloud-cephstorage-0 ~]$ sudo ceph auth del osd.0
[heat-admin@overcloud-cephstorage-0 ~]$ sudo ceph auth del osd.1
  7. Remove the OSD from the cluster. 
    [heat-admin@overcloud-cephstorage-0 ~]$ sudo ceph osd rm 0
[heat-admin@overcloud-cephstorage-0 ~]$ sudo ceph osd rm 1
  8. Leave the node and return to the director host as the stack user. 
    [heat-admin@overcloud-cephstorage-0 ~]$ exit
[stack@director ~]$
  9. Disable the Ceph Storage node so the director does not reprovision it. 
    [stack@director ~]$ ironic node-list
[stack@director ~]$ ironic node-set-maintenance [UUID] true
  10. Removing a Ceph Storage node requires an update to the overcloud stack in the director using the local template files. First identify the UUID of the Overcloud stack: 
    $ heat stack-list
    Identify the UUIDs of the Ceph Storage node to delete: 
    $ nova list
    Run the following command to delete the nodes from the stack and update the plan accordingly: 
    $ openstack overcloud node delete --stack [STACK_UUID] --templates -e [ENVIRONMENT_FILE] [NODE1_UUID] [NODE2_UUID] [NODE3_UUID]

    Important

    If you passed any extra environment files when you created the Overcloud, pass them again here using the -e or --environment-file option to avoid making undesired changes to the Overcloud.
    Wait until the stack completes its update. Monitor the stack update using the heat stack-list --show-nested.
  11. Follow the procedure in Section 8.1, “Adding Compute or Ceph Storage Nodes” to add new nodes to the director's node pool and deploy them as Ceph Storage nodes. Use the --ceph-storage-scale to define the total number of Ceph Storage nodes in the Overcloud. For example, if you removed a faulty node from a three node cluster and you want to replace it, use --ceph-storage-scale 3 to return the number of Ceph Storage nodes to its original value: 
    $ openstack overcloud deploy --templates --ceph-storage-scale 3 -e [ENVIRONMENT_FILES]

    Important

    If you passed any extra environment files when you created the Overcloud, pass them again here using the -e or --environment-file option to avoid making undesired changes to the Overcloud.
    The director provisions the new node and updates the entire stack with the new node's details
  12. Log into a Controller node as the heat-admin user and check the status of the Ceph Storage node. For example: 
    [heat-admin@overcloud-controller-0 ~]$ sudo ceph status
    Confirm that the value in the osdmap section matches the number of desired nodes in your cluster.
The failed Ceph Storage node has now been replaced with a new node.