The Procedure for replacing the HW Node running mon/mgr/rgw which is using the same IP Address and Host Name before.

Solution Verified - Updated -

Environment

Red Hat Ceph Storage 4

Issue

This KCS will be used if following conditions are matched.

  • Replace a HW Node with Ceph mon, mgr, and rgw services running
  • Use the same IP address and Host Name for the Node before and after the replace.
  • Monitor will remain in quorum even after removing the needed monitor.
  • There are more than one Rados Gateway services to serve the traffic.

Resolution

Replace procedure are below (Step 1-5):
It is basically according to Operations Guide.

Prerequisites:

  • An Ansible administration node.
  • Root or sudo access to an Ansible administration node.
  • A running Red Hat Ceph Storage cluster deployed by Ansible.

Step 1. Removing a Ceph Monitor using Ansible

Check if the monitor is ok-to-stop:
ceph mon ok-to-stop MONITOR_ID

 $  ceph mon ok-to-stop <replacing-ceph-mon-node>

It is executed on admin node.

 $ cd /usr/share/ceph-ansible
 $ ansible-playbook -vvv infrastructure-playbooks/shrink-mon.yml -e mon_to_kill=<replacing-ceph-mon-node> -i hosts

Step 2. Removing a Ceph Manager using Ansible
It is executed on admin node.

 $ cd /usr/share/ceph-ansible
 $ ansible-playbook -vvv infrastructure-playbooks/shrink-mgr.yml -e mgr_to_kill=<replacing-ceph-mgr-node> -i hosts

Step 3. Removing a Rados Gateway using Ansible
It is executed on admin node.

 $ cd /usr/share/ceph-ansible
 $ ansible-playbook -vvv infrastructure-playbooks/shrink-rgw.yml -e rgw_to_kill=<replacing-ceph-rgw> -i hosts

Step 4. Replace the HW
In this case, use the same IP address and Host Name before and after replacement.

Step 5. Adding a Ceph Monitor/Manager/Rados Gateway using Ansible
It is executed on admin node.

Bare-metal deployments:

 $ cd /usr/share/ceph-ansible
 $ ansible-playbook -vvv -i hosts site.yml  --limit mons,mgrs,rgws

or

Container deployments:

 $ cd /usr/share/ceph-ansible
 $ ansible-playbook -vvv -i hosts site-container.yml --limit mons,mgrs,rgws

check the status of the storage cluster:

 $ ceph -s

There is not need to edit inventory file (hosts file), because the same IP address and Host Name are used before and after the exchange.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments