Chapter 12. Replacing the primary storage host

Follow this section to replace the Red Hat Gluster Storage host that initiated the peering process.

Important

When self-signed encryption is enabled, replacing a node is a disruptive process that requires virtual machines and the Hosted Engine to be shut down.

  1. (Optional) If encryption using a Certificate Authority is enabled, follow the steps under Expanding Volumes in the Network Encryption chapter of the Red Hat Gluster Storage 3.4 Administration Guide.

  2. Move the server to be replaced into Maintenance mode.

    1. In Red Hat Virtualization Manager, click Compute → Hosts and select the host to replace.
    2. Click Management → Maintenance and click OK to move the host to Maintenance mode.

  3. Install the replacement host

    Follow the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization for Virtualization to install the physical machine and configure storage on the new host.

    1. Installing virtualization hosts
    2. Configuring Public Key based SSH Authentication
    3. Configuring RHGS for Hosted Engine using the Cockpit UI

  4. Prepare the replacement host.

    1. Create a file called replace_host_prep.conf based on the template provided in Section B.2, “Example gdeploy configuration file for preparing a replacement host”.

    2. From a host with gdeploy installed (usually the server that hosts the Hosted Engine virtual machine), run gdeploy using the new configuration file: 

      # gdeploy -c replace_host_prep.conf

  5. (Optional) If encryption with self-signed certificates is enabled:

    1. Generate the private key and self-signed certificate on the replacement host. See the Red Hat Gluster Storage Administration Guide for details: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Prereqs.

    2. On a healthy host, create a copy of the /etc/ssl/glusterfs.ca file. 

      # cp /etc/ssl/glusterfs.ca /etc/ssl/glusterfs.ca.bk
    3. Append the new host’s certificate to the content of the original /etc/ssl/glusterfs.ca file.
    4. Distribute the /etc/ssl/glusterfs.ca file to all hosts in the cluster, including the new host.

    5. Run the following command on the replacement host to enable management encryption: 

      # touch /var/lib/glusterd/secure-access

    6. Include the new host in the value of the auth.ssl-allow volume option by running the following command for each volume. 

      # gluster volume set <volname> auth.ssl-allow "<old_host1>,<old_host2>,<new_host>"

    7. Restart the glusterd service on all hosts. 

      # systemctl restart glusterd
    8. Follow the steps in Section 4.1, “Configuring TLS/SSL using self-signed certificates” to remount all gluster processes.

  6. Add the replacement host to the cluster.

    Run the following command from any host already in the cluster. 

    # peer probe <new_host>

  7. Move the Hosted Engine into Maintenance mode: 

    # hosted-engine --set-maintenance --mode=global

  8. Stop the ovirt-engine service. 

    # systemctl stop ovirt-engine

  9. Update the database. 

    # hosted-engine --set-shared-config storage <new_host_IP>:/engine

  10. Start the ovirt-engine service. 

    # systemctl start ovirt-engine
  11. Stop all virtual machines except the Hosted Engine.
  12. Move all storage domains except the Hosted Engine domain into Maintenance mode.

  13. Stop the Hosted Engine virtual machine.

    Run the following command on the existing server that hosts the Hosted Engine. 

    # hosted-engine --vm-shutdown

  14. Stop high availability services on all hosts. 

    # systemctl stop ovirt-ha-agent
# systemctl stop ovirt-ha-broker

  15. Disconnect Hosted Engine storage from the virtualization host.

    Run the following command on the existing server that hosts the Hosted Engine. 

    # hosted-engine --disconnect-storage

  16. Update the Hosted Engine configuration file.

    Edit the storage parameter in the /etc/ovirt-hosted-engine/hosted-engine.conf file to use the replacement host. 

    storage=<new_server_IP>:/engine
    Note

    To configure the Hosted Engine for new hosts, use the command: 

    # hosted-engine --set-shared-config storage <new_server_IP>:/engine

  17. Restart high availability services on all hosts. 

    # systemctl restart ovirt-ha-agent
# systemctl restart ovirt-ha-broker

  18. Reboot the existing and replacement hosts.

    Wait until all hosts are available before continuing.

  19. Take the Hosted Engine out of Maintenance mode. 

    # hosted-engine --set-maintenance --mode=none

  20. Verify that the replacement host is used.

    On all virtualization hosts, verify that the engine volume is mounted from the replacement host by checking the IP address in the output of the mount command.

  21. Activate storage domains.

    Verify that storage domains mount using the IP address of the replacement host.

  22. Using the RHV Management UI, add the replacement host.

    Specify that the replacement host be used to host the Hosted Engine.

  23. Move the replacement host into Maintenance mode. 

    # hosted-engine --set-maintenance --mode=global

  24. Reboot the replacement host.

    Wait until the host is back online before continuing.

  25. Activate the replacement host from the RHV Management UI.

    Ensure that all volumes are mounted using the IP address of the replacement host.

  26. Replace engine volume brick.

    Replace the brick on the old host that belongs to the engine volume with a new brick on the replacement host.

    1. Click Storage → Volumes and select the volume.
    2. Click the Bricks subtab.
    3. Select the brick to replace, and then click Replace brick.
    4. Select the host that hosts the brick being replaced.
    5. In the Replace brick window, provide the path to the new brick.

  27. Remove the old host.

    1. Click Compute → Hosts and select the old host.
    2. Click Management → Maintenance to move the host to maintenance mode.
    3. Click Remove. The Remove Host(s) confirmation dialog appears.
    4. If there are still volume bricks on this host, or the host is non-responsive, check the Force Remove checkbox.
    5. Click OK.

    6. Detach the old host from the cluster. 

      # gluster peer detach <old_host_IP> force

  28. On the replacement host, run the following command to remove metadata from the previous host. 

    # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean