Chapter 13. Replacing a Gluster Storage Host

If a Red Hat Gluster Storage host needs to be replaced, there are two options for the replacement host:

  1. Replace the host with a new host that has a different fully-qualified domain name by following the instructions in Section 13.1, “Replacing a Gluster Storage Host (Different FQDN)”.
  2. Replace the host with a new host that has the same fully-qualified domain name by following the instructions in Section 13.2, “Replacing a Gluster Storage Node (Same FQDN)”.

Follow the instructions in whichever section is appropriate for your deployment.

13.1. Replacing a Gluster Storage Host (Different FQDN)

Important

When self-signed encryption is enabled, replacing a node is a disruptive process that requires virtual machines and the Hosted Engine to be shut down.

  1. Install the replacement host

    Follow the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization for Virtualization to install the physical machine.

    1. Installing host physical machines
    2. Configuring Public Key based SSH Authentication

  2. Stop any existing geo-replication sessions 

    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL stop

    For further information, see the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-starting_geo-replication#Stopping_a_Geo-replication_Session.

  3. Move the host to be replaced into Maintenance mode

    Perform the following steps in Red Hat Virtualization Manager:

    1. Click Compute → Hosts and select the Red Hat Gluster Storage host in the results list.
    2. Click Management → Maintenance and click OK to move the host to Maintenance mode.

  4. Prepare the replacement host

    1. Configure key-based SSH authentication without a password

      Configure key-based SSH authentication without a password from a physical machine still in the cluster to the replacement host. For details, see https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infrastructure_for_virtualization/1.5/html/deploying_red_hat_hyperconverged_infrastructure_for_virtualization/task-configure-key-based-ssh-auth.

    2. Prepare the replacement host

      Create a file called replace_host_prep.conf based on the template provided in Section B.2, “Example gdeploy configuration file for preparing a replacement host”.

      From a host with gdeploy installed (usually the host that hosts the Hosted Engine), run gdeploy using the new configuration file: 

      # gdeploy -c replace_host_prep.conf

  5. Create replacement brick directories

    Ensure the new directories are owned by the vdsm user and the kvm group. 

    # mkdir /gluster_bricks/engine/engine
# chmod vdsm:kvm /gluster_bricks/engine/engine
# mkdir /gluster_bricks/data/data
# chmod vdsm:kvm /gluster_bricks/data/data
# mkdir /gluster_bricks/vmstore/vmstore
# chmod vdsm:kvm /gluster_bricks/vmstore/vmstore

  6. (Optional) If encryption is enabled

    1. Generate the private key and self-signed certificate on the new server using the steps in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Prereqs.

      If encryption using a Certificate Authority is enabled, follow the steps under Expanding Volumes in the Network Encryption chapter of the Red Hat Gluster Storage 3.4 Administration Guide.

    2. Add the new host’s certificate to existing certificates.

      1. On a healthy host, make a backup copy of the /etc/ssl/glusterfs.ca file.
      2. Add the new host’s certificate to the /etc/ssl/glusterfs.ca file on the healthy host.
      3. Distribute the updated /etc/ssl/glusterfs.ca file to all other hosts, including the new host.

    3. Enable management encryption

      Run the following command on the new host to enable management encryption: 

      # touch /var/lib/glusterd/secure-access

    4. Include the new host in the value of the auth.ssl-allow volume option by running the following command for each volume. 

      # gluster volume set <volname> auth.ssl-allow "<old_host1>,<old_host2>,<new_host>"

    5. Restart the glusterd service on all hosts 

      # systemctl restart glusterd
    6. If encryption uses self-signed certificates, follow the steps in Section 4.1, “Configuring TLS/SSL using self-signed certificates” to remount all gluster processes.

  7. Add the new host to the existing cluster

    1. Run the following command from one of the healthy hosts: 

      # gluster peer probe <new_host>

    2. Add the new host to the existing cluster

      1. Click Compute → Hosts and then click New to open the New Host dialog.
      2. Provide a Name, Address, and Password for the new host.
      3. Uncheck the Automatically configure host firewall checkbox, as firewall rules are already configured by gdeploy.
      4. In the Hosted Engine tab of the New Host dialog, set the value of Choose hosted engine deployment action to Deploy.
      5. Click OK.
      6. When the host is available, click the name of the new host.
      7. Click the Network Interfaces subtab and then click Setup Host Networks. The Setup Host Networks dialog appears.

      8. Drag and drop the network you created for gluster to the IP associated with this host, and click OK.

        See the Red Hat Virtualization 4.2 Self-Hosted Engine Guide for further details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/self-hosted_engine_guide/chap-installing_additional_hosts_to_a_self-hosted_environment.

  8. Configure and mount shared storage on the new host 

    # cp /etc/fstab /etc/fstab.bk
# echo "<new_host>:/gluster_shared_storage /var/run/gluster/shared_storage/ glusterfs defaults 0 0" >> /etc/fstab
# mount /gluster_shared_storage

  9. Replace the old brick with the brick on the new host

    1. In Red Hat Virtualization Manager, click Storage → Volumes and select the volume.
    2. Click the Bricks subtab.
    3. Select the brick that you want to replace and click Replace Brick. The Replace Brick dialog appears.
    4. Specify the Host and the Brick Directory of the new brick.
    5. Verify that brick heal completes successfully.
  10. Click Compute → Hosts.

  11. Select the old host and click Remove.

    Use gluster peer status to verify that that the old host is no longer part of the cluster. If the old host is still present in the status output, run the following command to forcibly remove it: 

    # gluster peer detach <old_host> force

  12. Clean old host metadata. 

    # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean

  13. Set up new SSH keys for geo-replication of new brick. 

    # gluster system:: execute gsec_create

  14. Recreate geo-replication session and distribute new SSH keys. 

    # gluster volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> create push-pem force

  15. Start the geo-replication session. 

    # gluster volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> start

13.2. Replacing a Gluster Storage Node (Same FQDN)

Important

When self-signed encryption is enabled, replacing a node is a disruptive process that requires virtual machines and the Hosted Engine to be shut down.

  1. (Optional) If encryption using a Certificate Authority is enabled, follow the steps under Expanding Volumes in the Network Encryption chapter of the Red Hat Gluster Storage 3.4 Administration Guide.

  2. Move the host to be replaced into Maintenance mode

    1. In Red Hat Virtualization Manager, click Compute → Hosts and select the Red Hat Gluster Storage host.
    2. Click Management → Maintenance.
    3. Click OK to move the host to Maintenance mode.

  3. Prepare the replacement host

    Follow the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization for Virtualization to install the physical machine and configure storage on the new host.

    1. Installing host physical machines
    2. Configuring Public Key based SSH Authentication
    3. Configuring RHGS for Hosted Engine using the Cockpit UI

  4. Prepare the replacement host

    1. Create a file called replace_host_prep.conf based on the template provided in Section B.2, “Example gdeploy configuration file for preparing a replacement host”.

    2. From a host with gdeploy installed (usually the host that hosts the Hosted Engine), run gdeploy using the new configuration file: 

      # gdeploy -c replace_host_prep.conf

  5. (Optional) If encryption with self-signed certificates is enabled

    1. Generate the private key and self-signed certificate on the replacement host. See the Red Hat Gluster Storage Administration Guide for details: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Prereqs.

    2. On a healthy host, make a backup copy of the /etc/ssl/glusterfs.ca file: 

      # cp /etc/ssl/glusterfs.ca /etc/ssl/glusterfs.ca.bk
    3. Append the new host’s certificate to the content of the /etc/ssl/glusterfs.ca file.
    4. Distribute the /etc/ssl/glusterfs.ca file to all hosts in the cluster, including the new host.

    5. Run the following command on the replacement host to enable management encryption: 

      # touch /var/lib/glusterd/secure-access

  6. Replace the host machine

    Follow the instructions in the Red Hat Gluster Storage Administration Guide to replace the host: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-replacing_hosts#Replacing_a_Host_Machine_with_the_Same_Hostname.

  7. Restart the glusterd service on all hosts 

    # systemctl restart glusterd

  8. Verify that all hosts reconnect 

    # gluster peer status
  9. (Optional) If encryption uses self-signed certificates, follow the steps in Section 4.1, “Configuring TLS/SSL using self-signed certificates” to remount all gluster processes.

  10. Verify that all hosts reconnect and that brick heal completes successfully 

    # gluster peer status

  11. Refresh fingerprint

    1. In Red Hat Virtualization Manager, click Compute → Hosts and select the new host.
    2. Click Edit.
    3. Click Advanced Parameters on the General tab.
    4. Click fetch to fetch the fingerprint from the host.
    5. Click OK.
  12. Click Installation → Reinstall and provide the root password when prompted.
  13. On the Hosted Engine tab set the value of Choose hosted engine deployment action to Deploy.

  14. Attach the gluster network to the host

    1. Click Compute → Hosts and click the name of the host.
    2. Click the Network Interfaces subtab and then click Setup Host Networks.
    3. Drag and drop the newly created network to the correct interface.
    4. Ensure that the Verify connectivity between Host and Engine checkbox is checked.
    5. Ensure that the Save network configuration checkbox is checked.
    6. Click OK to save.

  15. Verify the health of the network

    Check the state of the host’s network. If the network interface enters an "Out of sync" state or does not have an IPv4 Address, click Management → Refresh Capabilities.