What is the procedure to remove a storage node from Red Hat Gluster Storage cluster without data loss?

Solution Verified - Updated -

Environment

  • Red Hat Gluster Storage 3.1
  • Distributed-replicated volume

Issue

  • How to consolidate Red Hat Gluster Storage Cluster?
  • What is the procedure to remove a storage node from Red Hat Gluster Storage cluster without data loss?

Resolution

NOTE:This document describes the procedure to remove nodes from a cluster where only distributed-replicated volumes exists. It is advised to open a service request with Red Hat to verify the procedure before applying this in production, since any mistake in the procedure can lead to data loss.

1. If there is Geo-replication configured, Stop the geo-replication session.

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL stop

2. Gracefully remove all the bricks from the node , which is planned to remove.

NOTE: When the bricks are removed from a distributed-replicated volume, number of bricks must be multiple of replica count , and bricks to be removed from the same replica set

i) Remove bricks of a replicated set

   #gluster volume remove-brick VOLNAME node1:brick1 node2:brick2 start

ii) Check the status and wait the status to become complete before moving to next step

   # gluster volume remove-brick VOLNAME BRICK status

Example output of the above command on completion.

 Node          Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
 ---------      -----------                ----------   -----------   -----------   -----------         ------------     --------------
  xxx               8                              1.1MB             8                0                  0                      completed               0.00

iii)Once the status become "completed" , commit the change by executing below. Before committing , just make sure the brick directories are empty and all the data is migrated from the bricks that are going to be removed.

   #gluster volume remove-brick VOLNAME node1:brick1 node2:brick2 commit
  • Confirm by giving yes when prompted.
  • Repeat this for all the volumes in the nodes to be removed. Once all the bricks are removed from the nodes proceed to next step.

3. Detach the peer from the cluster.

   # gluster peer detach <node name>

NOTE: It is recommended not to remove node which was used in geo-rep session creation on Master side as well as Slave side. In case , the node from which the geo-rep session created, is removed. The session needs to be recreated using the --force option.

4. Start the Geo-replication session

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL start

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments