1. Run the gluster volume create command to create the replicated volume.
   The syntax is # gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
   The default value for transport is tcp. Other options can be passed such as auth.allow or auth.reject. See Section 8.1, “Configuring Volume Options” for a full list of parameters.

   Example 6.3. Replicated Volume with Two Storage Servers

   The order in which bricks are specified determines how bricks are replicated with each other. For example, every 2 bricks, where 2 is the replica count forms a replica set. If more bricks were specified, the next two bricks in sequence would replicate each other. The same is illustrated in Figure 6.2. Illustration of a Replicated Volume.

   # gluster volume create test-volume replica 2 transport tcp server1:/exp1/brick server2:/exp2/brick
   Creation of test-volume has been successful
   Please start the volume to access data.

2. Run # gluster volume start VOLNAME to start the volume.

   # gluster volume start test-volume
   Starting test-volume has been successful

3. Run gluster volume info command to optionally display the volume information.

1. Run the gluster volume create command to create the replicated volume.
   The syntax is # gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
   The default value for transport is tcp. Other options can be passed such as auth.allow or auth.reject. See Section 8.1, “Configuring Volume Options” for a full list of parameters.

   Example 6.4. Replicated Volume with Three Storage Servers

   The order in which bricks are specified determines how bricks are replicated with each other. For example, every n bricks, where n is the replica count forms a replica set. If more bricks were specified, the next three bricks in sequence would replicate each other. The same is illustrated in Figure 6.2. Illustration of a Replicated Volume.

   # gluster volume create test-volume replica 3 transport tcp server1:/exp1/brick server2:/exp2/brick server3:/exp3/brick
   Creation of test-volume has been successful
   Please start the volume to access data.

2. Run # gluster volume start VOLNAME to start the volume.

   # gluster volume start test-volume
   Starting test-volume has been successful

3. Run gluster volume info command to optionally display the volume information.

1. Run the gluster volume create command to create the distributed replicated volume.
   The syntax is # gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
   The default value for transport is tcp. Other options can be passed such as auth.allow or auth.reject. See Section 8.1, “Configuring Volume Options” for a full list of parameters.

   Example 6.5. Four Node Distributed Replicated Volume with a Two-way Replication

   The order in which bricks are specified determines how bricks are replicated with each other. For example, first 2 bricks, where 2 is the replica count. In this scenario, the first two bricks specified replicate each other. If more bricks were specified, the next two bricks in sequence would replicate each other.

   # gluster volume create test-volume replica 2 transport tcp server1:/exp1/brick server2:/exp2/brick server3:/exp3/brick server4:/exp4/brick
   Creation of test-volume has been successful
   Please start the volume to access data.

   Example 6.6. Six Node Distributed Replicated Volume with a Two-way Replication

   # gluster volume create test-volume replica 2 transport tcp server1:/exp1/brick server2:/exp2/brick server3:/exp3/brick server4:/exp4/brick server5:/exp5/brick server6:/exp6/brick
   Creation of test-volume has been successful
   Please start the volume to access data.

2. Run # gluster volume start VOLNAME to start the volume.

   # gluster volume start test-volume
   Starting test-volume has been successful

3. Run gluster volume info command to optionally display the volume information.

1. Run the gluster volume create command to create the distributed replicated volume.
   The syntax is # gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
   The default value for transport is tcp. Other options can be passed such as auth.allow or auth.reject. See Section 8.1, “Configuring Volume Options” for a full list of parameters.

   Example 6.7. Six Node Distributed Replicated Volume with a Three-way Replication

   The order in which bricks are specified determines how bricks are replicated with each other. For example, first 3 bricks, where 3 is the replica count. In this scenario, the first three bricks specified replicate each other. If more bricks were specified, the next three bricks in sequence would replicate each other.

   # gluster volume create test-volume replica 3 transport tcp server1:/exp1/brick server2:/exp2/brick server3:/exp3/brick server4:/exp4/brick server5:/exp5/brick server6:/exp6/brick
   Creation of test-volume has been successful
   Please start the volume to access data.

2. Run # gluster volume start VOLNAME to start the volume.

   # gluster volume start test-volume
   Starting test-volume has been successful

3. Run gluster volume info command to optionally display the volume information.

1. Run gluster volume set VOLNAME stat-prefetch off to disable stat-prefetch for the volume.
2. Run gluster volume set VOLNAME server.allow-insecure on to permit insecure ports.

   Note

   This allows Samba to communicate with brick processes even with untrusted ports.
3. Edit the /etc/glusterfs/glusterd.vol in each Red Hat Storage node, and add the following setting:

   option rpc-auth-allow-insecure on

   Note

   This allows Samba to communicate with glusterd even with untrusted ports.
4. Restart glusterd service on each Red Hat Server node.
5. Run the following command to verify proper lock and I/O coherency.

   gluster volume set VOLNAME storage.batch-fsync-delay-usec 0

   Note

   To enable Samba to start on boot, run the following command:

   # chkconfig smb on

1. Either disable sharing over SMB for the whole cluster as detailed in Section 7.4.1, “Sharing Volumes over SMB” or run the following command to disable automatic SMB sharing per-volume:

   # gluster volume set user.smb disable

2. Open the /etc/samba/smb.conf file in a text editor and add the following lines for a simple configuration:

   [gluster-VOLNAME]
   comment = For samba share of volume VOLNAME
   vfs objects = glusterfs
   glusterfs:volume = VOLNAME
   glusterfs:logfile = /var/log/samba/VOLNAME.log
   glusterfs:loglevel = 7
   path = /
   read only = no
   guest ok = yes

   The configuration options are described in the following table:

   Table 7.7. Configuration Options

   Configuration Options	Required?	Default Value	Description
   Path	Yes	n/a	It represents the path that is relative to the root of the gluster volume that is being shared. Hence / represents the root of the gluster volume. Exporting a subdirectory of a volume is supported and /subdir in path exports only that subdirectory of the volume.
   glusterfs:volume	Yes	n/a	The volume name that is shared.
   glusterfs:logfile	No	NULL	Path to the log file that will be used by the gluster modules that are loaded by the vfs plugin. Standard Samba variable substitutions as mentioned in smb.conf are supported.
   glusterfs:loglevel	No	7	This option is equivalent to the client-log-level option of gluster. 7 is the default value and corresponds to the INFO level.
   glusterfs:volfile_server	No	localhost	The gluster server to be contacted to fetch the volfile for the volume.

3. Run service smb [re]start to start or restart the smb service.
4. Run smbpasswd to set the SMB password.

   # smbpasswd -a username

   Specify the SMB password. This password is used during the SMB mount.

1. Add the user on all the Samba servers based on your configuration:
   # adduser username
2. Add the user to the list of Samba users on all Samba servers and assign password by executing the following command:
   # smbpasswd -a username
3. Perform a FUSE mount of the gluster volume on any one of the Samba servers and provide required permissions to the user by executing the following commands:
   # mount -t glusterfs -oacl ip address volname mountpoint
   # setfacl -muser:<username>:rwx <mountpoint>
4. Provide required permissions to the user by executing appropriate setfacl command. For example:
   # setfacl -m user:username:rwx mountpoint

Start CTDB and Verify the Configuration

1. Run # service ctdb start to start the CTDB service.
2. Run # chkconfig smb off to prevent CTDB starting Samba automatically when the server is restarted.
3. Verify that CTDB is running using the following commands:

   # ctdb status
   # ctdb ip
   # ctdb ping -n all

4. Mount a Red Hat Storage volume using any one of the VIPs.
5. Run # ctdb ip to locate the physical server serving the VIP.
6. Shut down the CTDB VIP server to verify successful configuration.
   When the Red Hat Storage Server serving the VIP is shut down there will be a pause for a few seconds, then I/O will resume.

1. Add the new bricks to the volume.

   # gluster volume add-brick VOLNAME [<stripe|replica> <COUNT>] NEW-BRICK

   Example 8.1. Adding a Brick to a Distribute Volume

   # gluster volume add-brick test-volume server5:/exp5
   Add Brick successful

2. Verify the volume information using the command:

   # gluster volume info
    Volume Name: test-volume
       Type: Distribute
       Status: Started
       Number of Bricks: 5
       Bricks:
       Brick1: server1:/exp1
       Brick2: server2:/exp2
       Brick3: server3:/exp3
       Brick4: server4:/exp4
       Brick5: server5:/exp5

   Note

   In case of a Distribute-replicate or stripe volume, you must specify the replica or stripe count in the add-brick command and provide the same number of bricks as the replica or stripe count to the add-brick command.
3. Remove the bricks to be replaced from the subvolume.
   1. Start the remove-brick operation using the command:

      # gluster volume remove-brick VOLNAME [replica <COUNT>] <BRICK> start

      Example 8.2. Start a remove-brick operation on a distribute volume

      #gluster volume remove-brick test-volume server2:/exp2 start
      Remove Brick start successful

   2. View the status of the remove-brick operation using the command:

      # gluster volume remove-brick VOLNAME [replica <COUNT>] BRICK status

      Example 8.3. View the Status of remove-brick Operation

      # gluster volume remove-brick test-volume server2:/exp2 status
      Node     Rebalanced-files size        scanned failures status
      ------------------------------------------------------------------
      server2  16               16777216    52      0        in progress

      Keep monitoring the remove-brick operation status by executing the above command. When the value of the status field is set to complete in the output of remove-brick status command, proceed further.
   3. Commit the remove-brick operation using the command:

      #gluster volume remove-brick VOLNAME [replica <COUNT>] <BRICK> commit

      Example 8.4. Commit the remove-brick Operation on a Distribute Volume

      #gluster volume remove-brick test-volume server2:/exp2 commit

   4. Verify the volume information using the command:

      # gluster volume info
      Volume Name: test-volume
      Type: Distribute
      Status: Started
      Number of Bricks: 4
      Bricks:
      Brick1: server1:/exp1
      Brick3: server3:/exp3
      Brick4: server4:/exp4
      Brick5: server5:/exp5

   5. Verify the content on the brick after committing the remove-brick operation on the volume. If there are any files leftover, copy it through FUSE or NFS mount.
      1. Verify if there are any pending files on the bricks of the subvolume.
         Along with files, all the application-specific extended attributes must be copied. glusterFS also uses extended attributes to store its internal data. The extended attributes used by glusterFS are of the form trusted.glusterfs.*, trusted.afr.*, and trusted.gfid. Any extended attributes other than ones listed above must also be copied.
         To copy the application-specific extended attributes and to achieve a an effect similar to the one that is described above, use the following shell script:
         Syntax:

         # copy.sh <glusterfs-mount-point> <brick>

         Example 8.5. Code Snippet Usage

         If the mount point is /mnt/glusterfs and brick path is /export/brick1, then the script must be run as:

         # copy.sh /mnt/glusterfs /export/brick

         #!/bin/bash
         
         MOUNT=$1
         BRICK=$2
         
         for file in `find $BRICK ! -type d`; do
             rpath=`echo $file | sed -e "s#$BRICK\(.*\)#\1#g"`
             rdir=`dirname $rpath`
         
             cp -fv $file $MOUNT/$rdir;
         
             for xattr in `getfattr -e hex -m. -d $file 2>/dev/null | sed -e '/^#/d' | grep -v -E "trusted.glusterfs.*" | grep -v -E "trusted.afr.*" | grep -v "trusted.gfid"`;
             do
                 key=`echo $xattr | cut -d"=" -f 1`
                 value=`echo $xattr | cut -d"=" -f 2`
         
                 setfattr $MOUNT/$rpath -n $key -v $value
             done
         done

      2. To identify a list of files that are in a split-brain state, execute the command:

         #gluster volume heal test-volume info

      3. If there are any files listed in the output of the above command, delete those files from the mount point and manually retain the correct copy of the file after comparing the files across the bricks in a replica set. Selecting the correct copy of the file needs manual intervention by the System Administrator.

1. Ensure that the new brick (sys5:/home/gfs/r2_5) that replaces the old brick (sys0:/home/gfs/r2_0) is empty. Ensure that all the bricks are online. The brick that must be replaced can be in an offline state.
2. Bring the brick that must be replaced to an offline state, if it is not already offline.
   1. Identify the PID of the brick to be replaced, by executing the command:

      # gluster volume status
      Status of volume: r2
      Gluster process                  Port    Online    Pid 
      ------------------------------------------------------- 
      Brick sys0:/home/gfs/r2_0         49152    Y    5342
        
      Brick sys1:/home/gfs/r2_1         49153    Y    5354 
      
      Brick sys2:/home/gfs/r2_2         49154    Y    5365
       
      Brick sys3:/home/gfs/r2_3         49155    Y    5376

   2. Log in to the host on which the brick to be replaced has its process running and kill the brick.

      #kill -9 <PID>

   3. Ensure that the brick to be replaced is offline and the other bricks are online by executing the command:

      # gluster volume status 
      Status of volume: r2 
      Gluster process                  Port    Online  Pid 
      ------------------------------------------------------
      Brick sys0:/home/gfs/r2_0         N/A      N    5342
      
      Brick sys1:/home/gfs/r2_1         49153    Y    5354
       
      Brick sys2:/home/gfs/r2_2         49154    Y    5365
      
      Brick sys3:/home/gfs/r2_3         49155    Y    5376

3. Create a FUSE mount point from any server to edit the extended attributes. Using the NFS and CIFS mount points, you will not be able to edit the extended attributes.
4. Perform the following operations to change the Automatic File Replication extended attributes so that the heal process happens from the other brick (sys1:/home/gfs/r2_1) in the replica pair to the new brick (sys5:/home/gfs/r2_5).
   Note that /mnt/r2 is the FUSE mount path.
   1. Create a new directory on the mount point and ensure that a directory with such a name is not already present.

      #mkdir /mnt/r2/<name-of-nonexistent-dir>

   2. Delete the directory and set the extended attributes.

      #rmdir /mnt/r2/<name-of-nonexistent-dir>

      #setfattr -n trusted.non-existent-key -v abc /mnt/r2 
      #setfattr -x trusted.non-existent-key /mnt/r2

   3. Ensure that the extended attributes on the other bricks in the replica (in this example, trusted.afr.r2-client-0) is not set to zero.

      #getfattr -d -m. -e hex /home/gfs/r2_1 # file: home/gfs/r2_1 
      security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 
      trusted.afr.r2-client-0=0x000000000000000300000002  
      trusted.afr.r2-client-1=0x000000000000000000000000 
      trusted.gfid=0x00000000000000000000000000000001 
      trusted.glusterfs.dht=0x0000000100000000000000007ffffffe 
      trusted.glusterfs.volume-id=0xde822e25ebd049ea83bfaa3c4be2b440

5. Execute the replace-brick command with the force option:

   # gluster volume replace-brick r2 sys0:/home/gfs/r2_0 sys5:/home/gfs/r2_5 commit force 
   volume replace-brick: success: replace-brick commit successful

6. Check if the new brick is online.

   # gluster volume status
   Status of volume: r2 
   Gluster process                    Port    Online    Pid 
   --------------------------------------------------------- 
   Brick sys5:/home/gfs/r2_5            49156    Y    5731 
   
   Brick sys1:/home/gfs/r2_1            49153    Y    5354 
   
   Brick sys2:/home/gfs/r2_2            49154    Y    5365 
   
   Brick sys3:/home/gfs/r2_3            49155    Y    5376

7. Ensure that after the self-heal completes, the extended attributes are set to zero on the other bricks in the replica.

   # getfattr -d -m. -e hex /home/gfs/r2_1 
   getfattr: Removing leading '/' from absolute path names 
   # file: home/gfs/r2_1 
   security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 
   trusted.afr.r2-client-0=0x000000000000000000000000
   trusted.afr.r2-client-1=0x000000000000000000000000 
   trusted.gfid=0x00000000000000000000000000000001 
   trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
   trusted.glusterfs.volume-id=0xde822e25ebd049ea83bfaa3c4be2b440

   Note that in this example, the extended attributes trusted.afr.r2-client-0 and trusted.afr.r2-client-1 are set to zero.

1. Replace a brick with a commit force option:

   # gluster volume replace-brick VOLNAME <BRICK> <NEW-BRICK> commit force

   Example 8.6. Replace a brick on a Distribute Volume

   # gluster volume replace-brick r2 sys0:/home/gfs/r2_0 sys5:/home/gfs/r2_5 commit force 
   volume replace-brick: success: replace-brick commit successful

2. Verify if the new brick is online.

   # gluster volume status
   Status of volume: r2 
   Gluster process                    Port    Online    Pid 
   --------------------------------------------------------- 
   Brick sys5:/home/gfs/r2_5            49156    Y    5731 
   
   Brick sys1:/home/gfs/r2_1            49153    Y    5354 
   
   Brick sys2:/home/gfs/r2_2            49154    Y    5365 
   
   Brick sys3:/home/gfs/r2_3            49155    Y    5376

1. Probe the new peer from one of the existing peers to bring it into the cluster.

   # gluster peer probe sys5.example.com

2. Ensure that the new brick (sys5.example.com:/rhs/brick1/b1) that replaces the old brick (sys0.example.com:/rhs/brick1/b1) is empty.
3. Create a FUSE mount point from any server to edit the extended attributes.

   #mount -t glusterfs server-ip:/VOLNAME mount-point

4. Perform the following operations to change the Automatic File Replication extended attributes so that the heal process happens from the other brick (sys1.example.com:/rhs/brick1/b1) in the replica pair to the new brick (sys5.example.com:/rhs/brick1/b1). Note that /mnt/r2 is the FUSE mount path.

   1. Create a new directory on the mount point and ensure that a directory with such a name is not already present.

      #mkdir /mnt/r2/<name-of-nonexistent-dir>

   2. Delete the directory and set and delete the extended attributes.

      #rmdir /mnt/r2/<name-of-nonexistent-dir>
                 #setfattr -n trusted.non-existent-key -v abc /mnt/r2
                 #setfattr -x trusted.non-existent-key /mnt/r2

   3. Ensure that the extended attributes on the other bricks in the replica (in this example, trusted.afr.vol-client-0) is not set to zero.

      #getfattr -d -m. -e hex /rhs/brick1/b1
      # file: rhs/brick1/b1
      security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 
      trusted.afr.vol-client-0=0x000000000000000300000002  
      trusted.afr.vol-client-1=0x000000000000000000000000 
      trusted.gfid=0x00000000000000000000000000000001 
      trusted.glusterfs.dht=0x0000000100000000000000007ffffffe 
      trusted.glusterfs.volume-id=0xde822e25ebd049ea83bfaa3c4be2b440

5. Retrieve the brick paths in sys0.example.com using the following command:

   #gluster volume info <VOLNAME>

    Volume Name: vol
   Type: Replicate
   Volume ID: 0xde822e25ebd049ea83bfaa3c4be2b440
   Status: Started
   Snap Volume: no
   Number of Bricks: 1 x 2 = 2
   Transport-type: tcp
   Bricks:
   Brick1: sys0.example.com:/rhs/brick1/b1
   Brick2: sys1.example.com:/rhs/brick1/b1
   Options Reconfigured:
   performance.readdir-ahead: on
   snap-max-hard-limit: 256
   snap-max-soft-limit: 90
   auto-delete: disable
   

   Brick path in sys0.example.com is /rhs/brick1/b1. This has to be replaced with the brick in the newly added host, sys5.example.com
6. Create the required brick path in sys5.example.com.For example, if /rhs/brick is the XFS mount point in sys5.example.com, then create a brick directory in that path.

   # mkdir /rhs/brick1/b1

7. Execute the replace-brick command with the force option:

   				# gluster volume replace-brick vol sys0.example.com:/rhs/brick1/b1 sys5.example.com:/rhs/brick1/b1 commit force
      volume replace-brick: success: replace-brick commit successful
   

8. Verify if the new brick is online.

   				# gluster volume status
   Status of volume: vol 
   Gluster process                                  Port    Online Pid 
   Brick sys5.example.com:/rhs/brick1/b1            49156    Y    5731 
   Brick sys1.example.com:/rhs/brick1/b1            49153    Y    5354
   

9. Initiate self-heal on the volume. The status of the heal process can be seen by executing the command:

   #gluster volume heal VOLNAME full

10. The status of the heal process can be seen by executing the command:

    # gluster volume heal VOLNAME info

11. Detach the original machine from the trusted pool.

    #gluster peer detach sys0.example.com

12. Ensure that after the self-heal completes, the extended attributes are set to zero on the other bricks in the replica.

    #getfattr -d -m. -e hex /rhs/brick1/b1
    getfattr: Removing leading '/' from absolute path names 
    #file: rhs/brick1/b1 
    security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 
    trusted.afr.vol-client-0=0x000000000000000000000000
    trusted.afr.vol-client-1=0x000000000000000000000000 
    trusted.gfid=0x00000000000000000000000000000001 
    trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
    trusted.glusterfs.volume-id=0xde822e25ebd049ea83bfaa3c4be2b440
    

    Note

    Note that in this example, the extended attributes trusted.afr.vol-client-0 and trusted.afr.vol-client-1 are set to zero.

1. Stop the glusterd service on the sys0.example.com.

   # service glusterd stop

2. Retrieve the UUID of the failed host (sys0.example.com) from another of the Red Hat Storage Trusted Storage Pool by executing the following command:

   # gluster peer status
   Number of Peers: 2
   
   Hostname: sys1.example.com
   Uuid: 1d9677dc-6159-405e-9319-ad85ec030880
   State: Peer in Cluster (Connected)
   
   Hostname: sys0.example.com
   Uuid: b5ab2ec3-5411-45fa-a30f-43bd04caf96b
   State: Peer Rejected (Connected)
   

   Note that the UUID of the failed host is b5ab2ec3-5411-45fa-a30f-43bd04caf96b
3. Edit the glusterd.info file in the new host and include the UUID of the host you retrieved in the previous step.

   # cat /var/lib/glusterd/glusterd.info
   UUID=b5ab2ec3-5411-45fa-a30f-43bd04caf96b
   operating-version=30000

4. Select any host (say for example, sys1.example.com) in the Red Hat Storage Trusted Storage Pool and retrieve its UUID from the glusterd.info file.

   # grep -i uuid /var/lib/glusterd/glusterd.info
   UUID=8cc6377d-0153-4540-b965-a4015494461c

5. Gather the peer information files from the host (sys1.example.com) in the previous step. Execute the following command in that host (sys1.example.com) of the cluster.

   # cp -a /var/lib/glusterd/peers /tmp/

6. Remove the peer file corresponding to the failed host (sys0.example.com) from the /tmp/peers directory.

   # rm /tmp/peers/b5ab2ec3-5411-45fa-a30f-43bd04caf96b

   Note that the UUID corresponds to the UUID of the failed host (sys0.example.com) retrieved in Step 2.
7. Archive all the files and copy those to the failed host(sys0.example.com).

   # cd /tmp; tar -cvf peers.tar peers

8. Copy the above created file to the new peer.

   # scp /tmp/peers.tar root@sys0.example.com:/tmp

9. Copy the extracted content to the /var/lib/glusterd/peers directory. Execute the following command in the newly added host with the same name (sys0.example.com) and IP Address.

   # tar -xvf /tmp/peers.tar
   # cp peers/* /var/lib/glusterd/peers/

10. Select any other host in the cluster other than the node (sys1.example.com) selected in step 4. Copy the peer file corresponding to the UUID of the host retrieved in Step 4 to the new host (sys0.example.com) by executing the following command:

    # scp /var/lib/glusterd/peers/<UUID-retrieved-from-step4> root@Example1:/var/lib/glusterd/peers/

11. Retrieve the brick directory information, by executing the following command in any host in the cluster.

    # gluster volume info

    Volume Name: vol
    Type: Replicate
    Volume ID: 0x8f16258c88a0498fbd53368706af7496
    Status: Started
    Snap Volume: no
    Number of Bricks: 1 x 2 = 2
    Transport-type: tcp
    Bricks:
    Brick1: sys0.example.com:/rhs/brick1/b1
    Brick2: sys1.example.com:/rhs/brick1/b1
    Options Reconfigured:
    performance.readdir-ahead: on
    snap-max-hard-limit: 256
    snap-max-soft-limit: 90
    auto-delete: disable

    In the above example, the brick path in sys0.example.com is, /rhs/brick1/b1. If the brick path does not exist in sys0.example.com, perform steps a, b, and c.
    1. Create a brick path in the host, sys0.example.com.

       mkdir /rhs/brick1/b1

    2. Retrieve the volume ID from the existing brick of another host by executing the following command on any host that contains the bricks for the volume.

       # getfattr -d -m. -ehex <brick-path>

       Copy the volume-id.

       # getfattr -d -m. -ehex /rhs/brick1/b1
       getfattr: Removing leading '/' from absolute path names
       # file: rhs/brick1/b1
       trusted.afr.vol-client-0=0x000000000000000000000000
       trusted.afr.vol-client-1=0x000000000000000000000000
       trusted.gfid=0x00000000000000000000000000000001
       trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
       trusted.glusterfs.volume-id=0x8f16258c88a0498fbd53368706af7496

       In the above example, the volume id is 0x8f16258c88a0498fbd53368706af7496
    3. Set this volume ID on the brick created in the newly added host and execute the following command on the newly added host (sys0.example.com).

       # setfattr -n trusted.glusterfs.volume-id -v <volume-id> <brick-path>

       For Example:

       # setfattr -n trusted.glusterfs.volume-id -v 0x8f16258c88a0498fbd53368706af7496 /rhs/brick2/drv2

    Data recovery is possible only if the volume type is replicate or distribute-replicate. If the volume type is plain distribute, you can skip steps 12 and 13.
12. Create a FUSE mount point to mount the glusterFS volume.

    # mount -t glusterfs <server-name>:/VOLNAME <mount>

13. Perform the following operations to change the Automatic File Replication extended attributes so that the heal process happens from the other brick (sys1.example.com:/rhs/brick1/b1) in the replica pair to the new brick (sys0.example.com:/rhs/brick1/b1). Note that /mnt/r2 is the FUSE mount path.

    1. Create a new directory on the mount point and ensure that a directory with such a name is not already present.

       #mkdir /mnt/r2/<name-of-nonexistent-dir>

    2. Delete the directory and set the extended attributes.

        #rmdir /mnt/r2/<name-of-nonexistent-dir>
       #setfattr -n trusted.non-existent-key -v abc /mnt/r2
       #setfattr -x trusted.non-existent-key /mnt/r2

    3. Ensure that the extended attributes on the other bricks in the replica (in this example, trusted.afr.vol-client-0) is not set to zero.

       #getfattr -d -m. -e hex /rhs/brick1/b1 # file: rhs/brick1/b1
       security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 
       trusted.afr.vol-client-0=0x000000000000000300000002  
       trusted.afr.vol-client-1=0x000000000000000000000000 
       trusted.gfid=0x00000000000000000000000000000001 
       trusted.glusterfs.dht=0x0000000100000000000000007ffffffe 
       trusted.glusterfs.volume-id=0x8f16258c88a0498fbd53368706af7496

14. Start the glusterd service.

    # service glusterd start

15. Perform the self-heal operation on the restored volume.

    # gluster volume heal VOLNAME full

16. You can view the gluster volume self-heal status by executing the following command:

    # gluster volume heal VOLNAME info

1. Stop the glusterd service on sys0.example.com.

   # service glusterd stop

2. Retrieve the UUID of the failed host (sys0.example.com) from another peer in the Red Hat Storage Trusted Storage Pool by executing the following command:

   # gluster peer status
   Number of Peers: 1
   
   Hostname: sys0.example.com
   Uuid: b5ab2ec3-5411-45fa-a30f-43bd04caf96b
   State: Peer Rejected (Connected)
   

   Note that the UUID of the failed host is b5ab2ec3-5411-45fa-a30f-43bd04caf96b
3. Edit the glusterd.info file in the new host (sys0.example.com) and include the UUID of the host you retrieved in the previous step.

   # cat /var/lib/glusterd/glusterd.info
   UUID=b5ab2ec3-5411-45fa-a30f-43bd04caf96b
   operating-version=30000

4. Create the peer file in the newly created host (sys0.example.com) in /var/lib/glusterd/peers/<uuid-of-other-peer> with the name of the UUID of the other host (sys1.example.com).
   UUID of the host can be obtained with the following:

   # gluster system:: uuid get

   Example 8.7. Example to obtain the UUID of a host

   For example,
   # gluster system:: uuid get
   UUID: 1d9677dc-6159-405e-9319-ad85ec030880

   In this case the UUID of other peer is 1d9677dc-6159-405e-9319-ad85ec030880
5. Create a file /var/lib/glusterd/peers/1d9677dc-6159-405e-9319-ad85ec030880 in sys0.example.com, with the following command:

   # touch /var/lib/glusterd/peers/1d9677dc-6159-405e-9319-ad85ec030880

   The file you create must contain the following information:

   UUID=<uuid-of-other-node>
   state=3
   hostname=<hostname>

6. Continue to perform steps 11 to 16 as documented in the previous procedure.

Steps to recover from a file split-brain

1. Run the following command to obtain the path of the file that is in split-brain:

   # gluster volume heal VOLNAME info split-brain

   From the command output, identify the files for which file operations performed from the client keep failing with Input/Output error.
2. Close the applications that opened split-brain file from the mount point. If you are using a virtual machine, you must power off the machine.
3. Obtain and verify the AFR changelog extended attributes of the file using the getfattr command. Then identify the type of split-brain to determine which of the bricks contains the 'good copy' of the file.

   getfattr -d -m . -e hex <file-path-on-brick>

   For example,

   # getfattr -d -e hex -m. brick-a/file.txt  
   \#file: brick-a/file.txt
   security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000
   trusted.afr.vol-client-2=0x000000000000000000000000
   trusted.afr.vol-client-3=0x000000000200000000000000
   trusted.gfid=0x307a5c9efddd4e7c96e94fd4bcdcbd1b

   The extended attributes with trusted.afr.VOLNAMEvolname-client-<subvolume-index> are used by AFR to maintain changelog of the file. The values of the trusted.afr.VOLNAMEvolname-client-<subvolume-index> are calculated by the glusterFS client (FUSE or NFS-server) processes. When the glusterFS client modifies a file or directory, the client contacts each brick and updates the changelog extended attribute according to the response of the brick.
   subvolume-index is the brick number - 1 of gluster volume info VOLNAME output.
   For example,

   # gluster volume info vol
   Volume Name: vol 
   Type: Distributed-Replicate  
   Volume ID: 4f2d7849-fbd6-40a2-b346-d13420978a01  
   Status: Created  
   Number of Bricks: 4 x 2 = 8 
   Transport-type: tcp  
   Bricks:  
   brick-a: server1:/gfs/brick-a  
   brick-b: server1:/gfs/brick-b  
   brick-c: server1:/gfs/brick-c  
   brick-d: server1:/gfs/brick-d  
   brick-e: server1:/gfs/brick-e  
   brick-f: server1:/gfs/brick-f  
   brick-g: server1:/gfs/brick-g  
   brick-h: server1:/gfs/brick-h

   In the example above:

   Brick             |    Replica set        |    Brick subvolume index
   ----------------------------------------------------------------------------
   -/gfs/brick-a     |       0               |       0
   -/gfs/brick-b     |       0               |       1
   -/gfs/brick-c     |       1               |       2
   -/gfs/brick-d     |       1               |       3
   -/gfs/brick-e     |       2               |       4
   -/gfs/brick-f     |       2               |       5
   -/gfs/brick-g     |       3               |       6
   -/gfs/brick-h     |       3               |       7
   ```

   Each file in a brick maintains the changelog of itself and that of the files present in all the other bricks in it's replica set as seen by that brick.
   In the example volume given above, all files in brick-a will have 2 entries, one for itself and the other for the file present in it's replica pair. The following is the changelog for brick-b,
   • trusted.afr.vol-client-0=0x000000000000000000000000 - is the changelog for itself (brick-a)
   • trusted.afr.vol-client-1=0x000000000000000000000000 - changelog for brick-b as seen by brick-a
   Likewise, all files in brick-b will have the following:
   • trusted.afr.vol-client-0=0x000000000000000000000000 - changelog for brick-a as seen by brick-b
   • trusted.afr.vol-client-1=0x000000000000000000000000 - changelog for itself (brick-b)
   The same can be extended for other replica pairs.
   Interpreting changelog (approximate pending operation count) value

   Each extended attribute has a value which is 24 hexa decimal digits. First 8 digits represent changelog of data. Second 8 digits represent changelog of metadata. Last 8 digits represent Changelog of directory entries.

   Pictorially representing the same is as follows:

   0x 000003d7 00000001 00000000110
           |      |       |
           |      |        \_ changelog of directory entries
           |       \_ changelog of metadata
            \ _ changelog of data

   For directories, metadata and entry changelogs are valid. For regular files, data and metadata changelogs are valid. For special files like device files and so on, metadata changelog is valid. When a file split-brain happens it could be either be data split-brain or meta-data split-brain or both.
   The following is an example of both data, metadata split-brain on the same file:

   # getfattr -d -m . -e hex /gfs/brick-?/a 
   getfattr: Removing leading '/' from absolute path names
   \#file: gfs/brick-a/a 
   trusted.afr.vol-client-0=0x000000000000000000000000  
   trusted.afr.vol-client-1=0x000003d70000000100000000  
   trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57  
   \#file: gfs/brick-b/a  
   trusted.afr.vol-client-0=0x000003b00000000100000000  
   trusted.afr.vol-client-1=0x000000000000000000000000 
   trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57

   Scrutinize the changelogs

   The changelog extended attributes on file /gfs/brick-a/a are as follows:
   • The first 8 digits of trusted.afr.vol-client-0 are all zeros (0x00000000................),
     The first 8 digits of trusted.afr.vol-client-1 are not all zeros (0x000003d7................).
     So the changelog on /gfs/brick-a/a implies that some data operations succeeded on itself but failed on /gfs/brick-b/a.
   • The second 8 digits of trusted.afr.vol-client-0 are all zeros (0x........00000000........), and the second 8 digits of trusted.afr.vol-client-1 are not all zeros (0x........00000001........).
     So the changelog on /gfs/brick-a/a implies that some metadata operations succeeded on itself but failed on /gfs/brick-b/a.
   The changelog extended attributes on file /gfs/brick-b/a are as follows:
   • The first 8 digits of trusted.afr.vol-client-0 are not all zeros (0x000003b0................).
     The first 8 digits of trusted.afr.vol-client-1 are all zeros (0x00000000................).
     So the changelog on /gfs/brick-b/a implies that some data operations succeeded on itself but failed on /gfs/brick-a/a.
   • The second 8 digits of trusted.afr.vol-client-0 are not all zeros (0x........00000001........)
     The second 8 digits of trusted.afr.vol-client-1 are all zeros (0x........00000000........).
     So the changelog on /gfs/brick-b/a implies that some metadata operations succeeded on itself but failed on /gfs/brick-a/a.
   Here, both the copies have data, metadata changes that are not on the other file. Hence, it is both data and metadata split-brain.
   Deciding on the correct copy

   You must inspect stat and getfattr output of the files to decide which metadata to retain and contents of the file to decide which data to retain. To continue with the example above, here, we are retaining the data of /gfs/brick-a/a and metadata of /gfs/brick-b/a.

   Resetting the relevant changelogs to resolve the split-brain

   Resolving data split-brain

   You must change the changelog extended attributes on the files as if some data operations succeeded on /gfs/brick-a/a but failed on /gfs/brick-b/a. But /gfs/brick-b/a should not have any changelog showing data operations succeeded on /gfs/brick-b/a but failed on /gfs/brick-a/a. You must reset the data part of the changelog on trusted.afr.vol-client-0 of /gfs/brick-b/a.

   Resolving metadata split-brain

   You must change the changelog extended attributes on the files as if some metadata operations succeeded on /gfs/brick-b/a but failed on /gfs/brick-a/a. But /gfs/brick-a/a should not have any changelog which says some metadata operations succeeded on /gfs/brick-a/a but failed on /gfs/brick-b/a. You must reset metadata part of the changelog on trusted.afr.vol-client-1 of /gfs/brick-a/a
   Run the following commands to reset the extended attributes.
   1. On /gfs/brick-b/a, for trusted.afr.vol-client-0 0x000003b00000000100000000 to 0x000000000000000100000000, execute the following command:

      # setfattr -n trusted.afr.vol-client-0 -v 0x000000000000000100000000 /gfs/brick-b/a

   2. On /gfs/brick-a/a, for trusted.afr.vol-client-1 0x0000000000000000ffffffff to 0x000003d70000000000000000, execute the following command:

      # setfattr -n trusted.afr.vol-client-1 -v 0x000003d70000000000000000 /gfs/brick-a/a

   After you reset the extended attributes, the changelogs would look similar to the following:

   # getfattr -d -m . -e hex /gfs/brick-?/a  
   getfattr: Removing leading '/' from absolute path names  
   \#file: gfs/brick-a/a  
   trusted.afr.vol-client-0=0x000000000000000000000000  
   trusted.afr.vol-client-1=0x000003d70000000000000000  
   trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57  
   
   \#file: gfs/brick-b/a  
   trusted.afr.vol-client-0=0x000000000000000100000000  
   trusted.afr.vol-client-1=0x000000000000000000000000  
   trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
   

   Resolving Directory entry split-brain

   AFR has the ability to conservatively merge different entries in the directories when there is a split-brain on directory. If on one brick directory storage has entries 1, 2 and has entries 3, 4 on the other brick then AFR will merge all of the entries in the directory to have 1, 2, 3, 4 entries in the same directory. But this may result in deleted files to re-appear in case the split-brain happens because of deletion of files on the directory. Split-brain resolution needs human intervention when there is at least one entry which has same file name but different gfid in that directory.

   For example:
   On brick-a the directory has 2 entries file1 with gfid_x and file2 . On brick-b directory has 2 entries file1 with gfid_y and file3. Here the gfid's of file1 on the bricks are different. These kinds of directory split-brain needs human intervention to resolve the issue. You must remove either file1 on brick-a or the file1 on brick-b to resolve the split-brain.
   In addition, the corresponding gfid-link file must be removed. The gfid-link files are present in the .glusterfs directory in the top-level directory of the brick. If the gfid of the file is 0x307a5c9efddd4e7c96e94fd4bcdcbd1b (the trusted.gfid extended attribute received from the getfattr command earlier), the gfid-link file can be found at /gfs/brick-a/.glusterfs/30/7a/307a5c9efddd4e7c96e94fd4bcdcbd1b.

   Warning

   Before deleting the gfid-link, you must ensure that there are no hard links to the file present on that brick. If hard-links exist, you must delete them.
4. Trigger self-heal by running the following command:

   # ls -l <file-path-on-gluster-mount>

   or

   # gluster volume heal VOLNAME

1. Edit the file /etc/sysctl.conf to update the parameters with the desired values in the file.

   Example 9.1. Update the Virtual Memory Parameters

   vm.dirty_ratio = 5
   vm.dirty_background_ratio = 2

2. Execute the tuned-adm command to apply these values:

   # tuned-adm profile PROFILE-NAME

   Example 9.2.  Applying Virtual Memory Parameters

   In this example, rhs-high-throughput is the profile which is being activated.

   # tuned-adm profile rhs-high-throughput

3. Verify the changes made to the virtual Memory parameters.

   # cat /proc/sys/vm/dirty_ratio
   5
   # cat /proc/sys/vm/dirty_background_ratio
   2
   

1. Verify that your environment matches the minimum system requirements. See Section 10.3.3, “Prerequisites”.
2. Determine the appropriate deployment scenario. See Section 10.3.1, “Exploring Geo-replication Deployment Scenarios”.
3. Start geo-replication on the master and slave systems. See Section 10.4, “Starting Geo-replication”.

Creating Geo-replication Sessions

1. To create a common pem pub file, run the following command on the master node where the password-less SSH connection is configured:

   # gluster system:: execute gsec_create

2. Create the geo-replication session using the following command. The push-pem option is needed to perform the necessary pem-file setup on the slave nodes.

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL create push-pem [force]

   For example:

   # gluster volume geo-replication master-vol example.com::slave-vol create push-pem

   Note

   There must be password-less SSH access between the node from which this command is run, and the slave host specified in the above command. This command performs the slave verification, which includes checking for a valid slave URL, valid slave volume, and available space on the slave. If the verification fails, you can use the force option which will ignore the failed verification and create a geo-replication session.
3. Verify the status of the created session by running the following command:

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL status

1. Create a new group. For example, geogroup.
2. Create a unprivileged account. For example, geoaccount. Add geoaccount as a member of geogroup group.
3. As a root, create a new directory with permissions 0711. Ensure that the location where this directory is created is writeable only by root but geoaccount is able to access it. For example, create a mountbroker-root directory at /var/mountbroker-root.
4. Add the following options to the glusterd.vol file, assuming the name of the slave Red Hat Storage volume as slavevol:

   option mountbroker-root /var/mountbroker-root
   option mountbroker-geo-replication.geoaccount slavevol
   option geo-replication-log-group geogroup 
   option rpc-auth-allow-insecure on

   See Section 2.4, “Storage Concepts” for information on volume file of a Red Hat Storage volume.
   If you are unable to locate the glusterd.vol file at /etc/glusterfs/ directory, create a vol file containing both the default configuration and the above options and save it at /etc/glusterfs/.
   The following is the sample glusterd.vol file along with default options:

   volume management
       type mgmt/glusterd
       option working-directory /var/lib/glusterd
       option transport-type socket,rdma
       option transport.socket.keepalive-time 10
       option transport.socket.keepalive-interval 2
       option transport.socket.read-fail-log off
       option rpc-auth-allow-insecure on
       
       option mountbroker-root /var/mountbroker-root 
       option mountbroker-geo-replication.geoaccount slavevol
       option geo-replication-log-group geogroup
   end-volume

   • If you have multiple slave volumes on Slave, repeat Step 2 for each of them and add the following options to the vol file:

     option mountbroker-geo-replication.geoaccount2 slavevol2
     option mountbroker-geo-replication.geoaccount3 slavevol3

   • You can add multiple slave volumes within the same account (geoaccount) by providing comma-separated list (without spaces) as the argument of mountbroker-geo-replication.geogroup. You can also have multiple options of the form mountbroker-geo-replication.*. It is recommended to use one service account per Master machine. For example, if there are multiple slave volumes on Slave for the master machines Master1, Master2, and Master3, then create a dedicated service user on Slave for them by repeating Step 2. for each (like geogroup1, geogroup2, and geogroup3), and then add the following corresponding options to the volfile:

     option mountbroker-geo-replication.geoaccount1 slavevol11,slavevol12,slavevol13
     option mountbroker-geo-replication.geoaccount2 slavevol21,slavevol22
     option mountbroker-geo-replication.geoaccount3 slavevol31

5. Restart glusterd service on all the Slave nodes.
   After you setup an auxiliary glusterFS mount for the unprivileged account on all the Slave nodes, perform the following steps to setup a non-root geo-replication session.:
6. Setup a passwdless SSH from one of the master node to the user on one of the slave node. For example, to geoaccount.
7. Create a common pem pub file by running the following command on the master node where the password-less SSH connection is configured to the user on the slave node:

   # gluster system:: execute gsec_create

8. Create a geo-replication relationship between master and slave to the user by running the following command on the master node:
   For example,

   # gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol create push-pem

   If you have multiple slave volumes and/or multiple accounts, create a geo-replication session with that particular user and volume.
   For example,

   # gluster volume geo-replication MASTERVOL geoaccount2@SLAVENODE::slavevol2 create push-pem

9. In the slavenode, which is used to create relationship, run /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh as a root with user name, master volume name, and slave volume names as the arguments.
   For example,

    # /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh geoaccount MASTERVOL SLAVEVOL_NAME 

10. Start the geo-replication with slave user by running the following command on the master node:
    For example,

    # gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol start

11. Verify the status of geo-replication session by running the following command on the master node:

    # gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol status

Starting Geo-replication for a New Brick on a New Node

1. Run the following command on the master node where password-less SSH connection is configured, in order to create a common pem pub file.

   # gluster system:: execute gsec_create

2. Create the geo-replication session using the following command. The push-pem and force options are required to perform the necessary pem-file setup on the slave nodes.

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL create push-pem force

   For example:

   # gluster volume geo-replication master-vol example.com::slave-vol create push-pem force

   Note

   There must be password-less SSH access between the node from which this command is run, and the slave host specified in the above command. This command performs the slave verification, which includes checking for a valid slave URL, valid slave volume, and available space on the slave.
3. Start the geo-replication session between the slave and master forcefully, using the following command:

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL start force

4. Verify the status of the created session, using the following command:

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL status

Performing a Failover and Failback

1. Create a new geo-replication session with the original slave as the new master, and the original master as the new slave. For more information on setting and creating geo-replication session, see Section 10.3.4, “Setting Up your Environment for Geo-replication Session”.
2. Start the special synchronization mode to speed up the recovery of data from slave.

   # gluster volume geo-replication ORIGINAL_SLAVE_VOL ORIGINAL_MASTER_HOST::ORIGINAL_MASTER_VOL config special-sync-mode recover

3. Set a checkpoint to help verify the status of the data synchronization.

   # gluster volume geo-replication ORIGINAL_SLAVE_VOL ORIGINAL_MASTER_HOST::ORIGINAL_MASTER_VOL config checkpoint now

4. Start the new geo-replication session using the following command:

   # gluster volume geo-replication ORIGINAL_SLAVE_VOL ORIGINAL_MASTER_HOST::ORIGINAL_MASTER_VOL start

5. Monitor the checkpoint output using the following command, until the status displays: checkpoint as of <time of checkpoint creation> is completed at <time of completion>.

   # gluster volume geo-replication ORIGINAL_SLAVE_VOL ORIGINAL_MASTER_HOST::ORIGINAL_MASTER_VOL status

6. To resume the original master and original slave back to their previous roles, stop the I/O operations on the original slave, and using steps 3 and 5, ensure that all the data from the original slave is restored back to the original master. After the data from the original slave is restored back to the original master, stop the current geo-replication session (the failover session) between the original slave and original master, and resume the previous roles.
7. Reset the options that were set for promoting the slave volume as the master volume by running the following commands:

   # gluster volume reset ORIGINAL_SLAVE_VOL geo-replication.indexing  # gluster volume reset ORIGINAL_SLAVE_VOL changelog

   For more information on promoting slave volume to be the master volume, see Section 10.6.1, “Promoting a Slave to Master”.