Administration Guide

1. Enable heketi by executing the following command:

   # systemctl enable heketi

2. Start the Heketi server, by executing the following command:

   # systemctl start heketi

3. To check the status of the Heketi server, execute the following command:

   # systemctl status heketi

4. To check the logs, execute the following command:

   # journalctl -u heketi

1. Topology file:

   {
       "clusters": [
           {
               "nodes": [
                   {
                       "node": {
                           "hostnames": {
                               "manage": [
                                   "10.0.0.1"
                               ],
                               "storage": [
                                   "10.0.0.1"
                               ]
                           },
                           "zone": 1
                       },
                       "devices": [
                           "/dev/sdb",
                           "/dev/sdc",
                           "/dev/sdd",
                           "/dev/sde",
                           "/dev/sdf",
                           "/dev/sdg",
                           "/dev/sdh",
                           "/dev/sdi"
                       ]
                   },
                   {
                       "node": {
                           "hostnames": {
                               "manage": [
                                   "10.0.0.2"
                               ],
                               "storage": [
                                   "10.0.0.2"
                               ]
                           },
                           "zone": 2
                       },
                       "devices": [
                           "/dev/sdb",
                           "/dev/sdc",
                           "/dev/sdd",
                           "/dev/sde",
                           "/dev/sdf",
                           "/dev/sdg",
                           "/dev/sdh",
                           "/dev/sdi"
                       ]
                   },
   
   .......
   .......

2. Load the Heketi JSON file:

   # heketi-cli topology load --json=topology_libvirt.json
   Creating cluster ... ID: a0d9021ad085b30124afbcf8df95ec06
           Creating node 192.168.10.100 ... ID: b455e763001d7903419c8ddd2f58aea0
                   Adding device /dev/vdb ... OK
                   Adding device /dev/vdc ... OK
   …….
           Creating node 192.168.10.101 ... ID: 4635bc1fe7b1394f9d14827c7372ef54
                   Adding device /dev/vdb ... OK
                   Adding device /dev/vdc ... OK
   ………….
   

3. Execute the following command to check the details of a particular node:

   # heketi-cli node info b455e763001d7903419c8ddd2f58aea0
   Node Id: b455e763001d7903419c8ddd2f58aea0
   Cluster Id: a0d9021ad085b30124afbcf8df95ec06
   Zone: 1
   Management Hostname: 192.168.10.100
   Storage Hostname: 192.168.10.100
   Devices:
   Id:0ddba53c70537938f3f06a65a4a7e88b   Name:/dev/vdi            Size (GiB):499     Used (GiB):0       Free (GiB):499
   Id:4fae3aabbaf79d779795824ca6dc433a   Name:/dev/vdg            Size (GiB):499     Used (GiB):0       Free (GiB):499
   …………….

4. Execute the following command to check the details of the cluster:

   # heketi-cli cluster info a0d9021ad085b30124afbcf8df95ec06
   Cluster id: a0d9021ad085b30124afbcf8df95ec06
   Nodes:
   4635bc1fe7b1394f9d14827c7372ef54
   802a3bfab2d0295772ea4bd39a97cd5e
   b455e763001d7903419c8ddd2f58aea0
   ff9eeb735da341f8772d9415166b3f9d
   Volumes:

5. To check the details of the device, execute the following command:

   # heketi-cli device info 0ddba53c70537938f3f06a65a4a7e88b
   Device Id: 0ddba53c70537938f3f06a65a4a7e88b
   Name: /dev/vdi
   Size (GiB): 499
   Used (GiB): 0
   Free (GiB): 499
   Bricks:
   

1. Create a physical volume(PV) by using the pvcreate command.
   For example:

   # pvcreate --dataalignment 1280K /dev/sdb

   Here, /dev/sdb is a storage device.
   Use the correct dataalignment option based on your device. For more information, see Section 20.2, “Brick Configuration”

   Note

   The device name and the alignment value will vary based on the device you are using.
2. Create a Volume Group (VG) from the PV using the vgcreate command:
   For example:

   # vgcreate --physicalextentsize 1280K rhs_vg /dev/sdb

3. Create a thin-pool using the following commands:

   # lvcreate --thin VOLGROUP/thin_pool --size pool_sz --chunksize chunk_sz --poolmetadatasize metadev_sz --zero n
   

   For example:

   # lvcreate --thin rhs_vg/rhs_pool --size 2T --chunksize 1280K --poolmetadatasize 16G --zero n

   To enhance the performance of Red Hat Gluster Storage, ensure you read Chapter 20, Tuning for Performance chapter.
4. Create a thinly provisioned volume that uses the previously created pool by running the lvcreate command with the --virtualsize and --thin options:

   # lvcreate --virtualsize size --thin volgroup/poolname --name volname

   For example:

   # lvcreate --virtualsize 1G --thin rhs_vg/rhs_pool --name rhs_lv

   It is recommended that only one LV should be created in a thin pool.

1. Run # mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10 DEVICE to format the bricks to the supported XFS file system format. Here, DEVICE is the created thin LV. The inode size is set to 512 bytes to accommodate for the extended attributes used by Red Hat Gluster Storage.
2. Run # mkdir /mountpoint to create a directory to link the brick to.
3. Add an entry in /etc/fstab:

   /dev/rhs_vg/rhs_lv /mountpoint  xfs rw,inode64,noatime,nouuid  1 2

4. Run # mount /mountpoint to mount the brick.
5. Run the df -h command to verify the brick is successfully mounted:

   # df -h /dev/rhs_vg/rhs_lv   16G  1.2G   15G   7% /rhgs

6. If SElinux is enabled, then the SELinux labels that has to be set manually for the bricks created using the following commands:

   # semanage fcontext -a -t glusterd_brick_t /rhgs/brick1
   # restorecon -Rv /rhgs/brick1

1. Open the /etc/fstab file in a text editor.
2. Append the following configuration to the fstab file.

   HOSTNAME|IPADDRESS:/VOLNAME /MOUNTDIR glusterfs defaults,_netdev 0 0

   Using the example server names, the entry contains the following replaced values.

   server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0

   If you want to specify the transport type then check the following example:

   server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev,transport=tcp 0 0

1. Create a replicate volume. This volume will host only a zero byte lock file, hence choose minimal sized bricks. To create a replicate volume run the following command:

   # gluster volume create volname replica n ipaddress:/brick path.......N times

   where,
   N: The number of nodes that are used as NFS servers. Each node must host one brick.
   For example:

   # gluster volume create ctdb replica 4 10.16.157.75:/rhgs/brick1/ctdb/b1 10.16.157.78:/rhgs/brick1/ctdb/b2 10.16.157.81:/rhgs/brick1/ctdb/b3 10.16.157.84:/rhgs/brick1/ctdb/b4

2. In the following files, replace "all" in the statement META="all" to the newly created volume name

   /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
   /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh

   For example:

   META="all"
     to
   META="ctdb"

3. Start the volume.
   The S29CTDBsetup.sh script runs on all Red Hat Gluster Storage servers, adds an entry in /etc/fstab/ for the mount, and mounts the volume at /gluster/lock on all the nodes with NFS server. It also enables automatic start of CTDB service on reboot.

   Note

   When you stop the special CTDB volume, the S29CTDB-teardown.sh script runs on all Red Hat Gluster Storage servers and removes an entry in /etc/fstab/ for the mount and unmounts the volume at /gluster/lock.
4. Verify if the file /etc/sysconfig/ctdb exists on all the nodes that is used as NFS server. This file contains Red Hat Gluster Storage recommended CTDB configurations.
5. Create /etc/ctdb/nodes file on all the nodes that is used as NFS servers and add the IPs of these nodes to the file.

   10.16.157.0
   10.16.157.3
   10.16.157.6
   10.16.157.9

   The IPs listed here are the private IPs of NFS servers.
6. On all the nodes that are used as NFS server which require IP failover, create /etc/ctdb/public_addresses file and add the virtual IPs that CTDB should create to this file. Add these IP address in the following format:

   <Virtual IP>/<routing prefix><node interface>

   For example:

   192.168.1.20/24 eth0
   192.168.1.21/24 eth0

7. Start the CTDB service on all the nodes by executing the following command:

   # service ctdb start

1. If a mount point has not yet been created for the volume, run the mkdir command to create a mount point.

   # mkdir /mnt/glusterfs

2. Run the correct mount command for the system.

   For Linux

   # mount -t nfs -o vers=3 server1:/test-volume /mnt/glusterfs

   For Solaris

   # mount -o vers=3 nfs://server1:38467/test-volume /mnt/glusterfs

1. If a mount point has not yet been created for the volume, run the mkdir command to create a mount point.

   # mkdir /mnt/glusterfs

2. Run the correct mount command for the system, specifying the TCP protocol option for the system.

   For Linux

   # mount -t nfs -o vers=3,mountproto=tcp server1:/test-volume /mnt/glusterfs

   For Solaris

   # mount -o proto=tcp, nfs://server1:38467/test-volume /mnt/glusterfs

1. Open the /etc/fstab file in a text editor.
2. Append the following configuration to the fstab file.

   HOSTNAME|IPADDRESS:/VOLNAME /MOUNTDIR nfs defaults,_netdev, 0 0

   Using the example server names, the entry contains the following replaced values.

   server1:/test-volume /mnt/glusterfs nfs defaults,_netdev, 0 0

1. Open the /etc/fstab file in a text editor.
2. Append the following configuration to the fstab file.

   HOSTNAME|IPADDRESS:/VOLNAME /MOUNTDIR nfs defaults,_netdev,mountproto=tcp 0 0

   Using the example server names, the entry contains the following replaced values.

   server1:/test-volume /mnt/glusterfs nfs defaults,_netdev,mountproto=tcp 0 0

1. Creating the ganesha-ha.conf file

   The ganesha-ha.conf.sample is created in the following location /etc/ganesha when Red Hat Gluster Storage is installed. Rename the file to ganesha-ha.conf and make the changes based on your environment.

   Following is an example:

   Sample ganesha-ha.conf file:
   
   # Name of the HA cluster created.
   # must be unique within the subnet
   HA_NAME="ganesha-ha-360"
   #
   #
   # You may use short names or long names; you may not use IP addresses.
   # Once you select one, stay with it as it will be mildly unpleasant to clean up if you switch later on. Ensure that all names - short and/or long - are in DNS or /etc/hosts on all machines in the cluster.
   #
   # The subset of nodes of the Gluster Trusted Pool that form the ganesha HA cluster. Hostname is specified.
   HA_CLUSTER_NODES="server1.lab.redhat.com,server2.lab.redhat.com,..."
   #
   # Virtual IPs for each of the nodes specified above.
   VIP.server1.lab.redhat.com="10.0.2.1"
   VIP.server2.lab.redhat.com="10.0.2.2"
   ....
   ....

   Note

   • Pacemaker handles the creation of the VIP and assigning a interface.
   • Ensure that the VIP is in the same network range.
2. Configuring NFS-Ganesha using gluster CLI

   The HA cluster can be set up or torn down using gluster CLI. In addition, it can export and unexport specific volumes. For more information, see section Configuring NFS-Ganesha using gluster CLI.

3. Modifying the HA cluster using the ganesha-ha.sh script

   After creating the cluster, any further modification can be done using the ganesha-ha.sh script. For more information, see Modifying the HA cluster using the ganesha-ha.sh script.

1. Install the krb5-workstation and the ntpdate packages on all the machines:

   # yum install krb5-workstation
   # yum install ntpdate

   Note

   • The krb5-libs package will be updated as a dependent package.
2. Configure the ntpdate based on the valid time server according to the environment:

   # echo <valid_time_server> >> /etc/ntp/step-tickers
   
   # systemctl enable ntpdate
   
   # systemctl start ntpdate

3. Ensure that all systems can resolve each other by FQDN in DNS.
4. Configure the /etc/krb5.conf file and add relevant changes accordingly. For example:

   [logging]
     default = FILE:/var/log/krb5libs.log
     kdc = FILE:/var/log/krb5kdc.log
     admin_server = FILE:/var/log/kadmind.log
   
     [libdefaults]
     dns_lookup_realm = false
     ticket_lifetime = 24h
     renew_lifetime = 7d
     forwardable = true
     rdns = false
     default_realm = EXAMPLE.COM
     default_ccache_name = KEYRING:persistent:%{uid}
   
     [realms]
     EXAMPLE.COM = {
     kdc = kerberos.example.com
       admin_server = kerberos.example.com
     }
   
     [domain_realm]
     .example.com = EXAMPLE.COM
      example.com = EXAMPLE.COM

   Note

   For further details regarding the file configuration, refer to man krb5.conf.
5. On the NFS-server and client, update the /etc/idmapd.conf file by making the required change. For example:

   Domain = example.com

1. Edit/add the required fields in the corresponding export file located at /etc/ganesha/exports/.
2. Execute the following command

   # /usr/libexec/ganesha/ganesha-ha.sh --refresh-config <HA_CONF_DIR> <volname>

Mounting a Volume Manually using SMB on Red Hat Enterprise Linux

1. Install the cifs-utils package on the client.

   # yum install cifs-utils

2. Run mount -t cifs to mount the exported SMB share, using the syntax example as guidance.

   # mount -t cifs -o user=<username>,pass=<password> //<hostname>/gluster-<volname> /<mountpoint>

   For example:

   # mount -t cifs -o user=cifsuser,pass=redhat //rhs-a/gluster-repvol /cifs

3. Run # smbstatus -S on the server to display the status of the volume:

   Service        pid     machine             Connected at
   -------------------------------------------------------------------
   gluster-VOLNAME 11967   __ffff_192.168.1.60  Mon Aug  6 02:23:25 2012

Mounting a Volume Manually using SMB through Microsoft Windows Explorer

1. In Windows Explorer, click Tools → Map Network Drive…. to open the Map Network Drive screen.
2. Choose the drive letter using the Drive drop-down list.
3. In the Folder text box, specify the path of the server and the shared resource in the following format: \\SERVER_NAME\VOLNAME.
4. Click Finish to complete the process, and display the network drive in Windows Explorer.
5. Navigate to the network drive to verify it has mounted correctly.

Mounting a Volume Manually using SMB on Microsoft Windows Command-line.

1. Click Start → Run, and then type cmd.
2. Enter net use z: \\SERVER_NAME\VOLNAME, where z: is the drive letter to assign to the shared volume.
   For example, net use y: \\server1\test-volume
3. Navigate to the network drive to verify it has mounted correctly.

Mounting a Volume Automatically using SMB on Red Hat Enterprise Linux

1. Open the /etc/fstab file in a text editor.
2. Append the following configuration to the fstab file.
   You must specify the filename and its path that contains the user name and/or password in the credentials option in /etc/fstab file. See the mount.cifs man page for more information.

   \\HOSTNAME|IPADDRESS\SHARE_NAME MOUNTDIR

   Using the example server names, the entry contains the following replaced values.

   \\server1\test-volume /mnt/glusterfs cifs credentials=/etc/samba/passwd,_netdev 0 0

3. Run # smbstatus -S on the client to display the status of the volume:

   Service        pid     machine             Connected at
   -------------------------------------------------------------------
   gluster-VOLNAME 11967   __ffff_192.168.1.60  Mon Aug  6 02:23:25 2012

Mounting a Volume Automatically on Server Start using SMB through Microsoft Windows Explorer

1. In Windows Explorer, click Tools → Map Network Drive…. to open the Map Network Drive screen.
2. Choose the drive letter using the Drive drop-down list.
3. In the Folder text box, specify the path of the server and the shared resource in the following format: \\SERVER_NAME\VOLNAME.
4. Click the Reconnect at logon check box.
5. Click Finish to complete the process, and display the network drive in Windows Explorer.
6. If the Windows Security screen pops up, enter the username and password and click OK.
7. Navigate to the network drive to verify it has mounted correctly.

1. Run service smb [re]start to start or restart the smb service.
2. Enable User Serviceable Snapshot (USS) for Samba. For more information see Section 8.13, “User Serviceable Snapshots”

1. Right Click on the file or directory for which the previous version is required.
2. Click on Restore previous versions.
3. In the dialog box, select the Date/Time of the previous version of the file, and select either Open, Restore, or Copy.
   where,
   Open: Lets you open the required version of the file in read-only mode.
   Restore: Restores the file back to the selected version.
   Copy: Lets you copy the file to a different location.
   

   Figure 6.1. Accessing Snapshot

1. To enable cache invalidation and increase the timeout to 10 minutes execute the following commands:

   # gluster volume set <volname> features.cache-invalidation on
   
   volume set success

   # gluster volume set <volname> features.cache-invalidation-timeout 600
   
   volume set success

   To enable metadata caching on the client and to maintain cache consistency execute the following commands:

   # gluster volume set <volname> performance.stat-prefetch on
   
   volume set success

   # gluster volume set <volname> performance.cache-invalidation on
   
   volume set success

   # gluster volume set <volname> performance.cache-samba-metadata on
   
   volume set success

2. To increase the client side metadata cache timeout to 10 minutes, execute the following command:

   # gluster volume set <volname> performance.md-cache-timeout 600
   
   volume set success