Red Hat Training

A Red Hat training course is available for Red Hat Gluster Storage

Chapter 6. Creating Access to Volumes

Red Hat Gluster Storage volumes can be accessed using a number of technologies:
Cross Protocol Data Access

Because of differences in locking semantics, a single Red Hat Gluster Storage volume cannot be concurrently accessed by multiple protocols. Current support for concurrent access is defined in the following table.

Table 6.1. Cross Protocol Data Access Matrix

  SMB Gluster NFS NFS-Ganesha Native FUSE Object
SMB Yes No No No No
Gluster NFS No Yes No No No
NFS-Ganesha No No Yes No No
Native FUSE No No No Yes Yes [a]
Object No No No Yes [a] Yes
[a] For more information, refer Section 6.5, “Managing Object Store”.
Access Protocols Supportability

The following table provides the support matrix for the supported access protocols with TCP/RDMA.

Table 6.2. Access Protocol Supportability Matrix

Access Protocols TCP RDMA
FUSEYes Yes
SMB Yes No
NFSYesYes

Important

Red Hat Gluster Storage requires certain ports to be open. You must ensure that the firewall settings allow access to the ports listed at Chapter 3, Considerations for Red Hat Gluster Storage.
Gluster user is created as a part of gluster installation. The purpose of gluster user is to provide privileged access to libgfapi based application (for example, nfs-ganesha and glusterfs-coreutils ). For a normal user of an application, write access to statedump directory is restricted. As a result, attempting to write a state dump to this directory fails. Privileged access is needed by these applications in order to be able to write to the statedump directory. In order to write to this location, the user that runs the application should ensure that the application is added to the gluster user group. After the application is added, restart gluster processes to apply the new group.

6.1. Native Client

Native Client is a FUSE-based client running in user space. Native Client is the recommended method for accessing Red Hat Gluster Storage volumes when high concurrency and high write performance is required.
This section introduces Native Client and describes how to perform the following:
  • Install Native Client packages
  • Mount Red Hat Gluster Storage volumes (manually and automatically)
  • Verify that the Gluster Storage volume has mounted successfully

Table 6.3. Red Hat Gluster Storage Support Matrix

Red Hat Enterprise Linux version Red Hat Gluster Storage version Native client version
6.5 3.0 3.0, 2.1*
6.6 3.0.2, 3.0.3, 3.0.4 3.0, 2.1*
6.73.1, 3.1.1, 3.1.23.1, 3.0, 2.1*
6.83.1.33.1.3
6.93.23.2, 3.1.3*
6.93.33.3, 3.2
6.93.3.13.3.1, 3.3, 3.2
6.103.43.4, 3.3.z
7.13.1, 3.1.13.1.1, 3.1, 3.0
7.23.1.23.1.2, 3.1, 3.0
7.23.1.33.1.3
7.33.23.2, 3.1.3
7.43.23.2, 3.1.3
7.43.33.3, 3.2
7.43.3.13.3.1, 3.3, 3.2
7.53.3.1, 3.43.3.z, 3.4.z
7.63.3.1, 3.43.3.z, 3.4.z

Warning

If you want to access a volume being provided by a server using Red Hat Gluster Storage 3.1.3 or higher, your client must also be using Red Hat Gluster Storage 3.1.3 or higher. Accessing these volumes from earlier client versions can result in data becoming unavailable and problems with directory operations. This requirement exists because Red Hat Gluster Storage 3.1.3 changed how the Distributed Hash Table works in order to improve directory consistency and remove the effects seen in BZ#1115367 and BZ#1118762.

Warning

The following issues are observed and recorded for Red Hat Gluster Storage 3.2 on RHEL 6.x and 7.x using Native Client 3.1.3:
  • gluster volume heal VOLNAME info is unresponsive for some volumes. (BZ#1500542)
  • Gluster brick process crashes frequently. (BZ#1510725)
  • Multiple disconnects on NFS mounts. (BZ#1425740)

Warning

  • For Red Hat Gluster Storage 3.4, Red Hat supports Red Hat Gluster Storage 3.3 and 3.4 clients only.
  • For Red Hat Gluster Storage 3.2, you need to have Red Hat Gluster Storage 3.2 clients. This version is not compatible with backward versions of the client.
For more information on the release version see, https://access.redhat.com/solutions/543123.

6.1.1. Installing Native Client

After installing the client operating system, register the target system to Red Hat Network and subscribe to the Red Hat Enterprise Linux Server channel.

Important

All clients must be of the same version. Red Hat strongly recommends upgrading the servers before upgrading the clients.

Use the Command Line to Register and Subscribe a System to Red Hat Subscription Management

Register the system using the command line, and subscribe to the correct repositories.

Prerequisites

  • Know the user name and password of the Red Hat Subscription Manager account with Red Hat Gluster Storage entitlements.
  1. Run the subscription-manager register command to list the available pools. Select the appropriate pool and enter your Red Hat Subscription Manager user name and password to register the system with Red Hat Subscription Manager. 
    # subscription-manager register
  2. Depending on your client, run one of the following commands to subscribe to the correct repositories.
    • For Red Hat Enterprise Linux 7.x clients: 
      # subscription-manager repos --enable=rhel-7-server-rpms --enable=rh-gluster-3-client-for-rhel-7-server-rpms

      Note

      The following command can also be used, but Red Hat Gluster Storage may deprecate support for this repository in future releases. 
      # subscription-manager repos --enable=rhel-7-server-rh-common-rpms
    • For Red Hat Enterprise Linux 6.1 and later clients: 
      # subscription-manager repos --enable=rhel-6-server-rpms --enable=rhel-6-server-rhs-client-1-rpms
    • For Red Hat Enterprise Linux 5.7 and later clients: 
      # subscription-manager repos --enable=rhel-5-server-rpms --enable=rhel-5-server-rhs-client-1-rpms
    For more information, see Section 3.2 Registering from the Command Line in Using and Configuring Red Hat Subscription Management.
  3. Verify that the system is subscribed to the required repositories. 
    # yum repolist

Use the Web Interface to Register and Subscribe a System

Register the system using the web interface, and subscribe to the correct channels.

Prerequisites

  • Know the user name and password of the Red Hat Subsrciption Management (RHSM) account with Red Hat Gluster Storage entitlements.
  1. Log on to Red Hat Subscription Management (https://access.redhat.com/management).
  2. Click the Systems link at the top of the screen.
  3. Click the name of the system to which the Red Hat Gluster Storage Native Client channel must be appended.
  4. Click Alter Channel Subscriptions in the Subscribed Channels section of the screen.
  5. Expand the node for Additional Services Channels for Red Hat Enterprise Linux 7 for x86_64 or Red Hat Enterprise Linux 6 for x86_64 or for Red Hat Enterprise Linux 5 for x86_64 depending on the client platform.
  6. Click the Change Subscriptions button to finalize the changes.
    When the page refreshes, select the Details tab to verify the system is subscribed to the appropriate channels.

Install Native Client Packages

Install Native Client packages from Red Hat Network
  1. Run the yum install command to install the native client RPM packages. 
    # yum install glusterfs glusterfs-fuse
  2. For Red Hat Enterprise 5.x client systems, run the modprobe command to load FUSE modules before mounting Red Hat Gluster Storage volumes. 
    # modprobe fuse
    For more information on loading modules at boot time, see https://access.redhat.com/knowledge/solutions/47028 .

6.1.2. Upgrading Native Client

Before updating the Native Client, subscribe the clients to the channels mentioned in Section 6.1.1, “Installing Native Client”

Warning

If you want to access a volume being provided by a server using Red Hat Gluster Storage 3.1.3 or higher, your client must also be using Red Hat Gluster Storage 3.1.3 or higher. Accessing these volumes from earlier client versions can result in data becoming unavailable and problems with directory operations. This requirement exists because Red Hat Gluster Storage 3.1.3 changed how the Distributed Hash Table works in order to improve directory consistency and remove the effects seen in BZ#1115367 and BZ#1118762.

  1. Unmount gluster volumes

    Unmount any gluster volumes prior to upgrading the native client. 
    # umount /mnt/glusterfs

  2. Upgrade the client

    Run the yum update command to upgrade the native client: 
    # yum update glusterfs glusterfs-fuse

  3. Remount gluster volumes

    Remount volumes as discussed in Section 6.1.3, “Mounting Red Hat Gluster Storage Volumes”.

6.1.3. Mounting Red Hat Gluster Storage Volumes

After installing Native Client, the Red Hat Gluster Storage volumes must be mounted to access data. Three methods are available:
After mounting a volume, test the mounted volume using the procedure described in Section 6.1.3.5, “Testing Mounted Volumes”.

Note

  • Clients should be on the same version as the server, and at least on the version immediately previous to the server version. For Red Hat Gluster Storage 3.4, the recommended native client version should either be 3.4 or 3.3.z. For other versions, see Section 6.1, “Native Client”.
  • Server names selected during volume creation should be resolvable in the client machine. Use appropriate /etc/hosts entries, or a DNS server to resolve server names to IP addresses.
  • Internet Protocol Version 6 (IPv6) support is available only for Red Hat Hyperconverged Infrastructure for Virtualization environments and not for Red Hat Gluster Storage standalone environments.

6.1.3.1. Mount Commands and Options

The following options are available when using the mount -t glusterfs command. All options must be separated with commas. 
# mount -t glusterfs -o backup-volfile-servers=volfile_server2:volfile_server3:.... ..:volfile_serverN,transport-type tcp,log-level=WARNING,reader-thread-count=2,log-file=/var/log/gluster.log server1:/test-volume /mnt/glusterfs
backup-volfile-servers=<volfile_server2>:<volfile_server3>:...:<volfile_serverN>
List of the backup volfile servers to mount the client. If this option is specified while mounting the fuse client, when the first volfile server fails, the servers specified in backup-volfile-servers option are used as volfile servers to mount the client until the mount is successful.

Note

This option was earlier specified as backupvolfile-server which is no longer valid.
log-level
Logs only specified level or higher severity messages in the log-file.
log-file
Logs the messages in the specified file.
transport-type
Specifies the transport type that FUSE client must use to communicate with bricks. If the volume was created with only one transport type, then that becomes the default when no value is specified. In case of tcp,rdma volume, tcp is the default.
dump-fuse
This mount option creates dump of fuse traffic between the glusterfs client (fuse userspace server) and the kernel. The interface to mount a glusterfs volume is the standard mount(8) command from the CLI. This feature enables the same in the mount option.
# mount -t glusterfs -odump-fuse=filename hostname:/volname mount-path
For example, 
# mount -t glusterfs -odump-fuse=/dumpfile  10.70.43.18:/arbiter /mnt/arbiter
The above command generates a binary file with the name dumpfile.

Note

The fusedump grows large with time and notably if the client gets a heavy load. So this is not an intended use case to do fusedump during normal usage. It is advised to use this to get a dump from a particular scenario, for diagnostic purposes.
You need to unmount and remount the volume without the fusedump option to stop dumping.
ro
Mounts the file system with read-only permissions.
acl
Enables POSIX Access Control List on mount. See Section 6.4.4, “Checking ACL enablement on a mounted volume” for further information.
background-qlen=n
Enables FUSE to handle n number of requests to be queued before subsequent requests are denied. Default value of n is 64.
enable-ino32
Enables file system to present 32-bit inodes instead of 64-bit inodes.
reader-thread-count=n
Enables FUSE to add n number of reader threads that can give better I/O performance. Default value of n is 1.
lru-limit
This mount command option clears the inodes from the least recently used (lru) list (which keeps non-referenced inodes) after the inode limit has reached.
For example, 
# mount -olru-limit=NNNN -t glusterfs hostname:/volname /mnt/mountdir
Where NNNN is a positive integer. The default value of NNNN is 128k (131072) and the recommended value is 20000 and above. If 0 is specified as the lru-limit then it means that no invalidation of inodes from the lru-list.

6.1.3.2. Mounting Volumes Manually

Manually Mount a Red Hat Gluster Storage Volume or Subdirectory

Create a mount point and run the following command as required:
For a Red Hat Gluster Storage Volume
mount -t glusterfs HOSTNAME|IPADDRESS:/VOLNAME /MOUNTDIR
For a Red Hat Gluster Storage Volume's Subdirectory
mount -t glusterfs HOSTNAME|IPADDRESS:/VOLNAME/SUBDIRECTORY /MOUNTDIR

Note

The server specified in the mount command is used to fetch the glusterFS configuration volfile, which describes the volume name. The client then communicates directly with the servers mentioned in the volfile (which may not actually include the server used for mount).
  1. If a mount point has not yet been created for the volume, run the mkdir command to create a mount point. 
    # mkdir /mnt/glusterfs
  2. Run the mount -t glusterfs command, using the key in the task summary as a guide.
    1. For a Red Hat Gluster Storage Volume: 
      # mount -t glusterfs server1:/test-volume /mnt/glusterfs
    2. For a Red Hat Gluster Storage Volume's Subdirectory 
      # mount -t glusterfs server1:/test-volume/sub-dir /mnt/glusterfs

6.1.3.3. Mounting Volumes Automatically

Volumes can be mounted automatically each time the systems starts.
The server specified in the mount command is used to fetch the glusterFS configuration volfile, which describes the volume name. The client then communicates directly with the servers mentioned in the volfile (which may not actually include the server used for mount).
Mounting a Volume Automatically
Mount a Red Hat Gluster Storage Volume automatically at server start.
  1. Open the /etc/fstab file in a text editor.
  2. Append the following configuration to the fstab file:
    For a Red Hat Gluster Storage Volume
    HOSTNAME|IPADDRESS:/VOLNAME /MOUNTDIR glusterfs defaults,_netdev 0 0
    For a Red Hat Gluster Storage Volume's Subdirectory
    HOSTNAME|IPADDRESS:/VOLNAME/SUBDIRECTORY /MOUNTDIR glusterfs defaults,_netdev 0 0
    Using the example server names, the entry contains the following replaced values. 
    server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0
    OR 
    server1:/test-volume/subdir /mnt/glusterfs glusterfs defaults,_netdev 0 0
    If you want to specify the transport type then check the following example: 
    server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev,transport=tcp 0 0
    OR 
    server1:/test-volume/sub-dir /mnt/glusterfs glusterfs defaults,_netdev,transport=tcp 0 0

6.1.3.4. Manually Mounting Sub-directories Using Native Client

With Red Hat Gluster Storage 3.4, you can share a single Gluster volume with different clients and they all can mount only a subset of the volume namespace. This feature is similar to the NFS subdirectory mount feature where you can export a subdirectory of an already exported volume. You can also use this feature to restrict full access to any particular volume.
Mounting subdirectories provides the following benefits:
  • Provides namespace isolation so that multiple users can access the storage without risking namespace collision with other users.
  • Prevents the root file system from becoming full in the event of a mount failure.
You can mount a subdirectory using native client by running either of the following commands: 
# mount -t glusterfs hostname:/volname/subdir /mount-point
OR 
# mount -t glusterfs hostname:/volname -osubdir-mount=subdir /mount-point
For example: 
# gluster volume set test-vol auth.allow "/(192.168.10.*|192.168.11.*),/subdir1(192.168.1.*),/subdir2(192.168.8.*)”
In the above example:
  • The auth.allow option allows only the directories specified as the value of the auth.allow option to be mounted.
  • Each group of auth-allow is separated by a comma (,).
  • Each group has a directory separated by parentheses, (), which contains the valid IP addresses.
  • All subdirectories start with /, that is, no relative path to a volume, but everything is an absolute path, taking / as the root directory of the volume.

Note

By default, the authentication is *, where any given subdirectory in a volume can be mounted by all clients.

6.1.3.5. Testing Mounted Volumes

Testing Mounted Red Hat Gluster Storage Volumes

Using the command-line, verify the Red Hat Gluster Storage volumes have been successfully mounted. All three commands can be run in the order listed, or used independently to verify a volume has been successfully mounted.
  1. Run the mount command to check whether the volume was successfully mounted. 
    # mount
server1:/test-volume on /mnt/glusterfs type fuse.glusterfs(rw,allow_other,default_permissions,max_read=131072
    OR 
    # mount
server1:/test-volume/sub-dir on /mnt/glusterfs type fuse.glusterfs(rw,allow_other,default_permissions,max_read=131072
    If transport option is used while mounting a volume, mount status will have the transport type appended to the volume name. For example, for transport=tcp: 
    # mount
server1:/test-volume.tcp on /mnt/glusterfs type fuse.glusterfs(rw,allow_other,default_permissions,max_read=131072
    OR 
    # mount
server1:/test-volume/sub-dir.tcp on /mnt/glusterfs type fuse.glusterfs(rw,allow_other,default_permissions,max_read=131072
  2. Run the df command to display the aggregated storage space from all the bricks in a volume. 
    # df -h /mnt/glusterfs
Filesystem           Size  Used  Avail  Use%  Mounted on
server1:/test-volume  28T  22T   5.4T   82%   /mnt/glusterfs
  3. Move to the mount directory using the cd command, and list the contents. 
    # cd /mnt/glusterfs
# ls