Red Hat Training

A Red Hat training course is available for Red Hat Gluster Storage

Configuring Red Hat Virtualization with Red Hat Gluster Storage

Red Hat Gluster Storage 3.2

Integrating Red Hat Gluster Storage and Red Hat Virtualization

Laura Bailey

Red Hat Customer Content Services

Divya Muntimadugu

Red Hat Customer Content Services

Pavithra Srinivasan

Red Hat Customer Content Services

Anjana Suparna Sriram

Red Hat Customer Content Services

Abstract

This document explains how to configure Red Hat Gluster Storage as a storage domain for Red Hat Virtualization in a non-hyperconverged configuration.

Chapter 1. Introduction to Red Hat Gluster Storage for Virtualization

Red Hat provides a way to create and optimize Red Hat Gluster Storage volumes using the Red Hat Virtualization Manager. Red Hat Virtualization platform and Red Hat Gluster Storage platform comprise various components that work seamlessly together, enabling system administrators to install, configure, and manage a virtualized environment using Red Hat Gluster Storage as the virtual machine image store.
Red Hat Gluster Storage provides a POSIX-compatible file system that allows you to store virtual machine images in a Red Hat Gluster Storage Server cluster instead of network-attached storage (NAS) appliances or a storage area network (SAN) array.
From version 3.3 onward, Red Hat Virtualization supports the use of POSIX-compliant file systems for storage. Red Hat Gluster Storage provides a POSIX-compatible file system in which you can create Red Hat Gluster Storage volumes as storage domains.
Red Hat Virtualization and Red Hat Gluster Storage integration consists of:
The following diagram illustrates Red Hat Virtualization integration with Red Hat Gluster Storage:
Integration architecture

Figure 1.1. Integration architecture

Important

In Red Hat Virtualization 3.3 and later, the GlusterFS storage domain uses the same access mechanism as the POSIXFS domain. The GlusterFS storage domain is introduced to support the libgfapi protocol in the future releases.

Chapter 2. Support Requirements

This chapter outlines the requirements for creating a supported integration of Red Hat Gluster Storage and Red Hat Virtualization.

2.1. Prerequisites

Integrating Red Hat Gluster Storage with Red Hat Virtualization has the following requirements:

2.2. Compatible Versions

The product and package versions required to create a supported deployment of Red Hat Gluster Storage (RHGS) nodes managed by the specified version of Red Hat Virtualization (RHV) are documented in the following knowledge base article: https://access.redhat.com/articles/2356261.

Chapter 3. Enabling Red Hat Gluster Storage in Red Hat Virtualization Manager

3.1. Using Red Hat Gluster Storage Volumes in the Red Hat Virtualization Manager

Red Hat Virtualization supports Red Hat Gluster Storage volumes as storage domains. Enabling Red Hat Gluster Storage volumes in Red Hat Virtualization Manager requires installation and configuration of several components to create a functioning virtualization environment. You must install and configure each component in the following order:
Installation Workflow

Figure 3.1. Installation Workflow

Procedure 3.1. To Enable Red Hat Gluster Storage in Red Hat Virtualization Manager

  1. Install Red Hat Virtualization Manager

    The Red Hat Virtualization Manager is the control center of the Red Hat Virtualization environment. Ensure that Red Hat Virtualization Manger is installed and configured.
    Install and configure Red Hat Virtualization Manager. See Installing the Red Hat Virtualization Manager in the Red Hat Virtualization Installation Guide for more information: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/installation_guide/part-installing_red_hat_enterprise_virtualization.
    Ensure that the installation of Red Hat Virtualization Manager is complete before proceeding.

  2. Install Hosts

    Install the host machines with either of the following operating systems:
    Ensure that all hosts meet the hardware requirements of the operating system before beginning the installation process.
    During installation, set the fully qualified domain name (FQDN) and IP address for the server in order to avoid network conflicts. For information on installing the hypervisor, see Installing Hypervisor Hosts in the Red Hat Virtualization Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/installation_guide/part-installing_hypervisor_hosts.

  3. Configure the data centers

    Create data centers and clusters to organise the virtualization and storage hosts. Red Hat Gluster Storage servers must be managed in a separate data center to Red Hat Virtualization hosts, so at least two data centers are required. The installation process creates one data center and associated cluster named Default.
    The data center is expected to appear non-operational at this point in the process.

    Note

    A virtualization host can have either a Red Hat Virtualization Host or a Red Hat Enterprise Linux operating system. A trusted storage pool requires a minimum of three Red Hat Gluster Storage servers, and at least one active virtualization host is required to connect the system to a storage pool.

  4. Install Red Hat Gluster Storage

    Install the latest version of Red Hat Gluster Storage on new servers (not the host machines).
    For more information on obtaining and installing the software, see the Red Hat Gluster Storage Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/installation_guide/chap-installing_red_hat_storage.
    If you want to configure combined virtualization hosts and storage servers, see the Red Hat Hyperconverged Infrastructure documentation: https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infrastructure/1.0/html/deploying_red_hat_hyperconverged_infrastructure/

  5. Configure Storage

    Configure the Red Hat Gluster Storage installation for your requirements.
    For instructions on creating a Red Hat Gluster Storage Volume, see chapter Red Hat Gluster Storage Volume in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-red_hat_storage_volumes.
    Optimize your Red Hat Gluster Storage volumes for storing virtual machine images by following the instructions in Procedure 4.1, “To Configure Volumes Using the Command Line Interface”.

    Note

    To reduce the risk of split-brain conditions developing in the cluster, Red Hat requires three-way replication when using Red Hat Gluster Storage with Red Hat Virtualization. Three-way replication is provided by the following volume types:

  6. Attach Storage to Hypervisor

    Check that you can access the Red Hat Virtualization Manager Administration Portal and that at least one virtualization host has Up status before you continue.
    Create a new storage domain with a Data domain function and the GlusterFS storage type. For further information, see the Adding a Red Hat Gluster Storage Volume as a Storage Domain section in the Red Hat Virtualization Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/administration_guide/sect-using_red_hat_gluster_storage_as_a_storage_domain
You can now create virtual machines and its disk images on the Red Hat Gluster Storage volumes by following the instructions in Section 4.2, “Configuring Virtual Machines on Red Hat Gluster Storage volumes using the Red Hat Virtualization Manager”.

3.2. Optimizing Virtual Machines Running On Red Hat Gluster Storage Volumes

The readahead mechanism is essential for sequential read performance on virtual machines. For virtual machines that run on Red Hat Gluster Storage, the virtual-guest profile does not provide sufficient readahead by default. You can improve virtual machine performance by editing the virtual-guest profile to provide a more suitable readahead value.

  1. Install the Tune daemon

    Install the tuned daemon that tunes the system settings dynamically. 
    # yum install tuned

  2. On the virtual machine, edit the virtual-guest profile

    In the /etc/tune-profiles/virtual-guest/ktune.sh file, change the value of the multiply_disk_readahead option to 16.

  3. Apply the modified virtual-guest profile on the virtual machines

    Activate the modified profile with the command: 
    # tuned-adm profile virtual-guest

Chapter 4. Hosting Virtual Machine Images on Red Hat Gluster Storage volumes

Red Hat Gluster Storage provides a POSIX-compatible file system to store virtual machine images in Red Hat Gluster Storage volumes.
This chapter describes how to configure volumes using the command line interface, and how to prepare Red Hat Gluster Storage servers for virtualization using Red Hat Virtualization Manager.

4.1. Configuring Volumes Using the Command Line Interface

Red Hat recommends configuring volumes before starting them. For information on creating volumes, see Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-red_hat_storage_volumes.

Important

Hosting virtual machine images requires the consistency of three-way replication, which is provided by three-way replicated volumes, three-way distributed replicated volumes, arbitrated replicated volumes, and distributed arbitrated replicated volumes.

Procedure 4.1. To Configure Volumes Using the Command Line Interface

  1. Configure the rhgs-random-io tuned profile

    Install the tuned tuning daemon and configure Red Hat Gluster Storage servers to use the rhgs-random-io profile: 
    # yum install tuned
# tuned-adm profile rhgs-random-io
    For more information on available tuning profiles, refer to the tuned-adm man page, or see the Red Hat Gluster Storage 3.2 Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/.

  2. Review virt volume group configuration details

    The settings stored in the /var/lib/glusterd/groups/virt file are used to configure volumes in the virt volume group.

    Important

    When you upgrade, a new virt file may be created in /var/lib/glusterd/groups/virt.rpmnew. Ensure to apply the new virt file on the existing volumes by renaming the virt.rpmnew file to virt, along with the customized settings.
    By default, the /var/lib/glusterd/groups/virt file contains the following recommended settings. 
    performance.quick-read=off
performance.read-ahead=off
performance.io-cache=off
performance.stat-prefetch=off
performance.low-prio-threads=32
network.remote-dio=enable
cluster.eager-lock=enable
cluster.quorum-type=auto
cluster.server-quorum-type=server
cluster.data-self-heal-algorithm=full
cluster.locking-scheme=granular
cluster.shd-max-threads=8
cluster.shd-wait-qlength=10000
features.shard=on
user.cifs=off
    With the exception of cluster.eager-lock, features.shard, and cluster.data-self-heal-algorithm, these settings prevent caching within GlusterFS client stack, as it is the preferred mode for attaching disks to a virtual machine. The cluster.eager-lock option optimizes write performance with synchronous replication when there is a single writer to a file. The features.shard option enables sharding behavior. The cluster.data-self-heal-algorithm option specifies how self-heal operations are performed. For more information about any of these settings, see the Red Hat Gluster Storage 3.2 Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.2/html/Administration_Guide/chap-Managing_Red_Hat_Storage_Volumes.html#Configuring_Volume_Options

    Note

    Server-Side and Client-Side Quorum are enabled by default in the /var/lib/glusterd/groups/virt file to minimize split-brain scenarios. If Server-Side Quorum is not met, then the Red Hat Gluster Storage volumes become unavailable causing the Virtual Machines (VMs) to move to a paused state. If Client-Side Quorum is not met, although a replica pair in a Red Hat Gluster Storage volume is available in the read-only mode, the VMs move to a paused state.
    Manual intervention is required to make the VMs resume the operations after the quorum is restored. Consistency is achieved at the cost of fault tolerance. If fault tolerance is preferred over consistency, disable server-side and client-side quorum with the commands: 
    # gluster volume reset <vol-name> server-quorum-type
# gluster volume reset <vol-name> quorum-type
    For more information on these configuration settings, see the following sections in the Red Hat Gluster Storage Administration Guide:

  3. Assign volumes to virt group

    Red Hat recommends assigning volumes that store virtual machine images to the virt volume group so that these volumes can use common configuration details for their common use case. This has the same effect as the Optimize for Virt Store option in the management console. 
    # gluster volume set VOLNAME group virt

    Important

    After tagging the volume as group virt, use the volume for storing virtual machine images only and always access the volume through the glusterFS native client.

  4. Allow KVM and VDSM brick access

    Set the brick permissions for vdsm and kvm. If you do not set the required brick permissions, creation of virtual machines fails.
    1. Set the user and group permissions using the following commands: 
      # gluster volume set VOLNAME storage.owner-uid 36
# gluster volume set VOLNAME storage.owner-gid 36

  5. Configure granular healing

    Red Hat recommends setting cluster.granular-entry-heal to enable for this use case. To configure granular healing, run the following commands. 
    # gluster volume set VOLNAME cluster.granular-entry-heal enable

See Also:

4.2. Configuring Virtual Machines on Red Hat Gluster Storage volumes using the Red Hat Virtualization Manager

The following procedure describes how to add a Red Hat Gluster Storage server for virtualization using Red Hat Virtualization Manager.

Note

It is recommended that you use a separate data center for Red Hat Gluster Storage nodes.

Procedure 4.2. To Add a Red Hat Gluster Storage Server for Virtualization Using Red Hat Virtualization Manager

  1. Create a data center:
    1. Select the Data Centers resource tab to list all data centers.
    2. Click New to open the New Data Center window.
      New Data Center Window

      Figure 4.1. New Data Center Window

    3. Enter the Name and Description of the data center.
    4. Select the storage Type as Shared from the drop-down menu.
    5. Select the Quota Mode as Disabled.
    6. Click OK.
    The new data center is Uninitialized until you configure the cluster, host, and storage settings.
  2. Create a cluster:
    1. Select the Clusters resource tab to list all clusters.
    2. Click New to open the New Cluster window.
      New Cluster Window

      Figure 4.2. New Cluster Window

    3. Select a Data Center for the cluster from the drop-down menu.
    4. Enter a Name and Description for the cluster.
    5. Select the CPU Name and Compatibility Version from the drop-down menus.
    6. Check Enable Virt Service.
    7. Click OK.
  3. Add hosts:
    1. Select the Hosts resource tab to view a list of all hosts in the system.
    2. Click New to open the New Host window.
      New Host Window

      Figure 4.3. New Host Window

      Important

      A Red Hat Enterprise Linux hypervisor and Red Hat Virtualization hypervisor on a single VDSM cluster accessing the same virtual machine image store is not supported.
    3. Select the Data Center and Host Cluster for the new host from the drop-down menus.
    4. Enter the Name, Address, and Root Password of the new hypervisor host.
    5. Check Automatically configure host firewall if required.
    6. Click OK.
    The new host appears in the list of hypervisor hosts with the status Installing. After the host is activated, the status changes to Up automatically.
  4. Create and configure volumes on the Red Hat Gluster Storage cluster using the command line interface. For information on creating and configuring volumes, see Section 4.1, “Configuring Volumes Using the Command Line Interface” and Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-red_hat_storage_volumes.
  5. Add a storage domain using Red Hat Virtualization Manager:
    1. Select the Storage resource tab to list existing storage domains.
    2. Click New Domain to open the New Domain window.
      New Domain Window

      Figure 4.4. New Domain Window

    3. Enter a Name for the storage domain.
    4. Select a shared Data Center to associate with the storage domain.
    5. Set the Domain Function to Data and the Storage Type to GlusterFS.
    6. Select a host from the Host to Use drop-down menu.
    7. Check the Use managed gluster volume checkbox and select the appropriate volume from the Gluster dropdown menu.

      Note

      This dropdown menu is only populated with volumes whose nodes are managed by Red Hat Virtualization Manager. See Chapter 5, Managing Red Hat Gluster Storage Servers and Volumes using Red Hat Virtualization Manager for instructions on how to set up management of your Red Hat Gluster Storage nodes by Red Hat Virtualization Manager.
    8. Enter the applicable Red Hat Gluster Storage native client Mount Options. Enter multiple mount options separated by commas. For more information on native client mount options, see Creating Access to Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-accessing_data_-_setting_up_clients. Note that only the native client is supported when integrating Red Hat Gluster Storage and Red Hat Virtualization.
    9. Click OK.
You can now create virtual machines using Red Hat Gluster Storage as a storage domain. For more information on creating virtual machines, see the Red Hat Virtualization Virtual Machine Management Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/.

Note

To prevent the risk of split brain incidents on Red Hat Gluster Storage domains, the use of shareable disks on Red Hat Gluster Storage domains is disabled. Attempting to create a shareable disk brings up a warning in the administration portal which recommends the use of Quorum on the Red Hat Gluster Storage server to ensure data integrity.
This policy is not enforced on Red Hat Gluster Storage domains created on a POSIX domain with GlusterFS specified as the virtual file system type.

Chapter 5. Managing Red Hat Gluster Storage Servers and Volumes using Red Hat Virtualization Manager

You can create and configure Red Hat Gluster Storage volumes using Red Hat Virtualization Manager 3.3 or later by creating a separate cluster with the Enable Gluster Service option enabled.

Note

Red Hat Gluster Storage nodes must be managed in a separate cluster to Red Hat Virtualization hosts. If you want to configure combined management of virtualization hosts and storage servers, see the Red Hat Hyperconverged Infrastructure documentation: https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infrastructure/1.0/html/deploying_red_hat_hyperconverged_infrastructure/
A volume is a logical collection of bricks where each brick is an export directory on a server in the trusted storage pool. Most of the management operations for Red Hat Gluster Storage happen on these volumes. You can use Red Hat Virtualization Manager to create and start new volumes featuring a single global namespace.

Note

With the exception of the volume operations described in this section, all other Red Hat Gluster Storage functionalities must be executed from the command line.

5.1. Creating a Data Center

  1. Select the Data Centers resource tab to list all data centers in the results list.
  2. Click the New button to open the New Data Center window.
    New Data Center Window

    Figure 5.1. New Data Center Window

  3. Enter the Name and Description of the data center.
  4. Set Type to Shared from the drop-down menu.
    Set Quota Mode as Disabled.
  5. Click OK.
The new data center is added to the virtualization environment. It will remain Uninitialized until a cluster, host, and storage are configured.

5.2. Creating a Cluster

  1. Create a cluster with the gluster service enabled.
    New Cluster Window

    Figure 5.2. New Cluster Window

  2. Select the Compatibility Version from the drop-down menu.
  3. Click OK.

5.3. Adding Red Hat Gluster Storage Server to the Cluster

There are two ways to add hosts. You can either add new hosts or import the existing gluster configuration into a cluster.
  • To import an existing gluster configuration:
    1. Select Enable Gluster Service.
    2. Select Import existing gluster configuration. With this option you can import the existing Gluster configurations into a cluster.
    3. Provide the IP address of one of the hosts.
  • To add new hosts:
    1. Use the drop-down lists to select the Data Center and Host Cluster for the new host.
    2. Click OK.
      The new host displays in the list of hosts with a status of Installing. The host is activated and the status changes to Up automatically.
    You can manage the lifecycle of a volume using hook scripts. For more information, see chapter Managing Gluster Hooks in the Red Hat Gluster Storage Console Administration Guide

    Note

    To add multiple servers to a cluster, you must first add a Red Hat Gluster Storage server to the cluster. An error message appears if you add multiple servers in the first attempt.
    New Host window

    Figure 5.3. New Host window

5.4. Optimizing Red Hat Gluster Storage Volumes for Virtual Machine Images

If you have not already selected Optimize for Virt Store while creating a volume, select this option now to optimize your storage for virtual machine images. This has the same effect as adding the volume to the virt group.

5.5. Starting the Red Hat Gluster Storage Volume

  • In the Volumes tab, select the new volume and click Start to start the volume.
    The volume status changes to Up.

    Important

    If you are using Red Hat Gluster Storage Volumes for hosting Virtual Machine images, you must optimize the volume before attaching the volume to the storage domain.

5.6. Adding Bricks to Volumes

  1. Click Add Bricks to add bricks to your volume.
    A brick is the basic unit of storage, represented by an export directory on a server in the storage cluster. You can expand or shrink your cluster by adding new bricks or deleting existing bricks.
    Add Bricks

    Figure 5.5. Add Bricks

  2. Enter the path for the brick and click OK.
  3. In the Allow Access From field, specify volume access control as a comma-separated list of IP addresses or hostnames.
    By default, an asterisk (*) is used as a wildcard to specify ranges of addresses such as IP addresses or hostnames. You need to use IP-based authentication for Gluster exports.
  4. Click OK to create the volume.
    The new volume is added and it appears on the Volumes tab.
You can reuse a brick by selecting Allow bricks in root partition and reuse the bricks by clearing xattrs
You can create a storage domain using the optimized volume and manage it using Red Hat Virtualization Manager. See the Red Hat Gluster Storage 3.2 Console Administration Guide for more information on managing volumes such as start, stop, expand, shrink, add and edit volume options: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/console_administration_guide/.

5.7. Performing Asynchronous Tasks

From the Red Hat Virtualization Manager 3.4 release onwards, you can perform asynchronous tasks on the Red Hat Gluster Storage volume such as rebalance and remove brick operations.
For detailed instructions on performing these asynchronous tasks, see the Red Hat Gluster Storage Console Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/console_administration_guide/.

Appendix A. Revision History

Revision History
Revision 3.2-2Fri Mar 17 2017Laura Bailey
Updated the virt group profile to reflect new defaults in version 3.2. (BZ#1405348, BZ#1424908)
Clarified support for RHV-RHGS integration mount protocols. (BZ#1402409)

Legal Notice

Copyright © 2015-2017 Red Hat, Inc.
This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.