Configuring Red Hat Virtualization with Red Hat Gluster Storage
Traditional, non-hyperconverged storage for virtual machines
Chapter 1. Introduction to Red Hat Gluster Storage for Virtualization
Figure 1.1. Architecture of integrated Red Hat Virtualization and Red Hat Gluster Storage
Chapter 2. Support Requirements
- All installations of Red Hat Virtualization and Red Hat Gluster Storage must have valid subscriptions to Red Hat Network channels and Subscription Management repositories.
- Red Hat Virtualization installations must adhere to the requirements laid out in the Red Hat Virtualization Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html-single/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_cockpit_web_interface/index#RHV_requirements.
- Red Hat Gluster Storage installations must adhere to the requirements laid out in the Red Hat Gluster Storage Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/installation_guide/chap-planning_red_hat_storage_installation.
- Red Hat Gluster Storage installations must be completely up to date with the latest patches and upgrades. Refer to the Red Hat Gluster Storage 3.5 Installation Guide to upgrade to the latest version: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/installation_guide/.
- The versions of Red Hat Virtualization and Red Hat Gluster Storage integrated must be compatible, according to the table in Section 2.2, “Compatible Versions”.
- A fully-qualified domain name must be set for each hypervisor and Red Hat Gluster Storage server node. Ensure that correct DNS records exist, and that the fully-qualified domain name is resolvable via both forward and reverse DNS lookup.
- Red Hat Gluster Storage volumes must either use three-way replication or arbitrated replication. This reduces the risk of split-brain condition developing in the cluster. The following volume types are supported:
- arbitrated replicated or distributed arbitrated replicated volumes (
replica 3 arbiter 1)
- Server-side quorum, client-side quorum, and sharding are all required for a supported configuration. These are enabled by default in the
virttuning profile covered in Chapter 4, Hosting Virtual Machine Images on Red Hat Gluster Storage volumes. See Preventing Split-brain for information about how quorum settings help prevent split brain. See Creating Sharded Volumes for information about why sharding reduces heal and geo-replication time.
2.2. Compatible Versions
Chapter 3. Enabling Red Hat Gluster Storage in Red Hat Virtualization Manager
3.1. Using Red Hat Gluster Storage Volumes in the Red Hat Virtualization Manager
Figure 3.1. Installation Workflow
Procedure 3.1. To Enable Red Hat Gluster Storage in Red Hat Virtualization Manager
Install Red Hat Virtualization ManagerThe Red Hat Virtualization Manager is the control center of the Red Hat Virtualization environment. Ensure that Red Hat Virtualization Manager is installed and configured.Install and configure Red Hat Virtualization Manager. See Installing the Red Hat Virtualization Manager in the Red Hat Virtualization Installation Guide for more information: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_cockpit_web_interface/installing_the_red_hat_virtualization_manager_she_cockpit_deploy.
Install HostsInstall the virtual machine hosts using either Red Hat Virtualization Hypervisor or Red Hat Enterprise Linux as hosts. Before beginning the installation process, ensure that all virtualization hosts meet the hardware requirements and that the installation of Red Hat Virtualization Manager is complete.You can use the following hypervisors as hosts:
During installation, set the fully qualified domain name (FQDN) and IP address for the server in order to avoid network conflicts. For information on installing the hypervisor, see Installing Hypervisor Hosts in the Red Hat Virtualization Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_cockpit_web_interface/installing_the_self-hosted_engine_deployment_host_she_cockpit_deploy#Installing_Red_Hat_Virtualization_Hosts_SHE_deployment_host.
- Red Hat Virtualization Host 4.3
- Red Hat Enterprise Linux 7
NoteYou must install Red Hat Enterprise Linux 7 on a system to use it as a virtualization host in a Red Hat Virtualization environment. See Installing Red Hat Enterprise Linux Hosts in the Red Hat Virtualization Installation Guide for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_cockpit_web_interface/installing_hosts_for_rhv_she_cockpit_deploy#Red_Hat_Enterprise_Linux_hosts_SHE_cockpit_deploy.
Subscribe hosts to channels or repositoriesRed Hat Enterprise Linux hosts need to be registered and subscribed to a number of entitlements using either Subscription Manager or Red Hat Network Classic. For more information, see Enabling the Red Hat Virtualization Manager Repositories in the Red Hat Virtualization Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_cockpit_web_interface/installing_hosts_for_rhv_she_cockpit_deploy.Subscribe Red Hat Enterprise Linux 7 hosts to the channel listed below:
For information on how to install native clients, see Installing Native Client in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-native_client#Installing_Native_Client
Configure the Hypervisor
- Red Hat Virtualization Hypervisor host
- Install Red Hat Virtualization Hypervisor 3.5 on bare metal. See Installing the Red Hat Virtualization Hypervisor chapter of the Red Hat Virtualization Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_standalone_manager_with_local_databases/installing_hosts_for_rhv_sm_localdb_deploy
- Red Hat Enterprise Linux 7 host
- A Red Hat Enterprise Linux virtualization host must be registered and subscribed to a number of entitlements using either Subscription Manager, See Subscribing to Required Channels using Red Hat Subscription Manager in the Installing Red Hat Enterprise Linux Hosts chapter of the Red Hat Virtualization Installation Guide for more information.Subscribe to the Red Hat Gluster Storage Native Client from the Additional Services Channels to ensure the server has the latest clients installed:
yum install glusterfs-fuse glusterfsFor more information, see Installing Native Client in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_standalone_manager_with_local_databases/installing_hosts_for_rhv_sm_localdb_deploy.
Configure the data centersDefine data centers and clusters to organize the virtualization hosts. The installation process creates a data center and associated cluster named
NoteA server can be a Red Hat Virtualization Hypervisor or a Red Hat Enterprise Linux host. A trusted storage pool requires a minimum of one storage host, and at least one active virtualization host is required to connect the system to a storage pool.
Install Red Hat Gluster StorageInstall the latest version of Red Hat Gluster Storage on new servers, not the virtualization hosts.For more information on obtaining and installing the software, see the Red Hat Gluster Storage Installation Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/installation_guide/chap-installing_red_hat_storage.If you want to configure combined virtualization hosts and storage servers, see the Red Hat Hyperconverged Infrastructure documentation: https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infrastructure_for_virtualization/1.6/html-single/deploying_red_hat_hyperconverged_infrastructure_for_virtualization/
Configure StorageConfigure Red Hat Gluster Storage volumes as required by your use case, ensuring that you adhere to the Chapter 2, Support Requirements.Optimize your Red Hat Gluster Storage volumes for storing virtual machine images by following the instructions in Procedure 4.1, “Configuring Volumes Using the Command Line Interface”.Red Hat Virtualization uses a centralized storage system for virtual machine disk images. Before adding storage, ensure that you have a working Red Hat Virtualization Manager environment. You must be able to access the Administration Portal, and there must be at least one host connected with Up status.
Attach Storage to HypervisorCheck that you can access the Red Hat Virtualization Manager Administration Portal and that at least one virtualization host has
Upstatus before you continue.Create a new storage domain with a
Datadomain function and the
GlusterFSstorage type. For further information, see the Adding a Red Hat Gluster Storage Volume as a Storage Domain section in the Red Hat Virtualization Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html-single/administration_guide/index#Adding_Red_Hat_Gluster_Storage
Chapter 4. Hosting Virtual Machine Images on Red Hat Gluster Storage volumes
4.1. Configuring Volumes Using the Command Line Interface
Procedure 4.1. Configuring Volumes Using the Command Line Interface
Configure the rhgs-random-io tuned profileInstall the tuned tuning daemon and configure Red Hat Gluster Storage servers to use the
# yum install tuned # tuned-adm profile rhgs-random-ioFor more information on available tuning profiles, refer to the
tuned-admman page, or see the Red Hat Gluster Storage 3.5 Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/.
NoteWhen you upgrade, a new virt file may be created in
/var/lib/glusterd/groups/virt.rpmnew. Apply the new
virtfile to the existing volumes by renaming the
Assign volumes to virt groupAssign volumes that store virtual machine images to the
virtvolume group to apply the settings in the
virtprofile. This has the same effect as the Optimize for Virt Store option in the management console.
# gluster volume set VOLNAME group virtSee Appendix A, The
virtgroup profile for more information about this configuration.
ImportantVolumes in the
virtgroup must only be used for storing machine images, and must only be accessed using the native FUSE client.
(Recommended) Configure improved self-heal performanceRun the following command to improve the performance of volume self-heal operations.
# gluster volume heal volname cluster.granular-entry-heal enable
Allow KVM and VDSM brick accessSet the brick permissions for
kvm. If you do not set the required brick permissions, creation of virtual machines fails.
- Set the user and group permissions using the following commands:
# gluster volume set VOLNAME storage.owner-uid 36 # gluster volume set VOLNAME storage.owner-gid 36
- If you are using QEMU/KVM as a hypervisor, set the user and group permissions using the following commands:
# gluster volume set VOLNAME storage.owner-uid 107 # gluster volume set VOLNAME storage.owner-gid 107
4.2. Configuring Virtual Machines on Red Hat Gluster Storage volumes using the Red Hat Virtualization Manager
Procedure 4.2. To Add a Red Hat Gluster Storage Server for Virtualization Using Red Hat Virtualization Manager
- Create a data center:
The new data center is
- Select the Data Centers resource tab to list all data centers.
- Click New Data Center window.to open the
Figure 4.1. New Data Center Window
- Enter the Name and Description of the data center.
- Select the storage Type as
Sharedfrom the drop-down menu.
- Select the Quota Mode as
Uninitializeduntil you configure the cluster, host, and storage settings.
- Create a cluster:
- Select the Clusters resource tab to list all clusters.
- Click New Cluster window.to open the
Figure 4.2. New Cluster Window
- Select a Data Center for the cluster from the drop-down menu.
- Enter a Name and Description for the cluster.
- Select the CPU Name and Compatibility Version from the drop-down menus.
- Check Enable Virt Service.
- Add hosts:
The new host appears in the list of hypervisor hosts with the status Installing. After the host is activated, the status changes to Up automatically.
- Select the Hosts resource tab to view a list of all hosts in the system.
- Click New Host window.to open the
Figure 4.3. New Host Window
ImportantA Red Hat Enterprise Linux hypervisor and Red Hat Virtualization hypervisor on a single VDSM cluster accessing the same virtual machine image store is not supported.
- Select the Data Center and Host Cluster for the new host from the drop-down menus.
- Enter the Name, Address, and Root Password of the new hypervisor host.
- Check Automatically configure host firewall if required.
- Create and configure volumes on the Red Hat Gluster Storage cluster using the command line interface. For information on creating and configuring volumes, see Section 4.1, “Configuring Volumes Using the Command Line Interface” and Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-red_hat_storage_volumes.
- Add a storage domain using Red Hat Virtualization Manager:
- Select the Storage resource tab to list existing storage domains.
- Clickto open the New Domain window.
Figure 4.4. New Domain Window
- Enter a Name for the storage domain.
- Select a shared Data Center to associate with the storage domain.
- Set the Domain Function to
Dataand the Storage Type to
- Select a host from the Host to Use drop-down menu.
- Check the Use managed gluster volume checkbox and select the appropriate volume from the Gluster dropdown menu.
NoteThis dropdown menu is only populated with volumes whose nodes are managed by Red Hat Virtualization Manager. See Chapter 5, Managing Red Hat Gluster Storage Servers and Volumes using Red Hat Virtualization Manager for instructions on how to set up management of your Red Hat Gluster Storage nodes by Red Hat Virtualization Manager.
- Enter the applicable Red Hat Gluster Storage native client Mount Options. Enter multiple mount options separated by commas. For more information on native client mount options, see Creating Access to Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-accessing_data_-_setting_up_clients. Note that only the native client is supported when integrating Red Hat Gluster Storage and Red Hat Virtualization.
Chapter 5. Managing Red Hat Gluster Storage Servers and Volumes using Red Hat Virtualization Manager
Enable Gluster Serviceoption enabled.
5.1. Creating a Data Center
- Select the Data Centers resource tab to list all data centers in the results list.
- Click the New button to open the New Data Center window.
Figure 5.1. New Data Center Window
- Enter the
Descriptionof the data center.
Sharedfrom the drop-down menu.Set Quota Mode as
Uninitializeduntil a cluster, host, and storage are configured.
5.2. Creating a Cluster
- Create a cluster with the gluster service enabled.
Figure 5.2. New Cluster Window
- Select the
Compatibility Versionfrom the drop-down menu.
iptablesfor the option
5.3. Adding Red Hat Gluster Storage Server to the Cluster
- To import an existing gluster configuration:
- Select Enable Gluster Service.
- Select Import existing gluster configuration. With this option you can import the existing Gluster configurations into a cluster.
- Provide the IP address of one of the hosts.
- To add new hosts:
You can manage the lifecycle of a volume using hook scripts.
- Use the drop-down lists to select the Data Center and Host Cluster for the new host.
- Click.The new host displays in the list of hosts with a status of
Installing. The host is activated and the status changes to Up automatically.
# yum install cockpit, before adding the Red Hat Gluster Storage 3.5 node to Red Hat Virtualization Manager 4.3 in 4.3 compatible cluster.
NoteTo add multiple servers to a cluster, you must first add a Red Hat Gluster Storage server to the cluster. An error message appears if you add multiple servers in the first attempt.
Figure 5.3. New Host window
5.4. Optimizing Red Hat Gluster Storage Volumes for Virtual Machine Images
- On the Volumes tab, select the volume to configure and click Optimize for Virt Store.The volume is configured and
storage-owner-gidoptions are set.
Figure 5.4. Volume OptionsFor more information on each
volume settunable options, see Managing Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-managing_red_hat_storage_volumes.
5.5. Starting the Red Hat Gluster Storage Volume
- In the Volumes tab, select the new volume and click to start the volume.The volume status changes to Up.
ImportantIf you are using Red Hat Gluster Storage Volumes for hosting Virtual Machine images, you must optimize the volume before attaching the volume to the storage domain.
5.6. Adding Bricks to Volumes
Add Bricksto add bricks to your volume.A brick is the basic unit of storage, represented by an export directory on a server in the storage cluster. You can expand or shrink your cluster by adding new bricks or deleting existing bricks.
Figure 5.5. Add Bricks
- Enter the path for the brick and click OK.
- In the Allow Access From field, specify volume access control as a comma-separated list of IP addresses or hostnames.By default, an asterisk (*) is used as a wildcard to specify ranges of addresses such as IP addresses or hostnames. You need to use IP-based authentication for Gluster exports.
- Click OK to create the volume.The new volume is added and it appears on thetab.
Allow bricks in root partition and reuse the bricks by clearing xattrs
5.7. Performing Asynchronous Tasks
Appendix A. The
virt group profile
/var/lib/glusterd/groups/virtfile defines a configuration that is used for all volumes in the
virtgroup. Defining a group configuration and adding volumes to a group helps save time and effort when you want to update the settings on all volumes in the group.
/var/lib/glusterd/groups/virtfile are as follows:
performance.quick-read=off performance.read-ahead=off performance.io-cache=off performance.low-prio-threads=32 network.remote-dio=enable cluster.eager-lock=enable cluster.quorum-type=auto cluster.server-quorum-type=server cluster.data-self-heal-algorithm=full cluster.locking-scheme=granular cluster.shd-max-threads=8 cluster.shd-wait-qlength=10000 features.shard=on user.cifs=off cluster.choose-local=off client.event-threads=4 server.event-threads=4 performance.client-io-threads=on
cluster.eager-lockoption optimizes write performance by using synchronous replication when there is a single writer to a file.
cluster.data-self-healoption specifies how self-heal operations are performed.
features.shard-block-sizeoptions enable sharding behavior in the volume and set a block size appropriate for storing disk images.
cluster.server-quorum-typeoptions enable client and server-side quorum to help ensure consistency and to prevent split-brain scenarios.
cluster.server-quorum-type=serverprovide consistency at the cost of fault tolerance. Using these settings, if server-side quorum is not met, virtual machines pause because volumes are not available. If client-side quorum is not met, virtual machines pause even though a read-only replica pair is available. Manual intervention is then required to make the VMs resume the operations after the quorum is restored.
# gluster volume reset volname server-quorum-type # gluster volume reset volname quorum-type
Appendix B. Revision History
|Revision 3.5-2||Tue Sep 04 2018|
|Revision 3.5-1||Tue Feb 06 2018|