-
Language:
English
-
Language:
English
Red Hat Training
A Red Hat training course is available for Red Hat Gluster Storage
Configuring Red Hat Enterprise Virtualization with Red Hat Gluster Storage
Getting Started with Red Hat Gluster Storage 3.1 and Red Hat Enterprise Virtualization 3.4 or later.
Abstract
Preface
Chapter 1. Introduction to Red Hat Gluster Storage for Virtualization
- Red Hat Enterprise Virtualization environment.See the Red Hat Enterprise Virtualization Technical Reference for an architectural overview: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Technical_Reference/index.html
- Red Hat Gluster Storage environment.A Red Hat Gluster Storage environment consists of bricks that are used to build volumes. Different types of optimized storage volumes can be used as virtual machine stores to hold resources such as virtual disks, templates, and snapshots.
Figure 1.1. Integration architecture
Important
libgfapi
protocol in the future releases.
Chapter 2. Support Requirements
2.1. Prerequisites
- All installations of Red Hat Enterprise Virtualization and Red Hat Gluster Storage must have valid subscriptions to Red Hat Network channels and Subscription Management repositories.
- Red Hat Enterprise Virtualization installations must adhere to the requirements laid out in the Red Hat Enterprise Virtualization Installation Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/chap-System_Requirements.html.
- Red Hat Gluster Storage installations must adhere to the requirements laid out in the Red Hat Gluster Storage Installation Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Installation_Guide/chap-Planning_Red_Hat_Storage_Installation.html.
- Red Hat Gluster Storage installations must be completely up to date with the latest patches and upgrades. Refer to the Red Hat Gluster Storage 3.1 Installation Guide to upgrade to the latest version: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Installation_Guide/index.html.
- The versions of Red Hat Enterprise Virtualization and Red Hat Gluster Storage integrated must be compatible, according to the table in Section 2.2, “Compatible Versions”.
- If you are using a QEMU/KVM hypervisor, ensure that the following SELinux booleans are enabled.
# setsebool -P sanlock_use_fusefs on # setsebool -P virt_use_sanlock on # setsebool -P virt_use_fusefs on
2.2. Compatible Versions
Table 2.1. Supported version information for Red Hat Gluster Storage
RHGS version | gluster version | gluster op-version | RHEV version |
---|---|---|---|
3.0 | 3.6.0.28-1 | 30000 | 3.4 |
3.0 Update 1 | 3.6.0.29-1 | 30000 | 3.4 |
3.0 Update 2 | 3.6.0.29-3 | 30000 | 3.4 |
3.0 Update 3 | 3.6.0.42-1 | 30000 | 3.4 |
3.0 Update 4 | 3.6.0.53-1 | 30004 | 3.5 |
3.1 | 3.7.1-11 | 30703 | 3.5, 3.6 |
3.1 Update 1 | 3.7.1-16 | 30703 | 3.5, 3.6 |
3.1 Update 2 | 3.7.5-19 | 30707 | 3.5, 3.6 |
Table 2.2. Cluster compatibility matrix
RHEV Version | RHGS Version | Cluster Compatibility |
---|---|---|
3.4 | 3.0.z | 3.4 |
3.1.z | 3.4 | |
3.5 | 3.0, 3.0.1, 3.0.2, 3.0.3 | 3.4 |
3.0.4 | 3.5 | |
3.1.z | 3.5 | |
3.6 | 3.0.z | 3.4 |
3.1.z | 3.5 |
Chapter 3. Enabling Red Hat Gluster Storage in Red Hat Enterprise Virtualization Manager
3.1. Using Red Hat Gluster Storage Volumes in the Red Hat Enterprise Virtualization Manager
Figure 3.1. Installation Workflow
Procedure 3.1. To Enable Red Hat Gluster Storage in Red Hat Enterprise Virtualization Manager
Install Red Hat Enterprise Virtualization Manager
The Red Hat Enterprise Virtualization Manager is the control center of the Red Hat Enterprise Virtualization environment. Ensure that Red Hat Enterprise Virtualization Manger is installed and configured.Install and configure Red Hat Enterprise Virtualization Manager. See Installing the Red Hat Enterprise Virtualization Manager in the Red Hat Enterprise Virtualization Installation Guide for more information: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/part-Installing_Red_Hat_Enterprise_Virtualization.html.Install Hosts
Install the virtual machine hosts using either Red Hat Enterprise Virtualization Hypervisor or Red Hat Enterprise Linux as hosts. Before beginning the installation process, ensure that all virtualization hosts meet the hardware requirements and that the installation of Red Hat Enterprise Virtualization Manager is complete.You can use the following hypervisors as hosts:- Red Hat Enterprise Virtualization Hypervisor 3.4
- Red Hat Enterprise Linux 6.3
- Red Hat Enterprise Linux 6.4
- Red Hat Enterprise Linux 6.5
During installation, set the fully qualified domain name (FQDN) and IP address for the server in order to avoid network conflicts. For information on installing the hypervisor, see Installing Hypervisor Hosts in the Red Hat Enterprise Virtualization Installation Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/part-Installing_Hypervisor_Hosts.html.Note
You must install Red Hat Enterprise Linux 6.3, Red Hat Enterprise Linux 6.4 Server or Red Hat Enterprise Linux 6.5 on a system to use it as a virtualization host in a Red Hat Enterprise Virtualization environment. See Installing Red Hat Enterprise Linux Hosts in the Red Hat Enterprise Virtualization Installation Guide for details: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/chap-Red_Hat_Enterprise_Linux_Hosts.html.Subscribe hosts to channels or repositories
Red Hat Enterprise Linux hosts need to be registered and subscribed to a number of entitlements using either Subscription Manager or Red Hat Network Classic. For more information, see Subscribing to Required Channels using RHN Classic in the Red Hat Enterprise Virtualization Installation Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/Subscribing_to_Required_Channels_using_Subscription_Manager1.html.Subscribe Red Hat Enterprise Linux 6.3, 6.4, and 6.5 hosts to the channels listed below:- rhel-x86_64-server-6
- rhel-x86_64-rhev-mgmt-agent-6
Apart from the above channels, you must also subscribe to the Red Hat Gluster Storage 3.1 Native Client channels to get the latest version:Run therhn-channel --add --channel
command to subscribe the system to the correct Red Hat Gluster Storage Native Client channel:- For Red Hat Enterprise Linux 7.x clients using Red Hat Satellite Server:
# rhn-channel --add --channel= rhel-x86_64-server-rh-common-7
- For Red Hat Enterprise Linux 6.x clients:
# rhn-channel --add --channel=rhel-x86_64-server-rhsclient-6
- For Red Hat Enterprise Linux 5.x clients:
# rhn-channel --add --channel=rhel-x86_64-server-rhsclient-5
For information on how to install native clients, see Installing Native Client in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Accessing_Data_-_Setting_Up_Clients.html#Installing_Native_ClientConfigure the Hypervisor
- Red Hat Enterprise Virtualization Hypervisor host
- Install Red Hat Enterprise Virtualization Hypervisor 3.4 on bare metal. See Installing the Red Hat Enterprise Virtualization Hypervisor chapter of the Red Hat Enterprise Virtualization Installation Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/part-Installing_Hypervisor_Hosts.html
- Red Hat Enterprise Linux 6.3, 6.4, or 6.5 host
- A Red Hat Enterprise Linux virtualization host must be registered and subscribed to a number of entitlements using either Subscription Manager or Red Hat Network Classic. See Subscribing to Required Channels using RHN Classic in the Installing Red Hat Enterprise Linux Hosts chapter of the Red Hat Enterprise Virtualization Installation Guide for more information.Subscribe to the Red Hat Gluster Storage Native Client from the Additional Services Channels to ensure the server has the latest clients installed:
#
yum install glusterfs-fuse glusterfs
For more information, see Installing Native Client in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/chap-Red_Hat_Enterprise_Linux_Hosts.html.
Configure the data centers
Define data centers and clusters to organize the virtualization hosts. The installation process creates a data center and associated cluster namedDefault
.Note
A server can be a Red Hat Enterprise Virtualization Hypervisor or a Red Hat Enterprise Linux host. A trusted storage pool requires a minimum of one storage host, and at least one active virtualization host is required to connect the system to a storage pool.Install Red Hat Gluster Storage
Install the latest version of Red Hat Gluster Storage on the system.For more information on obtaining and installing the software, see the Red Hat Gluster Storage Installation Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Installation_Guide/chap-Installing_Red_Hat_Storage.html.Configure Storage
Configure the Red Hat Gluster Storage installation for your requirements.For instructions on creating a Red Hat Gluster Storage Volume, see chapter Red Hat Gluster Storage Volume in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Red_Hat_Storage_Volumes.html.Red Hat Enterprise Virtualization uses a centralized storage system for virtual machine disk images. Before adding storage, ensure that you have a working Red Hat Enterprise Virtualization Manager environment. You must be able to access the Administration Portal, and there must be at least one host connected with Up status.Note
Red Hat Gluster Storage recommends that you use Replicate or Distributed Replicate volume type for hosting virtual machine disk images.Attach Storage to Hypervisor
Attach the Red Hat Gluster Storage volume as a storage domain.
3.2. Optimizing Virtual Machines On Red Hat Gluster Storage Volumes
multiply_disk_readahead 4
to multiply_disk_readahead 16
. If you are running a Red Hat Enterprise Linux host, run the following commands to enable read-ahead.
Install the Tune daemon
Install the tuned daemon that tunes the system settings dynamically.#
yum install tuned
Edit the virtual-guest profile
Edit the/etc/tune-profiles/virtual-guest/ktune.sh
file.Identify the linemultiply_disk_readahead 4
and replace it withmultiply_disk_readahead 16
Apply the virtual-guest profile on the virtual machines
Activate the updated profile with the command:#
tuned-adm profile virtual-guest
Chapter 4. Hosting Virtual Machine Images on Red Hat Gluster Storage volumes
4.1. Configuring Volumes Using the Command Line Interface
Important
replica 3
or replica count 3
) is recommended for hosting virtual machine images.
Procedure 4.1. To Configure Volumes Using the Command Line Interface
- Configure the Red Hat Gluster Storage volume using the following command:
#
gluster volume set VOLNAME group virt
Important
After tagging the volume asgroup virt
, use the volume for storing virtual machine images only and always access the volume through the glusterFS native client. - The
gluster volume set <VOLNAME> group virt
command configures the specified volume using the settings in the/var/lib/glusterd/groups/virt
file. The following settings are the recommended defaults:quick-read=off read-ahead=off io-cache=off stat-prefetch=off eager-lock=enable remote-dio=on quorum-type=auto server-quorum-type=server
Important
When you upgrade to Red Hat Gluster Storage 3.1, a new virt file is created in/var/lib/glusterd/groups/virt.rpmnew
. Ensure to apply the newvirt
file on the existing volumes by renaming thevirt.rpmnew
file tovirt
, along with the customized settings.All the performance-related settings other thancluster.eager-lock
prevent caching within GlusterFS client stack, as it is the preferred mode for attaching disks to a virtual machine. Thecluster.eager-lock
option optimizes write performance with synchronous replication when there is a single writer to a file. For description on each of these performance-related settings, see chapter Configuring Volume Options in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Managing_Red_Hat_Storage_Volumes.html#Configuring_Volume_Options.Note
Server-Side and Client-Side Quorum are enabled by default in the/var/lib/glusterd/groups/virt
file to minimize split-brain scenarios. If Server-Side Quorum is not met, then the Red Hat Gluster Storage volumes become unavailable causing the Virtual Machines (VMs) to move to a paused state. If Client-Side Quorum is not met, although a replica pair in a Red Hat Gluster Storage volume is available in the read-only mode, the VMs move to a paused state.Manual intervention is required to make the VMs resume the operations after the quorum is restored. Consistency is achieved at the cost of fault tolerance. If fault tolerance is preferred over consistency, disable server-side and client-side quorum with the commands:# gluster volume reset <vol-name> server-quorum-type # gluster volume reset <vol-name> quorum-type
For more information on these configuration settings, see Managing Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Managing_Red_Hat_Storage_Volumes.html. - Set the brick permissions for
vdsm
andkvm
. If you do not set the required brick permissions, creation of virtual machines fails.- Set the user and group permissions using the following commands:
# gluster volume set VOLNAME storage.owner-uid 36 # gluster volume set VOLNAME storage.owner-gid 36
- If you are using QEMU/KVM as a hypervisor, set the user and group permissions using the following commands:
# gluster volume set VOLNAME storage.owner-uid 107 # gluster volume set VOLNAME storage.owner-gid 107
4.2. Configuring Virtual Machines on Red Hat Gluster Storage volumes using the Red Hat Enterprise Virtualization Manager
Note
Procedure 4.2. To Add a Red Hat Gluster Storage Server for Virtualization Using Red Hat Enterprise Virtualization Manager
- Create a data center:
- Select the Data Centers resource tab to list all data centers.
- Click New to open the New Data Center window.
Figure 4.1. New Data Center Window
- Enter the Name and Description of the data center.
- Select the storage Type as
Shared
from the drop-down menu. - Select the Quota Mode as
Disabled
. - Click OK.
The new data center isUninitialized
until you configure the cluster, host, and storage settings. - Create a cluster:
- Select the Clusters resource tab to list all clusters.
- Click New to open the New Cluster window.
Figure 4.2. New Cluster Window
- Select a Data Center for the cluster from the drop-down menu.
- Enter a Name and Description for the cluster.
- Select the CPU Name and Compatibility Version from the drop-down menus.
- Check Enable Virt Service.
- Click OK.
- Add hosts:
- Select the Hosts resource tab to view a list of all hosts in the system.
- Click New to open the New Host window.
Figure 4.3. New Host Window
Important
A Red Hat Enterprise Linux hypervisor and Red Hat Enterprise Virtualization hypervisor on a single VDSM cluster accessing the same virtual machine image store is not supported. - Select the Data Center and Host Cluster for the new host from the drop-down menus.
- Enter the Name, Address, and Root Password of the new hypervisor host.
- Check Automatically configure host firewall if required.
- Click OK.
The new host appears in the list of hypervisor hosts with the status Installing. After the host is activated, the status changes to Up automatically. - Create and configure volumes on the Red Hat Gluster Storage cluster using the command line interface. For information on creating and configuring volumes, see Section 4.1, “Configuring Volumes Using the Command Line Interface” and Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Red_Hat_Storage_Volumes.html.
- Add a storage domain using Red Hat Enterprise Virtualization Manager:
- Select the Storage resource tab to list existing storage domains.
- Click New Domain to open the New Domain window.
Figure 4.4. New Domain Window
- Enter a Name for the storage domain.
- Select a shared Data Center to associate with the storage domain.
- Select
Data / glusterFS
from the Domain Function/Storage Type drop-down menu. - Select a host from the Use Host drop-down menu.
- Enter the Path of a Red Hat Gluster Storage server in the cluster. For example,
RHS-Server:/VOLNAME
- Enter
glusterfs
as VFS Type. This selection is made by default if the data center is of typeglusterFS
. - Enter the applicable Red Hat Gluster Storage native client Mount Options. Enter multiple mount options separated by commas. For more information on native client mount options, see Accessing Data - Setting up Clients in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Accessing_Data_-_Setting_Up_Clients.html.
- Click OK.
- Click Activate to activate the new Red Hat Gluster Storage domain.
Figure 4.5. Activated Red Hat Gluster Storage Domain
Note
Chapter 5. Managing Red Hat Gluster Storage Servers and Volumes using Red Hat Enterprise Virtualization Manager
Enable Gluster Service
option enabled.
Note
5.1. Creating a Data Center
- Select the Data Centers resource tab to list all data centers in the results list.
- Click the New button to open the New Data Center window.
Figure 5.1. New Data Center Window
- Enter the
Name
andDescription
of the data center. - Set
Type
toShared
from the drop-down menu.Set Quota Mode asDisabled
. - Click
OK
.
Uninitialized
until a cluster, host, and storage are configured.
5.2. Creating a Cluster
- Create a cluster with the gluster service enabled.
Figure 5.2. New Cluster Window
- Select the
Compatibility Version
from the drop-down menu. - Click OK.
5.3. Adding Red Hat Gluster Storage Server to the Cluster
- To import an existing gluster configuration:
- Select Enable Gluster Service.
- Select Import existing gluster configuration. With this option you can import the existing Gluster configurations into a cluster.
- Provide the IP address of one of the hosts.
- To add new hosts:
- Use the drop-down lists to select the Data Center and Host Cluster for the new host.
- Click OK.The new host displays in the list of hosts with a status of
Installing
. The host is activated and the status changes to Up automatically.
You can manage the lifecycle of a volume using hook scripts. For more information, see chapter Managing Gluster Hooks in the Red Hat Gluster Storage Console Administration GuideNote
To add multiple servers to a cluster, you must first add a Red Hat Gluster Storage server to the cluster. An error message appears if you add multiple servers in the first attempt.Figure 5.3. New Host window
5.4. Optimizing Red Hat Gluster Storage Volumes for Virtual Machine Images
- On the Volumes tab, select the volume to configure and click Optimize for Virt Store.The volume is configured and
group
,storage-owner-uid
, andstorage-owner-gid
options are set.Figure 5.4. Volume Options
For more information on eachvolume set
tunable options, see Managing Red Hat Gluster Storage Volumes in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Managing_Red_Hat_Storage_Volumes.html.
5.5. Starting the Red Hat Gluster Storage Volume
- In the Volumes tab, select the new volume and click Start to start the volume.The volume status changes to Up.
Note
If you are using Red Hat Gluster Storage Volumes for hosting Virtual Machine images, you must optimize the volume before attaching the volume to the storage domain.
5.6. Adding Bricks to Volumes
- Click
Add Bricks
to add bricks to your volume.A brick is the basic unit of storage, represented by an export directory on a server in the storage cluster. You can expand or shrink your cluster by adding new bricks or deleting existing bricks.Figure 5.5. Add Bricks
- Enter the path for the brick and click OK.
- In the Allow Access From field, specify volume access control as a comma-separated list of IP addresses or hostnames.By default, an asterisk (*) is used as a wildcard to specify ranges of addresses such as IP addresses or hostnames. You need to use IP-based authentication for Gluster and NFS exports.
- Click OK to create the volume.The new volume is added and it appears on the Volumes tab.
Allow bricks in root partition and reuse the bricks by clearing xattrs
5.7. Performing Asynchronous Tasks
Chapter 6. Recommended Best Practices and Troubleshooting
- Enable the server side quorum at all times for the Red Hat Gluster Storage volumes hosting virtual machine images. For more information see Configuring Server-Side Quorum in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-Managing_Split-brain.html#sect-Preventing_Split-brain.
- Red Hat Gluster Storage Volumes that serve as Red Hat Enterprise Virtualization Master Storage Domain contain SANLocks and have the Red Hat Gluster Storage volume profile set to
virt
. This enables the Client Side Quorum and avoids the inconsistency problems for Multiple Writer situations. For more information see Configuring Client-Side Quorum in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-Managing_Split-brain.html#sect-Preventing_Split-brain. - To reduce the risk of split-brain conditions developing in the cluster, Red Hat recommends using three-way replication when you create Red Hat Gluster Storage volumes. This involves setting
replica 3
when creating replicate volumes, orreplica count 3
when creating distributed-replicated volumes.
6.1. Troubleshooting the Integration
- Red Hat Gluster Storage 3.1 Volumes do not get mounted with Red Hat Enterprise Virtualization Hypervisor 3.3 and 3.4 versions as the glusterfs native clients available in those Hypervisors are not compatible with Red Hat Gluster Storage 3.1 servers.In order to make use of Red Hat Gluster Storage 3.1 volumes in Red Hat Enterprise Virtualization Hypervisor 3.3 and 3.4 versions, disable the readdir-ahead option on those volumes. This makes the volumes compatible with the older native client available with Red Hat Enterprise Virtualization Hypervisor 3.3 and 3.4.
# gluster volume set <vol> readdir-ahead off
Appendix A. Revision History
Revision History | ||||
---|---|---|---|---|
Revision 3.1-11 | Wed Mar 09 2016 | Laura Bailey | ||
| ||||
Revision 3.1-10 | Tue Mar 08 2016 | Laura Bailey | ||
| ||||
Revision 3.1-9 | Mon Feb 22 2016 | Laura Bailey | ||
| ||||
Revision 3.1-8 | Tue Dec 15 2015 | Laura Bailey | ||
| ||||
Revision 3.1-7 | Tue Dec 08 2015 | Laura Bailey | ||
| ||||
Revision 3.1-5 | Tue Nov 03 2015 | Laura Bailey | ||
| ||||
Revision 3.1-4 | Tue Oct 27 2015 | Laura Bailey | ||
| ||||
Revision 3.1-3 | Thu Oct 09 2015 | Laura Bailey | ||
| ||||
Revision 3.1-1 | Tue July 28 2015 | Ella Deon Ballard | ||
| ||||
Revision 3-24 | Tue Jul 28 2015 | Rakesh Ghatvisave | ||
| ||||
Revision 3-23 | Wed Jul 22 2015 | Rakesh Ghatvisave | ||
| ||||
Revision 3-22 | Fri Mar 13 2015 | Pavithra Srinivasan | ||
| ||||
Revision 3-20 | Wed Feb 11 2015 | Pavithra Srinivasan | ||
| ||||
Revision 3-12 | Mon Oct 20 2014 | Pavithra Srinivasan | ||
|