Chapter 5. Preparing Storage for Red Hat Virtualization

You need to prepare storage to be used for storage domains in the new environment. A Red Hat Virtualization environment must have at least one data storage domain, but adding more is recommended.

Warning

When installing or reinstalling the host’s operating system, Red Hat strongly recommends that you first detach any existing non-OS storage that is attached to the host to avoid accidental initialization of these disks, and with that, potential data loss.

A data domain holds the virtual hard disks and OVF files of all the virtual machines and templates in a data center, and cannot be shared across data centers while active (but can be migrated between data centers). Data domains of multiple storage types can be added to the same data center, provided they are all shared, rather than local, domains.

You can use one of the following storage types:

5.1. Preparing NFS Storage

Set up NFS shares on your file storage or remote server to serve as storage domains on Red Hat Enterprise Virtualization Host systems. After exporting the shares on the remote storage and configuring them in the Red Hat Virtualization Manager, the shares will be automatically imported on the Red Hat Virtualization hosts.

For information on setting up, configuring, mounting and exporting NFS, see Managing file systems for Red Hat Enterprise Linux 8.

Specific system user accounts and system user groups are required by Red Hat Virtualization so the Manager can store data in the storage domains represented by the exported directories. The following procedure sets the permissions for one directory. You must repeat the chown and chmod steps for all of the directories you intend to use as storage domains in Red Hat Virtualization.

Prerequisites

  1. Install the NFS utils package.

    # dnf install nfs-utils -y
  2. To check the enabled versions:

    # cat /proc/fs/nfsd/versions
  3. Enable the following services:

    # systemctl enable nfs-server
    # systemctl enable rpcbind

Procedure

  1. Create the group kvm:

    # groupadd kvm -g 36
  2. Create the user vdsm in the group kvm:

    # useradd vdsm -u 36 -g kvm
  3. Create the storage directory and modify the access rights.

    # mkdir /storage
    # chmod 0755 /storage
    # chown 36:36 /storage/
  4. Add the storage directory to /etc/exports with the relevant permissions.

    # vi /etc/exports
    # cat /etc/exports
     /storage *(rw)
  5. Restart the following services:

    # systemctl restart rpcbind
    # systemctl restart nfs-server
  6. To see which export are available for a specific IP address:

    # exportfs
     /nfs_server/srv
                   10.46.11.3/24
     /nfs_server       <world>
Note

If changes in /etc/exports have been made after starting the services, the exportfs -ra command can be used to reload the changes. After performing all the above stages, the exports directory should be ready and can be tested on a different host to check that it is usable.

5.2. Preparing iSCSI Storage

Red Hat Virtualization supports iSCSI storage, which is a storage domain created from a volume group made up of LUNs. Volume groups and LUNs cannot be attached to more than one storage domain at a time.

For information on setting up and configuring iSCSI storage, see Getting started with iSCSI in Managing storage devices for Red Hat Enterprise Linux 8.

Important

If you are using block storage and you intend to deploy virtual machines on raw devices or direct LUNs and to manage them with the Logical Volume Manager, you must create a filter to hide the guest logical volumes. This will prevent guest logical volumes from being activated when the host is booted, a situation that could lead to stale logical volumes and cause data corruption. See https://access.redhat.com/solutions/2662261 for details.

Important

Red Hat Virtualization currently does not support block storage with a block size of 4K. You must configure block storage in legacy (512b block) mode.

Important

If your host is booting from SAN storage and loses connectivity to the storage, the storage file systems become read-only and remain in this state after connectivity is restored.

To prevent this situation, add a drop-in multipath configuration file on the root file system of the SAN for the boot LUN to ensure that it is queued when there is a connection:

# cat /etc/multipath/conf.d/host.conf
multipaths {
    multipath {
        wwid boot_LUN_wwid
        no_path_retry queue
    }

5.3. Preparing FCP Storage

Red Hat Virtualization supports SAN storage by creating a storage domain from a volume group made of pre-existing LUNs. Neither volume groups nor LUNs can be attached to more than one storage domain at a time.

Red Hat Virtualization system administrators need a working knowledge of Storage Area Networks (SAN) concepts. SAN usually uses Fibre Channel Protocol (FCP) for traffic between hosts and shared external storage. For this reason, SAN may occasionally be referred to as FCP storage.

For information on setting up and configuring FCP or multipathing on Red Hat Enterprise Linux, see the Storage Administration Guide and DM Multipath Guide.

Important

If you are using block storage and you intend to deploy virtual machines on raw devices or direct LUNs and to manage them with the Logical Volume Manager, you must create a filter to hide the guest logical volumes. This will prevent guest logical volumes from being activated when the host is booted, a situation that could lead to stale logical volumes and cause data corruption. See https://access.redhat.com/solutions/2662261 for details.

Important

Red Hat Virtualization currently does not support block storage with a block size of 4K. You must configure block storage in legacy (512b block) mode.

Important

If your host is booting from SAN storage and loses connectivity to the storage, the storage file systems become read-only and remain in this state after connectivity is restored.

To prevent this situation, add a drop-in multipath configuration file on the root file system of the SAN for the boot LUN to ensure that it is queued when there is a connection:

# cat /etc/multipath/conf.d/host.conf
multipaths {
    multipath {
        wwid boot_LUN_wwid
        no_path_retry queue
    }

5.4. Preparing POSIX-compliant File System Storage

POSIX file system support allows you to mount file systems using the same mount options that you would normally use when mounting them manually from the command line. This functionality is intended to allow access to storage not exposed using NFS, iSCSI, or FCP.

Any POSIX-compliant file system used as a storage domain in Red Hat Virtualization must be a clustered file system, such as Global File System 2 (GFS2), and must support sparse files and direct I/O. The Common Internet File System (CIFS), for example, does not support direct I/O, making it incompatible with Red Hat Virtualization.

For information on setting up and configuring POSIX-compliant file system storage, see Red Hat Enterprise Linux Global File System 2.

Important

Do not mount NFS storage by creating a POSIX-compliant file system storage domain. Always create an NFS storage domain instead.

5.5. Preparing Local Storage

A local storage domain can be set up on a host. When you set up a host to use local storage, the host is automatically added to a new data center and cluster that no other hosts can be added to. Multiple-host clusters require that all hosts have access to all storage domains, which is not possible with local storage. Virtual machines created in a single-host cluster cannot be migrated, fenced, or scheduled.

Important

On Red Hat Virtualization Host (RHVH), local storage should always be defined on a file system that is separate from / (root). Use a separate logical volume or disk, to prevent possible loss of data during upgrades.

Preparing Local Storage for Red Hat Enterprise Linux hosts

  1. On the host, create the directory to be used for the local storage:

    # mkdir -p /data/images
  2. Ensure that the directory has permissions allowing read/write access to the vdsm user (UID 36) and kvm group (GID 36):

    # chown 36:36 /data /data/images
    # chmod 0755 /data /data/images

Preparing Local Storage for Red Hat Virtualization Hosts

Create the local storage on a logical volume:

  1. Create a local storage directory:

    # mkdir /data
    # lvcreate -L $SIZE rhvh -n data
    # mkfs.ext4 /dev/mapper/rhvh-data
    # echo "/dev/mapper/rhvh-data /data ext4 defaults,discard 1 2" >> /etc/fstab
    # mount /data
  2. Mount the new local storage, and then modify the permissions and ownership:

    # mount -a
    # chown 36:36 /data /rhvh-data
    # chmod 0755 /data /rhvh-data

5.6. Preparing Red Hat Gluster Storage

For information on setting up and configuring Red Hat Gluster Storage, see the Red Hat Gluster Storage Installation Guide.

For the Red Hat Gluster Storage versions that are supported with Red Hat Virtualization, see https://access.redhat.com/articles/2356261.

5.7. Customizing Multipath Configurations for SAN Vendors

If your RHV environment is configured to use multipath connections with SANs, you can customize the multipath configuration settings to meet requirements specified by your storage vendor. These customizations can override both the default settings and settings that are specified in /etc/multipath.conf.

To override the multipath settings, do not customize /etc/multipath.conf. Because VDSM owns /etc/multipath.conf, installing or upgrading VDSM or Red Hat Virtualization can overwrite this file including any customizations it contains. This overwriting can cause severe storage failures.

Instead, you create a file in the /etc/multipath/conf.d/user.conf directory that contains the settings you want to customize or override.

VDSM executes the files in /etc/multipath/conf.d/user.conf in alphabetical order. So, to control the order of execution, you begin the filename with a number that makes it come last. For example, /etc/multipath/conf.d/user.conf/90-myfile.conf.

To avoid causing severe storage failures, follow these guidelines:

Warning

Not following these guidelines can cause catastrophic storage errors.

Prerequisites

  • VDSM is configured to use the multipath module. To verify this, enter:

    # vdsm-tool is-configured --module multipath

Procedure

  1. Create a new configuration file in the /etc/multipath/conf.d/user.conf directory.
  2. Copy the individual setting you want to override from /etc/multipath.conf to the new configuration file in /etc/multipath/conf.d/user.conf. Remove any comment marks, edit the setting values, and save your changes.
  3. Apply the new configuration settings by entering:

    # systemctl reload multipathd
    Note

    Do not restart the multipathd service. Doing so generates errors in the VDSM logs.

Verification steps

Test the new configuration performs as expected on a non-production cluster in a variety of failure scenarios. For example, disable all of the storage connections. Then enable one connection at a time and verify that doing so makes the storage domain reachable.