Prerequisites

• Virtualization is enabled on your host system.
• You have a sufficient amount of system resources to allocate to your VMs, such as disk space, RAM, or CPUs. The recommended values may vary significantly depending on the intended tasks and workload of the VMs.

• An operating system (OS) installation source is available locally or on a network. This can be one of the following:

  • An ISO image of an installation medium

  • A disk image of an existing VM installation

    Warning

    Installing from a host CD-ROM or DVD-ROM device is not possible in RHEL 8. If you select a CD-ROM or DVD-ROM as the installation source when using any VM installation method available in RHEL 8, the installation will fail. For more information, see the Red Hat Knowledgebase.

    Also note that Red Hat provides support only for a limited set of guest operating systems.

• Optional: A Kickstart file can be provided for faster and easier configuration of the installation.

• --name: the name of the new machine
• --memory: the amount of allocated memory
• --vcpus: the number of allocated virtual CPUs
• --disk: the type and size of the allocated storage
• --cdrom or --location: the type and location of the OS installation source

• The following command creates a VM named demo-guest1 that installs the Windows 10 OS from an ISO image locally stored in the /home/username/Downloads/Win10install.iso file. This VM is also allocated with 2048 MiB of RAM and 2 vCPUs, and an 80 GiB qcow2 virtual disk is automatically configured for the VM.

  # virt-install \
      --graphics vnc \
      --name demo-guest1 --memory 2048 \
      --vcpus 2 --disk size=80 --os-variant win10 \
      --cdrom /home/username/Downloads/Win10install.iso

• The following command creates a VM named demo-guest2 that uses the /home/username/Downloads/rhel8.iso image to run a RHEL 8 OS from a live CD. No disk space is assigned to this VM, so changes made during the session will not be preserved. In addition, the VM is allocated with 4096 MiB of RAM and 4 vCPUs.

  # virt-install \
      --graphics vnc \
      --name demo-guest2 --memory 4096 --vcpus 4 \
      --disk none --livecd --os-variant rhel8.0 \
      --cdrom /home/username/Downloads/rhel8.iso

• The following command creates a RHEL 8 VM named demo-guest3 that connects to an existing disk image, /home/username/backup/disk.qcow2. This is similar to physically moving a hard drive between machines, so the OS and data available to demo-guest3 are determined by how the image was handled previously. In addition, this VM is allocated with 2048 MiB of RAM and 2 vCPUs.

  # virt-install \
      --graphics vnc \
      --name demo-guest3 --memory 2048 --vcpus 2 \
      --os-variant rhel8.0 --import \
      --disk /home/username/backup/disk.qcow2

  Note that the --os-variant option is highly recommended when importing a disk image. If it is not provided, the performance of the created VM will be negatively affected.

• The following command creates a VM named demo-guest4 that installs from the http://example.com/OS-install URL. For the installation to start successfully, the URL must contain a working OS installation tree. In addition, the OS is automatically configured by using the /home/username/ks.cfg kickstart file. This VM is also allocated with 2048 MiB of RAM, 2 vCPUs, and a 160 GiB qcow2 virtual disk.

  # virt-install \
      --graphics vnc \
      --name demo-guest4 --memory 2048 --vcpus 2 --disk size=160 \
      --os-variant rhel8.0 --location http://example.com/OS-install \
      --initrd-inject /home/username/ks.cfg --extra-args="inst.ks=file:/ks.cfg console=tty0 console=ttyS0,115200n8"

• The following command creates a VM named demo-guest5 that installs from a RHEL8.iso image file in text-only mode, without graphics. It connects the guest console to the serial console. The VM has 16384 MiB of memory, 16 vCPUs, and 280 GiB disk. This kind of installation is useful when connecting to a host over a slow network link.

  # virt-install \
      --name demo-guest5 --memory 16384 --vcpus 16 --disk size=280 \
      --os-variant rhel8.0 --location RHEL8.iso \
      --graphics none --extra-args='console=ttyS0'

• The following command creates a VM named demo-guest6, which has the same configuration as demo-guest5, but resides on the 192.0.2.1 remote host.

  # virt-install \
      --connect qemu+ssh://root@192.0.2.1/system --name demo-guest6 --memory 16384 \
      --vcpus 16 --disk size=280 --os-variant rhel8.0 --location RHEL8.iso \
      --graphics none --extra-args='console=ttyS0'

Verification

• If the VM is created successfully, a virt-viewer window opens with a graphical console of the VM and starts the guest OS installation.

Troubleshooting

• If virt-install fails with a cannot find default network error:

  • Ensure that the libvirt-daemon-config-network package is installed:

    # {PackageManagerCommand} info libvirt-daemon-config-network
    Installed Packages
    Name         : libvirt-daemon-config-network
    [...]

  • Verify that the libvirt default network is active and configured to start automatically:

    # virsh net-list --all
     Name      State    Autostart   Persistent
    --------------------------------------------
     default   active   yes         yes

  • If it is not, activate the default network and set it to auto-start:

    # virsh net-autostart default
    Network default marked as autostarted
    
    # virsh net-start default
    Network default started

    • If activating the default network fails with the following error, the libvirt-daemon-config-network package has not been installed correctly.

      error: failed to get network 'default'
      error: Network not found: no network with matching name 'default'

      To fix this, re-install libvirt-daemon-config-network:

      # {PackageManagerCommand} reinstall libvirt-daemon-config-network

    • If activating the default network fails with an error similar to the following, a conflict has occurred between the default network’s subnet and an existing interface on the host.

      error: Failed to start network default
      error: internal error: Network is already in use by interface ens2

      To fix this, use the virsh net-edit default command and change the 192.0.2.* values in the configuration to a subnet not already in use on the host.

Prerequisites

• The web console VM plug-in is installed on your system.
• Virtualization is enabled on your host system.
• You have a sufficient amount of system resources to allocate to your VMs, such as disk space, RAM, or CPUs. The recommended values may vary significantly depending on the intended tasks and workload of the VMs.

Prerequisites

• An inactive VM that is already defined.
• The name of the VM.

• For remote VMs:

  • The IP address of the host where the VM is located.
  • Root access privileges to the host.

Procedure

• For a local VM, use the virsh start utility.

  For example, the following command starts the demo-guest1 VM.

  # virsh start demo-guest1
  Domain 'demo-guest1' started

• For a VM located on a remote host, use the virsh start utility along with the QEMU+SSH connection to the host.

  For example, the following command starts the demo-guest1 VM on the 192.0.2.1 host.

  # virsh -c qemu+ssh://root@192.0.2.1/system start demo-guest1
  
  root@192.0.2.1's password:
  
  Domain 'demo-guest1' started

Prerequisites

• The web console VM plug-in is installed on your system.
• An inactive VM that is already defined.
• The name of the VM.

Prerequisites

• A created virtual machine

• To interact with the VM’s graphical interface in the web console, use the graphical console.
• To interact with the VM’s graphical interface in a remote viewer, use the graphical console in remote viewers.
• To interact with the VM’s CLI in the web console, use the serial console.

Prerequisites

• Your system, as well as the VM you are connecting to, must support graphical displays.
• If the target VM is located on a remote host, connection and root access privileges to the host are needed.
• Optional: If the target VM is located on a remote host, set up your libvirt and SSH for more convenient access to remote hosts.

Procedure

• To connect to a local VM, use the following command and replace guest-name with the name of the VM you want to connect to:

  # virt-viewer guest-name

• To connect to a remote VM, use the virt-viewer command with the SSH protocol. For example, the following command connects as root to a VM called guest-name, located on remote system 192.0.2.1. The connection also requires root authentication for 192.0.2.1.

  # virt-viewer --direct --connect qemu+ssh://root@192.0.2.1/system guest-name
  root@192.0.2.1's password:

Troubleshooting

• If clicking in the graphical console does not have any effect, expand the console to full screen. This is a known issue with the mouse cursor offset.

Prerequisites

• You have network connection and root access privileges to the target VM.
• If the target VM is located on a remote host, you also have connection and root access privileges to that host.

• Your VM network assigns IP addresses by dnsmasq generated by libvirt. This is the case for example in libvirt NAT networks.

  Notably, if your VM is using one of the following network configurations, you cannot connect to the VM by using SSH:

  • hostdev interfaces
  • Direct interfaces
  • Bridge interaces

• The libvirt-nss component is installed and enabled on the VM’s host. If it is not, do the following:

  1. Install the libvirt-nss package:

     # yum install libvirt-nss

  2. Edit the /etc/nsswitch.conf file and add libvirt_guest to the hosts line:

     ...
     passwd:      compat
     shadow:      compat
     group:       compat
     hosts:       files libvirt_guest dns
     ...

Troubleshooting

• If you do not know the VM’s name, you can list all VMs available on the host by using the virsh list --all command:

  # virsh list --all
  Id    Name                           State
  ----------------------------------------------------
  2     testguest1                    running
  -     testguest2                    shut off

• Does not provide VNC or SPICE protocols, and thus does not offer video display for GUI tools.
• Does not have a network connection, and thus cannot be interacted with using SSH.

Prerequisites

• The VM must have a serial console device configured, such as console type='pty'. To verify, do the following:

  # virsh dumpxml vm-name | grep console
  
  <console type='pty' tty='/dev/pts/2'>
  </console>

• The VM must have the serial console configured in its kernel command line. To verify this, the cat /proc/cmdline command output on the VM should include console=ttyS0. For example:

  # cat /proc/cmdline
  BOOT_IMAGE=/vmlinuz-3.10.0-948.el7.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,9600n8 rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb

  If the serial console is not set up properly on a VM, using virsh console to connect to the VM connects you to an unresponsive guest console. However, you can still exit the unresponsive console by using the Ctrl+] shortcut.

  • To set up serial console on the VM, do the following:

    1. On the VM, enable the console=ttyS0 kernel option:

       # grubby --update-kernel=ALL --args="console=ttyS0"

    2. Clear the kernel options that might prevent your changes from taking effect.

       # grub2-editenv - unset kernelopts

    3. Reboot the VM.

• You can edit the /etc/libvirt/libvirt.conf file and set the value of the uri_default parameter to example-qemu-host-alias as a default libvirt target.

  # These can be used in cases when no URI is supplied by the application
  # (@uri_default also prevents probing of the hypervisor driver).
  #
  uri_default = "example-qemu-host-alias"

  As a result, all libvirt-based commands will automatically be performed on the specified remote host.

  $ virsh list
  root@192.0.2.1's password:
  
  Id   Name              State
  ---------------------------------
  1   example-remote-guest      running

• When connecting to a remote host, you can avoid providing the root password to the remote system. To do so, use one or more of the following methods:

  • Set up key-based SSH access to the remote host
  • Use SSH connection multiplexing to connect to the remote system
  • Kerberos authentication in Identity Management
• The -c (or --connect) option can be used to run the virt-install, virt-viewer, virsh and virt-manager commands on a remote host.

• Use a shutdown command appropriate to the guest OS while connected to the guest.

• Use the virsh shutdown command on the host:

  • If the VM is on a local host:

    # virsh shutdown demo-guest1
    Domain 'demo-guest1' is being shutdown

  • If the VM is on a remote host, in this example 192.0.2.1:

    # virsh -c qemu+ssh://root@192.0.2.1/system shutdown demo-guest1
    
    root@192.0.2.1's password:
    Domain 'demo-guest1' is being shutdown

Prerequisites

• Back up important data from the VM.
• Shut down the VM.
• Make sure no other VMs use the same associated storage.

Procedure

• Use the virsh undefine utility.

  For example, the following command removes the guest1 VM, its associated storage volumes, and non-volatile RAM, if any.

  # virsh undefine guest1 --remove-all-storage --nvram
  Domain 'guest1' has been undefined
  Volume 'vda'(/home/images/guest1.qcow2) removed.

Prerequisites

• The web console VM plug-in is installed on your system.
• Back up important data from the VM.
• Make sure no other VM uses the same associated storage.
• Optional: Shut down the VM.

Prerequisites

• The web console VM plug-in is installed on your system.

Procedure

• Click Virtual Machines in the web console’s side menu.

  A dialog box appears with information about the available storage pools, available networks, and the VMs to which the web console is connected.

  

• Storage Pools - The number of storage pools, active or inactive, that can be accessed by the web console and their state.
• Networks - The number of networks, active or inactive, that can be accessed by the web console and their state.
• Name - The name of the VM.
• Connection - The type of libvirt connection, system or session.
• State - The state of the VM.

Prerequisites

• The web console VM plug-in is installed on your system.

Prerequisites

• The web console VM plug-in is installed on your system.

Prerequisites

• The web console VM plug-in is installed on your system.

Prerequisites

• The web console VM plug-in is installed on your system.

• Device - The device type of the disk.
• Used - The amount of disk currently allocated.
• Capacity - The maximum size of the storage volume.
• Bus - The type of disk device that is emulated.
• Access - Whether the disk is Writeable or Read-only. For raw disks, you can also set the access to Writeable and shared.
• Source - The disk device or file.

Prerequisites

• The web console VM plug-in is installed on your system.

Prerequisites

• The libguestfs-tools-c package, which contains the virt-sysprep utility, is installed on your host:

  # yum install libguestfs-tools-c

• The source VM intended as a template is shut down.

• You know where the disk image for the source VM is located, and you are the owner of the VM’s disk image file.

  Note that disk images for VMs created in the system connection of libvirt are located in the /var/lib/libvirt/images directory and owned by the root user by default:

  # ls -la /var/lib/libvirt/images
  -rw-------.  1 root root  9665380352 Jul 23 14:50 a-really-important-vm.qcow2
  -rw-------.  1 root root  8591507456 Jul 26  2017 an-actual-vm-that-i-use.qcow2
  -rw-------.  1 root root  8591507456 Jul 26  2017 totally-not-a-fake-vm.qcow2
  -rw-------.  1 root root 10739318784 Sep 20 17:57 another-vm-example.qcow2

• Optional: Any important data on the source VM’s disk has been backed up. If you want to preserve the source VM intact, clone it first and turn the clone into a template.

Verification

• To confirm that the process was successful, compare the modified disk image to the original one. The following example shows a successful creation of a template:

  # virt-diff -a /var/lib/libvirt/images/a-really-important-vm-orig.qcow2 -A /var/lib/libvirt/images/a-really-important-vm.qcow2
  - - 0644       1001 /etc/group-
  - - 0000        797 /etc/gshadow-
  = - 0444         33 /etc/machine-id
  [...]
  - - 0600        409 /home/username/.bash_history
  - d 0700          6 /home/username/.ssh
  - - 0600        868 /root/.bash_history
  [...]

Prerequisites

• Ensure that you know the location of the disk image for the source VM and are the owner of the VM’s disk image file.

  Note that disk images for VMs created in the system connection of libvirt are by default located in the /var/lib/libvirt/images directory and owned by the root user:

  # ls -la /var/lib/libvirt/images
  -rw-------.  1 root root  9665380352 Jul 23 14:50 a-really-important-vm.qcow2
  -rw-------.  1 root root  8591507456 Jul 26  2017 an-actual-vm-that-i-use.qcow2
  -rw-------.  1 root root  8591507456 Jul 26  2017 totally-not-a-fake-vm.qcow2
  -rw-------.  1 root root 10739318784 Sep 20 17:57 another-vm-example.qcow2

• Ensure that the VM is shut down.
• Optional: Any important data on the VM’s disk has been backed up. If you want to preserve the source VM intact, clone it first and edit the clone to create a template.

• Live migrating a VM under a heavy load

• Live migrating a VM that uses a large amount of memory, such as 1 TB or more

  Important

  Red Hat has successfully tested live migration of VMs with up to 6 TB of memory. However, for live migration scenarios that involve VMs with more than 1 TB of memory, customers should reach out to Red Hat technical support.

• Reduce the workload of the VM, especially memory updates.

  • To do this, stop or cancel non-essential processes in the guest operating system of the source VM.

• Increase the downtime allowed for the live migration:

  1. Display the current maximum downtime at the end of a live migration for the VM that is being migrated:

     # virsh migrate-getmaxdowntime vm-name

  2. Set a higher maximum downtime:

     # virsh migrate-setmaxdowntime vm-name downtime-in-miliseconds

     The higher you set the maximum downtime, the more likely it will be for the migration to complete.

• Switch the live migration to post-copy mode.

  # virsh migrate-start-postcopy vm-name

  • This ensures that the memory pages of the VM can converge on the destination host, and that the migration can complete.

    However, when post-copy mode is active, the VM might slow down significantly, due to remote page requests from the destination host to the source host. In addition, if the network connection between the source host and the destination host stops working during post-copy migration, some of the VM processes may halt due to missing memory pages.

    Therefore, do not use post-copy migration if the VM availability is critical or if the migration network is unstable.

• If your workload allows it, suspend the VM and let the migration finish as a non-live migration. This increases the downtime of the VM, but in most cases ensures that the migration completes successfully.

• The workload of the VM during the migration

  • Before starting the migration, stop or cancel non-essential processes in the guest operating system of the VM.

• The network bandwidth that the host can use for migration

  • For optimal results of a live migration, the bandwidth of the network used for the migration must be significantly higher than the dirty page generation of the VM. For instructions on obtaining the VM dirty page generation rate, see the Prerequisites in Migrating a virtual machine by using the command-line interface.
  • Both the source host and the destination host must have a dedicated network interface controller (NIC) for the migration. For live migrating a VM with more than 1 TB of memory, Red Hat recommends a NIC with the speed of 25 Gb/s or more.
  • You can also specify the network bandwidth assigned to the live migration by using the --bandwidth option when initiating the migration. For migrating very large VMs, assign as much bandwidth as viable for your deployment.

• The mode of live migration

  • The default pre-copy migration mode copies memory pages repeatedly if they become dirty.

  • Post-copy migration copies memory pages only once.

    To enable your live migration to switch to post-copy mode if the migration stalls, use the --postcopy option with virsh migrate when starting the migration.

• The downtime specified for the deployment

  • You can adjust this during the migration by using virsh migrate-setmaxdowntime as described previously.

Prerequisites

• Obtain the required options for the device you intend to attach to a VM. To see the available options for a specific device, use the virt-xml --device=? command. For example:

  # virt-xml --network=?
  --network options:
  [...]
  address.unit
  boot_order
  clearxml
  driver_name
  [...]

• Use the virsh dumpxml command and see if the device’s XML definition has been added to the <devices> section in the VM’s XML configuration.

  For example, the following output shows the configuration of the testguest VM and confirms that the 002.004 USB flash disk device has been added.

  # virsh dumpxml testguest
  [...]
  <hostdev mode='subsystem' type='usb' managed='yes'>
    <source>
      <vendor id='0x4146'/>
      <product id='0x902e'/>
      <address bus='2' device='4'/>
    </source>
    <alias name='hostdev0'/>
    <address type='usb' bus='0' port='3'/>
  </hostdev>
  [...]

• Run the VM and test if the device is present and works properly.

Prerequisites

• Obtain the required options for the device you intend to attach to a VM. To see the available options for a specific device, use the virt-xml --device=? command. For example:

• Optional: Back up the XML configuration of your VM by using virsh dumpxml vm-name and sending the output to a file. For example, the following backs up the configuration of your testguest1 VM as the testguest1.xml file:

• Run the VM and test if the device is present and reflects the modifications.

• Use the virsh dumpxml command and see if the device’s XML definition has been modified in the VM’s XML configuration.

  For example, the following output shows the configuration of the testguest VM and confirms that the CPU mode has been configured as host-model.

  # virsh dumpxml testguest
  [...]
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  [...]

Troubleshooting

• If modifying a device causes your VM to become unbootable, use the virsh define utility to restore the XML configuration by reloading the XML configuration file you backed up previously.

  # virsh define testguest.xml

Prerequisites

• Optional: Back up the XML configuration of your VM by using virsh dumpxml vm-name and sending the output to a file. For example, the following backs up the configuration of your testguest1 VM as the testguest1.xml file:

Troubleshooting

• If removing a device causes your VM to become unbootable, use the virsh define utility to restore the XML configuration by reloading the XML configuration file you backed up previously.

  # virsh define testguest.xml

Prerequisites

• The web console VM plug-in is installed on your system.

Prerequisites

• If you are attaching PCI devices, ensure that the status of the managed attribute of the hostdev element is set to yes.

  Note

  When attaching PCI devices to your VM, do not omit the managed attribute of the hostdev element, or set it to no. If you do so, PCI devices cannot automatically detach from the host when you pass them to the VM. They also cannot automatically reattach to the host when you turn off the VM.

  As a consequence, the host may become unresponsive or shut down unexpectedly.

  You can find the status of the managed attribute in your VM’s XML configuration. The following example opens the XML configuration of the example-VM-1 VM.

  # virsh edit example-VM-1

• Back up important data from the VM.

• Optional: Back up the XML configuration of your VM. For example, to back up the example-VM-1 VM:

  # virsh dumpxml example-VM-1 > example-VM-1.xml

• The web console VM plug-in is installed on your system.

Verification

• Run the VM and check if the device appears in the Host devices section.

Prerequisites

• The web console VM plug-in is installed on your system.

• Optional: Back up the XML configuration of your VM by using virsh dumpxml example-VM-1 and sending the output to a file. For example, the following backs up the configuration of your testguest1 VM as the testguest1.xml file:

  # virsh dumpxml testguest1 > testguest1.xml
  # cat testguest1.xml
  <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    <name>testguest1</name>
    <uuid>ede29304-fe0c-4ca4-abcd-d246481acd18</uuid>
    [...]
  </domain>

Troubleshooting

• If removing a host device causes your VM to become unbootable, use the virsh define utility to restore the XML configuration by reloading the XML configuration file you backed up previously.

  # virsh define testguest1.xml

Prerequisites

• Ensure the device you want to pass through to the VM is attached to the host.

Verification

• Run the VM and test if the device is present and works as expected.

• Use the virsh dumpxml command to see if the device’s XML definition has been added to the <devices> section in the VM’s XML configuration file.

  # virsh dumpxml example-VM-1
  [...]
  <hostdev mode='subsystem' type='usb' managed='yes'>
    <source>
      <vendor id='0x0407'/>
      <product id='0x6252'/>
      <address bus='1' device='5'/>
    </source>
    <alias name='hostdev0'/>
    <address type='usb' bus='0' port='3'/>
  </hostdev>
  [...]

Verification

• Run the VM and check if the device has been removed from the list of devices.

Prerequisites

• Ensure the smart card reader you want to pass through to the VM is attached to the host.
• Ensure the smart card reader type is supported in RHEL 8.

Procedure

• Create and attach a virtual smart card reader device to a VM. For example, to attach a smart card reader to the testguest VM:

  # virt-xml testguest --add-device --smartcard mode=passthrough,type=spicevmc
  Domain 'testguest' defined successfully.
  Changes will take effect after the domain is fully powered off.

  Note

  To attach a virtual smart card reader device to a running VM, add the --update argument to the previous command.

Prerequisites

• You must store and copy path of the ISO image on the host machine.

Procedure

• Use the virt-xml utility with the --add-device argument:

  For example, the following command attaches the example-ISO-name ISO image, stored in the /home/username/Downloads directory, to the example-VM-name VM.

  # virt-xml example-VM-name --add-device --disk /home/username/Downloads/example-ISO-name.iso,device=cdrom
  Domain 'example-VM-name' defined successfully.

Verification

• Run the VM and test if the device is present and works as expected.

Prerequisites

• You have installed the web console VM plug-in on your system.

Verification

• In the Virtual Machines interface, the file will appear under the Disks section.

Prerequisites

• You must store the ISO image on the host machine.
• You must know the path to the ISO image.

Verification

• Run the VM and test if the device is replaced and works as expected.

Verification

• Run the VM and check that image is no longer available.

Verification

• Confirm that the device is no longer listed in the XML configuration file of the VM.

Prerequisites

• You have installed the web console VM plug-in on your system.

Verification

• In the Virtual Machines interface, the attached file is no longer displayed under the Disks section.

• Is able to provide the same or similar service as the original PCIe device.
• Appears at a different address on the host PCI bus.
• Can be assigned to a different VM by using VFIO assignment.

• Physical functions (PFs) - A PCIe function that provides the functionality of its device (for example networking) to the host, but can also create and manage a set of VFs. Each SR-IOV capable device has one or more PFs.
• Virtual functions (VFs) - Lightweight PCIe functions that behave as independent devices. Each VF is derived from a PF. The maximum number of VFs a device can have depends on the device hardware. Each VF can be assigned only to a single VM at a time, but a VM can have multiple VFs assigned to it.

• Improved performance
• Reduced use of host CPU and memory resources

Disadvantages

• To modify the configuration of a PF, you must first change the number of VFs exposed by the PF to zero. Therefore, you also need to remove the devices provided by these VFs from the VM to which they are assigned.
• A VM with an VFIO-assigned devices attached, including SR-IOV VFs, cannot be migrated to another host. In some cases, you can work around this limitation by pairing the assigned device with an emulated device. For example, you can bond an assigned networking VF to an emulated vNIC, and remove the VF before the migration.
• In addition, VFIO-assigned devices require pinning of VM memory, which increases the memory consumption of the VM and prevents the use of memory ballooning on the VM.

Prerequisites

• The CPU and the firmware of your host support the I/O Memory Management Unit (IOMMU).

  • If using an Intel CPU, it must support the Intel Virtualization Technology for Directed I/O (VT-d).
  • If using an AMD CPU, it must support the AMD-Vi feature.

• The host system uses Access Control Service (ACS) to provide direct memory access (DMA) isolation for PCIe topology. Verify this with the system vendor.

  For additional information, see Hardware Considerations for Implementing SR-IOV.

• The physical network device supports SR-IOV. To verify if any network devices on your system support SR-IOV, use the lspci -v command and look for Single Root I/O Virtualization (SR-IOV) in the output.

  # lspci -v
  [...]
  02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
  	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
  	Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0
  	Memory at fcba0000 (32-bit, non-prefetchable) [size=128K]
  [...]
  	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
  	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
  	Kernel driver in use: igb
  	Kernel modules: igb
  [...]

• The host network interface you want to use for creating VFs is running. For example, to activate the eth1 interface and verify it is running:

  # ip link set eth1 up
  # ip link show eth1
  8: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
     link/ether a0:36:9f:8f:3f:b8 brd ff:ff:ff:ff:ff:ff
     vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
     vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
     vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

• For SR-IOV device assignment to work, the IOMMU feature must be enabled in the host BIOS and kernel. To do so:

  • On an Intel host, enable VT-d:

    1. Regenerate the GRUB configuration with the intel_iommu=on and iommu=pt parameters:

       # grubby --args="intel_iommu=on iommu=pt" --update-kernel=ALL

    2. Reboot the host.

  • On an AMD host, enable AMD-Vi:

    1. Regenerate the GRUB configuration with the iommu=pt parameter:

       # grubby --args="iommu=pt" --update-kernel=ALL

    2. Reboot the host.

Verification

• If the procedure is successful, the guest operating system detects a new network interface card.

Networking devices

• Intel 82599ES 10 Gigabit Ethernet Controller - uses the ixgbe driver
• Intel Ethernet Controller XL710 Series - uses the i40e driver
• Mellanox ConnectX-5 Ethernet Adapter Cards - use the mlx5_core driver
• Intel Ethernet Network Adapter XXV710 - uses the i40e driver
• Intel 82576 Gigabit Ethernet Controller - uses the igb driver
• Broadcom NetXtreme II BCM57810 - uses the bnx2x driver
• Ethernet Controller E810-C for QSFP - uses the ice driver
• SFC9220 10/40G Ethernet Controller - uses the sfc driver
• FastLinQ QL41000 Series 10/25/40/50GbE Controller - uses the qede driver
• MT2892 Family [ConnectX-6 Dx] - uses the mlx5_core driver

Storage devices

• Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa/172Xb (rev 01)
• Non-Volatile memory controller: Toshiba Corporation Cx5 NVMe SSD Controller (rev 01)

• Storage pools can be persistent or transient:

  • A persistent storage pool survives a system restart of the host machine. You can use the virsh pool-define to create a persistent storage pool.
  • A transient storage pool only exists until the host reboots. You can use the virsh pool-create command to create a transient storage pool.

• Local storage pools

  Local storage pools are attached directly to the host server. They include local directories, directly attached disks, physical partitions, and Logical Volume Management (LVM) volume groups on local devices.

  Local storage pools are useful for development, testing, and small deployments that do not require migration or have a large number of VMs.

• Networked (shared) storage pools

  Networked storage pools include storage devices shared over a network by using standard protocols.

• You can define an NFS storage pool on the virtualization host to describe the exported server path and the client target path. Consequently, libvirt can mount the storage either automatically when libvirt is started or as needed while libvirt is running.
• You can simply add the storage pool and storage volume to a VM by name. You do not need to add the target path to the volume. Therefore, even if the target client path changes, it does not affect the VM.
• You can configure storage pools to autostart. When you do so, libvirt automatically mounts the NFS shared disk on the directory which is specified when libvirt is started. libvirt mounts the share on the specified directory, similar to the command mount nfs.example.com:/path/to/share /vmdata.
• You can query the storage volume paths by using the libvirt API. These storage volumes are basically the files present in the NFS shared disk. You can then copy these paths into the section of a VM’s XML definition that describes the source storage for the VM’s block devices.

• In the case of NFS, you can use an application that uses the libvirt API to create and delete storage volumes in the storage pool (files in the NFS share) up to the limit of the size of the pool (the storage capacity of the share).

  Note that, not all storage pool types support creating and deleting volumes.

• You can stop a storage pool when no longer required. Stopping a storage pool (pool-destroy) undoes the start operation, in this case, unmounting the NFS share. The data on the share is not modified by the destroy operation, despite what the name of the command suggests. For more information, see man virsh.

• Directory-based storage pools
• Disk-based storage pools
• Partition-based storage pools
• GlusterFS storage pools
• iSCSI-based storage pools
• LVM-based storage pools
• NFS-based storage pools
• SCSI-based storage pools with vHBA devices
• Multipath-based storage pools
• RBD-based storage pools

• Sheepdog-based storage pools
• Vstorage-based storage pools
• ZFS-based storage pools

Procedure

• Use the virsh pool-list command to view storage pool information.

  # virsh pool-list --all --details
   Name                State    Autostart  Persistent    Capacity  Allocation   Available
   default             running  yes        yes          48.97 GiB   23.93 GiB   25.03 GiB
   Downloads           running  yes        yes         175.62 GiB   62.02 GiB  113.60 GiB
   RHEL-Storage-Pool   running  yes        yes         214.62 GiB   93.02 GiB  168.60 GiB

Prerequisites

• Ensure your hypervisor supports directory storage pools:

  # virsh pool-capabilities | grep "'dir' supported='yes'"

  If the command displays any output, directory pools are supported.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_dir
    Name:           guest_images_dir
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

Prerequisites

• Ensure your hypervisor supports disk-based storage pools:

  # virsh pool-capabilities | grep "'disk' supported='yes'"

  If the command displays any output, disk-based pools are supported.

• Prepare a device on which you will base the storage pool. For this purpose, prefer partitions (for example, /dev/sdb1) or LVM volumes. If you provide a VM with write access to an entire disk or block device (for example, /dev/sdb), the VM will likely partition it or create its own LVM groups on it. This can result in system errors on the host.

  However, if you require using an entire block device for the storage pool, Red Hat recommends protecting any important partitions on the device from GRUB’s os-prober function. To do so, edit the /etc/default/grub file and apply one of the following configurations:

  • Disable os-prober.

    GRUB_DISABLE_OS_PROBER=true

  • Prevent os-prober from discovering a specific partition. For example:

    GRUB_OS_PROBER_SKIP_LIST="5ef6313a-257c-4d43@/dev/sdb1"

• Back up any data on the selected storage device before creating a storage pool. Depending on the version of libvirt being used, dedicating a disk to a storage pool may reformat and erase all data currently stored on the disk device.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_disk
    Name:           guest_images_disk
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

Prerequisites

• Ensure your hypervisor supports filesystem-based storage pools:

  # virsh pool-capabilities | grep "'fs' supported='yes'"

  If the command displays any output, file-based pools are supported.

• Prepare a device on which you will base the storage pool. For this purpose, prefer partitions (for example, /dev/sdb1) or LVM volumes. If you provide a VM with write access to an entire disk or block device (for example, /dev/sdb), the VM will likely partition it or create its own LVM groups on it. This can result in system errors on the host.

  However, if you require using an entire block device for the storage pool, Red Hat recommends protecting any important partitions on the device from GRUB’s os-prober function. To do so, edit the /etc/default/grub file and apply one of the following configurations:

  • Disable os-prober.

    GRUB_DISABLE_OS_PROBER=true

  • Prevent os-prober from discovering a specific partition. For example:

    GRUB_OS_PROBER_SKIP_LIST="5ef6313a-257c-4d43@/dev/sdb1"

Prerequisites

• Before you can create GlusterFS-based storage pool on a host, prepare a Gluster.

  1. Obtain the IP address of the Gluster server by listing its status with the following command:

     # gluster volume status
     Status of volume: gluster-vol1
     Gluster process                           Port	Online	Pid
     ------------------------------------------------------------
     Brick 222.111.222.111:/gluster-vol1       49155	  Y    18634
     
     Task Status of Volume gluster-vol1
     ------------------------------------------------------------
     There are no active volume tasks

  2. If not installed, install the glusterfs-fuse package.

  3. If not enabled, enable the virt_use_fusefs boolean. Check that it is enabled.

     # setsebool virt_use_fusefs on
     # getsebool virt_use_fusefs
     virt_use_fusefs --> on

• Ensure your hypervisor supports GlusterFS-based storage pools:

  # virsh pool-capabilities | grep "'gluster' supported='yes'"

  If the command displays any output, GlusterFS-based pools are supported.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_glusterfs
    Name:           guest_images_glusterfs
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

Prerequisites

• Ensure your hypervisor supports iSCSI-based storage pools:

  # virsh pool-capabilities | grep "'iscsi' supported='yes'"

  If the command displays any output, iSCSI-based pools are supported.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_iscsi
    Name:           guest_images_iscsi
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

• LVM-based storage pools do not provide the full flexibility of LVM.
• libvirt supports thin logical volumes, but does not provide the features of thin storage pools.

• LVM-based storage pools are volume groups. You can create volume groups by using the virsh utility, but this way you can only have one device in the created volume group. To create a volume group with multiple devices, use the LVM utility instead, see How to create a volume group in Linux with LVM.

  For more detailed information about volume groups, refer to the Red Hat Enterprise Linux Logical Volume Manager Administration Guide.

• LVM-based storage pools require a full disk partition. If you activate a new partition or device by using virsh commands, the partition will be formatted and all data will be erased. If you are using a host’s existing volume group, as in these procedures, nothing will be erased.

Prerequisites

• Ensure your hypervisor supports LVM-based storage pools:

  # virsh pool-capabilities | grep "'logical' supported='yes'"

  If the command displays any output, LVM-based pools are supported.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_lvm
    Name:           guest_images_lvm
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

Prerequisites

• Ensure your hypervisor supports NFS-based storage pools:

  # virsh pool-capabilities | grep "<value>nfs</value>"

  If the command displays any output, NFS-based pools are supported.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_netfs
    Name:           guest_images_netfs
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

Prerequisites

• Ensure your hypervisor supports SCSI-based storage pools:

  # virsh pool-capabilities | grep "'scsi' supported='yes'"

  If the command displays any output, SCSI-based pools are supported.

• Before creating a SCSI-based storage pools with vHBA devices, create a vHBA. For more information, see Creating vHBAs.

Verification

• Use the virsh pool-info command to verify that the storage pool is in the running state. Check if the sizes reported are as expected and if autostart is configured correctly.

  # virsh pool-info guest_images_vhba
    Name:           guest_images_vhba
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB

Verification

• Confirm that the storage pool was deleted.

  # virsh pool-list --all
  Name                 State      Autostart
  -------------------------------------------
  default              active     yes
  rhel-Storage-Pool   active     yes

Prerequisites

• The web console VM plug-in is installed on your system.