16.7. Assigning GPU Devices
- GPU PCI Device Assignment - Using this method, it is possible to remove a GPU device from the host and assign it to a single guest.
- NVIDIA vGPU Assignment - This method makes it possible to create multiple mediated devices from a physical GPU, and assign these devices as virtual GPUs to multiple guests. This is only supported on selected NVIDIA GPUs, and only one mediated device can be assigned to a single guest.
16.7.1. GPU PCI Device Assignment
- NVIDIA Quadro K-Series, M-Series, and P-Series (models 2000 series or higher)
- NVIDIA GRID K-Series
- NVIDIA Tesla K-Series and M-Series
lspcicommand, detach the device from host, attach it to the guest, and configure Xorg on the guest - as described in the following procedures:
Procedure 16.13. Enable IOMMU support in the host machine kernel
Edit the kernel command lineFor an Intel VT-d system, IOMMU is activated by adding the
iommu=ptparameters to the kernel command line. For an AMD-Vi system, the option needed is
amd_iommu=pt. To enable this option, edit or add the
GRUB_CMDLINX_LINUXline to the
/etc/sysconfig/grubconfiguration file as follows:
GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_VolGroup00/LogVol01 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_VolGroup_1/root vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet intel_iommu=on iommu=pt"
NoteFor further information on IOMMU, see Appendix E, Working with IOMMU Groups.
Regenerate the boot loader configurationFor the changes to the kernel command line to apply, regenerate the boot loader configuration using the
grub2-mkconfig -o /etc/grub2.cfgNote that if you are using a UEFI-based host, the target file should be
Reboot the hostFor the changes to take effect, reboot the host machine:
Procedure 16.14. Excluding the GPU device from binding to the host physical machine driver
Identify the PCI bus addressTo identify the PCI bus address and IDs of the device, run the following
lspcicommand. In this example, a VGA controller such as an NVIDIA Quadro or GRID card is used:
lspci -Dnn | grep VGA0000:02:00.0 VGA compatible controller : NVIDIA Corporation GK106GL [Quadro K4000] [10de:11fa] (rev a1)The resulting search reveals that the PCI bus address of this device is 0000:02:00.0 and the PCI IDs for the device are 10de:11fa.
Prevent the native host machine driver from using the GPU deviceTo prevent the native host machine driver from using the GPU device, you can use a PCI ID with the pci-stub driver. To do this, append the
pci-stub.idsoption, with the PCI IDs as its value, to the
GRUB_CMDLINX_LINUXline located in the
/etc/sysconfig/grubconfiguration file, for example as follows:
GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_VolGroup00/LogVol01 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_VolGroup_1/root vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet intel_iommu=on iommu=pt pci-stub.ids=10de:11fa"To add additional PCI IDs for pci-stub, separate them with a comma.
Regenerate the boot loader configurationRegenerate the boot loader configuration using the
grub2-mkconfigto include this option:
grub2-mkconfig -o /etc/grub2.cfgNote that if you are using a UEFI-based host, the target file should be
Reboot the host machineIn order for the changes to take effect, reboot the host machine:
Procedure 16.15. Optional: Editing the GPU IOMMU configuration
Display the XML information of the GPUTo display the settings of the GPU in XML form, you first need to convert its PCI bus address to libvirt-compatible format by appending
pci_and converting delimiters to underscores. In this example, the GPU PCI device identified with the 0000:02:00.0 bus address (as obtained in the previous procedure) becomes
pci_0000_02_00_0. Use the libvirt address of the device with the
virsh nodedev-dumpxmlto display its XML configuration:
virsh nodedev-dumpxml pci_0000_02_00_0
<device> <name>pci_0000_02_00_0</name> <path>/sys/devices/pci0000:00/0000:00:03.0/0000:02:00.0</path> <parent>pci_0000_00_03_0</parent> <driver> <name>pci-stub</name> </driver> <capability type='pci'> <domain>0</domain> <bus>2</bus> <slot>0</slot> <function>0</function> <product id='0x11fa'>GK106GL [Quadro K4000]</product> <vendor id='0x10de'>NVIDIA Corporation</vendor> <!-- pay attention to the following lines --> <iommuGroup number='13'> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='8' width='16'/> <link validity='sta' speed='2.5' width='16'/> </pci-express> </capability> </device>Note the
<iommuGroup>element of the XML. The iommuGroup indicates a set of devices that are considered isolated from other devices due to IOMMU capabilities and PCI bus topologies. All of the endpoint devices within the iommuGroup (meaning devices that are not PCIe root ports, bridges, or switch ports) need to be unbound from the native host drivers in order to be assigned to a guest. In the example above, the group is composed of the GPU device (0000:02:00.0) as well as the companion audio device (0000:02:00.1). For more information, see Appendix E, Working with IOMMU Groups.
Adjust IOMMU settingsIn this example, assignment of NVIDIA audio functions is not supported due to hardware issues with legacy interrupt support. In addition, the GPU audio function is generally not useful without the GPU itself. Therefore, in order to assign the GPU to a guest, the audio function must first be detached from native host drivers. This can be done using one of the following:
- Detect the PCI ID for the device and append it to the
pci-stub.idsoption in the
/etc/sysconfig/grubfile, as detailed in Procedure 16.14, “Excluding the GPU device from binding to the host physical machine driver”
- Use the
virsh nodedev-detachcommand, for example as follows:
virsh nodedev-detach pci_0000_02_00_1Device pci_0000_02_00_1 detached
Procedure 16.16. Attaching the GPU
- Using the Virtual Machine Manager interface. For details, see Section 16.1.2, “Assigning a PCI Device with virt-manager”.
- Creating an XML configuration fragment for the GPU and attaching it with the
- Create an XML for the device, similar to the following:
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> </hostdev>
- Save this to a file and run
virsh attach-device [domain] [file] --persistentto include the XML in the guest configuration. Note that the assigned GPU is added in addition to the existing emulated graphics device in the guest machine. The assigned GPU is handled as a secondary graphics device in the virtual machine. Assignment as a primary graphics device is not supported and emulated graphics devices in the guest's XML should not be removed.
- Editing the guest XML configuration using the
virsh editcommand and adding the appropriate XML segment manually.
Procedure 16.17. Ḿodifying the Xorg configuration on the guest
- In the guest, use the
lspcicommand to determine the PCI bus adress of the GPU:
lspci | grep VGA00:02.0 VGA compatible controller: Device 1234:111 00:09.0 VGA compatible controller: NVIDIA Corporation GK106GL [Quadro K4000] (rev a1)In this example, the bus address is 00:09.0.
- In the
/etc/X11/xorg.conffile on the guest, add a
BusIDoption with the detected address adjusted as follows:
Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BusID "PCI:0:9:0" EndSection
ImportantIf the bus address detected in Step 1 is hexadecimal, you need to convert the values between delimiters to the decimal system. For example, 00:0a.0 should be converted into PCI:0:10:0.
modprobe.blacklist=nouveauon the kernel command line during install. For information on other guest virtual machines, see the operating system's specific documentation.
16.7.2. NVIDIA vGPU Assignment
NVIDIA vGPU Setup
- Obtain the NVIDIA vGPU drivers and install them on your system. For instructions, see the NVIDIA documentation.
- If the NVIDIA software installer did not create the
/etc/modprobe.d/nvidia-installer-disable-nouveau.conffile, create a
.conffile (of any name) in the
/etc/modprobe.d/directory. Add the following lines in the file:
blacklist nouveau options nouveau modeset=0
- Regenerate the initial ramdisk for the current kernel, then reboot:
rebootIf you need to use a prior supported kernel version with mediated devices, regenerate the initial ramdisk for all installed kernel versions:
dracut --regenerate-all --force#
- Check that the
nvidia_vgpu_vfiomodule has been loaded by the kernel and that the
nvidia-vgpu-mgr.serviceservice is running.
lsmod | grep nvidia_vgpu_vfionvidia_vgpu_vfio 45011 0 nvidia 14333621 10 nvidia_vgpu_vfio mdev 20414 2 vfio_mdev,nvidia_vgpu_vfio vfio 32695 3 vfio_mdev,nvidia_vgpu_vfio,vfio_iommu_type1 #
systemctl status nvidia-vgpu-mgr.servicenvidia-vgpu-mgr.service - NVIDIA vGPU Manager Daemon Loaded: loaded (/usr/lib/systemd/system/nvidia-vgpu-mgr.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2018-03-16 10:17:36 CET; 5h 8min ago Main PID: 1553 (nvidia-vgpu-mgr) [...]
- Write a device UUID to
/sys/class/mdev_bus/pci_dev/mdev_supported_types/type-id/create, where pci_dev is the PCI address of the host GPU, and type-id is an ID of the host GPU type.The following example shows how to create a mediated device of
nvidia-63vGPU type on an NVIDIA Tesla P4 card:
echo "30820a6f-b1a5-4503-91ca-0c10ba58692a" > /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-63/createFor type-id values for specific devices, see section 1.3.1. Virtual GPU Types in Virtual GPU software documentation. Note that only Q-series NVIDIA vGPUs, such as
GRID P4-2Q, are supported as mediated device GPU types on Linux guests.
- Add the following lines to the <devices/> sections in XML configurations of guests that you want to share the vGPU resources. Use the UUID value generated by the
uuidgencommand in the previous step. Each UUID can only be assigned to one guest at a time.
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci'> <source> <address uuid='30820a6f-b1a5-4503-91ca-0c10ba58692a'/> </source> </hostdev>
ImportantFor the vGPU mediated devices to work properly on the assigned guests, NVIDIA vGPU guest software licensing needs to be set up for the guests. For further information and instructions, see the NVIDIA virtual GPU software documentation.
Setting up VNC console for video streaming with NVIDIA vGPU
- Configure NVIDIA vGPU on your as described in the section above. Ensure the mediated device's XML configuration includes the display='on' parameter. For example:
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'> <source> <address uuid='ba26a3e2-8e1e-4f39-9de7-b26bd210268a'/> </source> </hostdev>
- Optionally, set the VM's video model type as none. For example:
<video> <model type='none'/> </video>If this is not specified, you receive two different display outputs - one from an emulated Cirrus or QXL card and one from NVIDIA vGPU. Also note that using model type='none' currently makes it impossible to see the boot graphical output. As a result, the first graphical output displayed is the login screen.
- Ensure the XML configuration of the VM's graphics type is spice or vnc. For example:
<graphics type='spice' autoport='yes'> <listen type='address'/> <image compression='off'/> </graphics>
- Start the virtual machine.
- Connect to the virtual machine using the VNC viewer remote desktop client.
Removing NVIDIA vGPU Devices
echo 1 > /sys/bus/mdev/devices/uuid/remove
echo: write error: Device or resource busy
Querying NVIDIA vGPU Capabilities
virsh nodedev-list --cap mdev_typesand
virsh nodedev-dumpxmlcommands. For example, the following displays available vGPU types on a Tesla P4 card:
$ virsh nodedev-list --cap mdev_types pci_0000_01_00_0 $ virsh nodedev-dumpxml pci_0000_01_00_0 <...> <capability type='mdev_types'> <type id='nvidia-70'> <name>GRID P4-8A</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>1</availableInstances> </type> <type id='nvidia-69'> <name>GRID P4-4A</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>2</availableInstances> </type> <type id='nvidia-67'> <name>GRID P4-1A</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>8</availableInstances> </type> <type id='nvidia-65'> <name>GRID P4-4Q</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>2</availableInstances> </type> <type id='nvidia-63'> <name>GRID P4-1Q</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>8</availableInstances> </type> <type id='nvidia-71'> <name>GRID P4-1B</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>8</availableInstances> </type> <type id='nvidia-68'> <name>GRID P4-2A</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>4</availableInstances> </type> <type id='nvidia-66'> <name>GRID P4-8Q</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>1</availableInstances> </type> <type id='nvidia-64'> <name>GRID P4-2Q</name> <deviceAPI>vfio-pci</deviceAPI> <availableInstances>4</availableInstances> </type> </capability> </...>
Remote Desktop Streaming Services for NVIDIA vGPU
- Mechdyne TGX - It is currently not possible to use Mechdyne TGX with Windows Server 2016 guests.
- NICE DCV - When using this streaming service, Red Hat recommends using fixed resolution settings, as using dynamic resolution in some cases results in a black screen.