Chapter 14. Configuring virtual GPUs for instances

1. Designate Compute nodes for vGPU.
2. Configure the Compute node for vGPU.
3. Deploy the overcloud.
4. Create a vGPU flavor for launching instances that have vGPU.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the overcloudrc credential file:

   $ source ~/overcloudrc

3. Create an instance with the hardware and software profile that your vGPU instances require:

   (overcloud)$ openstack server create --flavor <flavor> \
    --image <image> temp_vgpu_instance

   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require. For information about creating a vGPU flavor, see Creating a vGPU flavor for instances.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information about downloading RHEL cloud images, see Image service.
4. Log in to the instance as a cloud-user.
5. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

6. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances. For more information about supported drivers, see Image metadata.

7. Create an image snapshot of the instance:

   (overcloud)$ openstack server image create \
    --name vgpu_image temp_vgpu_instance

8. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor, for example:

   (overcloud)$ openstack flavor create --vcpus 6 \
    --ram 8192 --disk 100 m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

2. Assign a vGPU resource to the flavor that you created. You can assign only one vGPU for each instance.

   (overcloud)$ openstack flavor set m1.small-gpu \
    --property "resources:VGPU=1"
   
   (overcloud)$ openstack flavor show m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | access_project_ids         | None                                 |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 | resources:VGPU='1'                   |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

Procedure

1. Create an instance using a GPU flavor and image, for example:

   (overcloud)$ openstack server create --flavor m1.small-gpu \
    --image vgpu_image --security-group web --nic net-id=internal0 \
    --key-name lambda vgpu-instance

2. Log in to the instance as a cloud-user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

Procedure

1. To determine the vendor ID and product ID for each passthrough device type, enter the following command on the physical server that has the PCI cards:

   # lspci -nn | grep -i <gpu_name>

   For example, to determine the vendor and product ID for an NVIDIA GPU, enter the following command:

   # lspci -nn | grep -i nvidia
   3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
   d8:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1db4] (rev a1)

2. To determine if each PCI device has Single Root I/O Virtualization (SR-IOV) capabilities, enter the following command on the physical server that has the PCI cards:

   # lspci -v -s 3b:00.0
   3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
             ...
             Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
             ...

3. To configure the Controller node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_controller.yaml.

4. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthru_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

5. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthru_controller.yaml:

   • If the PCI device has SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     For more information on configuring the device_type field, see PCI passthrough device type field.

     Note

     If the nova-api service is running in a role other than the Controller, then replace ControllerExtraConfig with the user role, in the format <Role>ExtraConfig.

6. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_compute.yaml.

7. To specify the available PCIs for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   parameter_defaults:
     NovaPCIPassthrough:
       - vendor_id: "10de"
         product_id: "1eb8"

8. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   • If the PCI device has SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     Note

     The Compute node aliases must be identical to the aliases on the Controller node.

9. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthru_compute.yaml:

   parameter_defaults:
     ...
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   Note

   When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each overcloud deployment. For more information, see Configuring manual node reboot to define KernelArgs.

10. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
      -e [your environment files] \
      -e /home/stack/templates/pci_passthru_controller.yaml \
      -e /home/stack/templates/pci_passthru_compute.yaml

11. Configure a flavor to request the PCI devices. The following example requests two devices, each with a vendor ID of 10de and a product ID of 13f2:

    # openstack flavor set m1.large \
     --property "pci_passthrough:alias"="t4:2"

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor m1.large \
    --image <custom_gpu> --wait test-pci

   Replace <custom_gpu> with the name of your custom instance image that has the required GPU driver installed.

2. Log in to the instance as a cloud user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

4. To check the NVIDIA System Management Interface status, enter the following command from the instance:

   $ nvidia-smi

   Example output:

   -----------------------------------------------------------------------------
   | NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
   |---------------------------------------------------------------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |===========================================================================|
   |   0  Tesla T4            Off  | 00000000:01:00.0 Off |                    0 |
   | N/A   43C    P0    20W /  70W |      0MiB / 15109MiB |      0%      Default |
   ---------------------------------------------------------------------------
   
   -----------------------------------------------------------------------------
   | Processes:                                                       GPU Memory |
   |  GPU       PID   Type   Process name                             Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   -----------------------------------------------------------------------------