Error when configuring PCI Pass-through for NVIDIA Telsa T4 GPU in OpenStack

Resolution

• First step is to determine GPU device type capabilities. Running the following command will obtain pci device details as shown in the following two examples:

  lspci -nnk
  

• Below example is not a SR-IOV capable GPU device type. Use device_type=type-PCI configuration option to allow PCI passthrough.

  d8:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
      Subsystem: NVIDIA Corporation Device 12a2
      Physical Slot: 3
      Flags: fast devsel, IRQ 11, NUMA node 1
          ...
      Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
      Capabilities: [900] #19
      Capabilities: [bb0] #15
      Kernel modules: nouveau
  

• Below example is a SR-IOV capable GPU device type. Use device_type=type-PF configuration option to allow PCI passthrough.
  Note the line Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV).

  d8:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
      Subsystem: NVIDIA Corporation Device 12a2
      Physical Slot: 3
      Flags: fast devsel, IRQ 11, NUMA node 1
          ...
      Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
      Capabilities: [900] #19
      Capabilities: [bb0] #15
      Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
      Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
      Kernel modules: nouveau
  

Solution for non SR-IOV capable GPU device type

1. In the yaml file (i.e., ~templates/computesriov_gpu-environment.yaml), set device_type=type-PCI, i.e.

   ComputeSriovGPUExtraConfig:
   nova::pci::aliases:
     - name: "t4"
       product_id: "1eb8"
       vendor_id: "10de"
       device_type: "type-PCI"
   

2. Re-run the deployment with the new change.

3. Review updated files:

   • On compute nodes, in configuration file: /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf set:

     alias={"device_type":"type-PCI","name":"t4","product_id":"1eb8","vendor_id":"10de"}
     

   • On controller nodes, in configuration file: /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf set:

     alias={"device_type":"type-PCI","name":"t4","product_id":"1eb8","vendor_id":"10de"}
     

Solution for SR-IOV capable GPU device type

1. In the yaml file (i.e., ~templates/computesriov_gpu-environment.yaml), set device_type=type-PF, i.e.

   ComputeSriovGPUExtraConfig:
   nova::pci::aliases:
     - name: "t4"
       product_id: "1eb8"
       vendor_id: "10de"
       device_type: "type-PF"
   

2. Re-run the deployment with the new change.

3. Review updated files:

   • On compute nodes, in configuration file: /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf set:

     alias={"device_type":"type-PF","name":"t4","product_id":"1eb8","vendor_id":"10de"}
     

   • On controller nodes, in configuration file: /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf set:

     alias={"device_type":"type-PF","name":"t4","product_id":"1eb8","vendor_id":"10de"}
     

• See Red Hat OSP 16.1 guide Configuring the Compute Service for Instance Creation Chapter 3. Configuring PCI passthrough for further information on how to use PCI passthrough to attach a physical PCI device, such as a graphics card or a network device, to a virtual machine instance.