Procedure

1. Copy the /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml file to your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml /home/stack/templates/enable-swap.yaml

2. Configure the swap size by adding the following parameters to your enable-swap.yaml file:

   parameter_defaults:
     swap_size_megabytes: <swap size in MB>
     swap_path: <full path to location of swap, default: /swap>

3. Add the enable_swap.yaml environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/enable-swap.yaml

1. Designate Compute nodes for CPU pinning.
2. Configure the Compute nodes to reserve host cores for pinned instance vCPU processes, floating instance vCPU processes, and host processes.
3. Deploy the overcloud.
4. Create a flavor for launching instances that require CPU pinning.
5. Create a flavor for launching instances that use shared, or floating, CPUs.

Procedure

1. Log in to the undercloud as the stack user.
2. Open your Compute environment file.

3. To reserve physical CPU cores for instances that require CPU pinning, configure the NovaComputeCpuDedicatedSet parameter in the Compute environment file. For example, the following configuration sets the dedicated CPUs on a Compute node with a 32-core CPU:

   parameter_defaults:
     ...
     NovaComputeCpuDedicatedSet: 2-15,18-31
     ...

   For more information, see Configuring CPU pinning on the Compute nodes.

4. To reserve physical CPU cores for the emulator threads, configure the NovaComputeCpuSharedSet parameter in the Compute environment file. For example, the following configuration sets the shared CPUs on a Compute node with a 32-core CPU:

   parameter_defaults:
     ...
     NovaComputeCpuSharedSet: 0,1,16,17
     ...

   Note

   The Compute scheduler also uses the CPUs in the shared set for instances that run on shared, or floating, CPUs. For more information, see Configuring CPU pinning on Compute nodes

5. Add the Compute scheduler filter NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter, if not already present.

6. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

7. Configure a flavor that runs emulator threads for the instance on a dedicated CPU, which is selected from the shared CPUs configured using NovaComputeCpuSharedSet:

   (overcloud)$ openstack flavor set --property hw:cpu_policy=dedicated \
    --property hw:emulator_threads_policy=share \
    dedicated_emulator_threads

   For more information about configuration options for hw:emulator_threads_policy, see Emulator threads policy in Flavor metadata.

Procedure

1. Open your Compute environment file.

2. Configure the amount of huge page memory to reserve on each NUMA node for processes that are not instances:

   parameter_defaults:
     ComputeParameters:
       NovaReservedHugePages: ["node:0,size:1GB,count:1","node:1,size:1GB,count:1"]

   • Replace the size value for each node with the size of the allocated huge page. Set to one of the following valid values:

     • 2048 (for 2MB)
     • 1GB
   • Replace the count value for each node with the number of huge pages used by OVS per NUMA node. For example, for 4096 of socket memory used by Open vSwitch, set this to 2.

3. Configure huge pages on the Compute nodes:

   parameter_defaults:
     ComputeParameters:
       ...
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32"

   Note

   If you configure multiple huge page sizes, you must also mount the huge page folders during first boot. For more information, see Mounting multiple huge page folders during first boot.

4. Optional: To allow instances to allocate 1GB huge pages, configure the CPU feature flags, NovaLibvirtCPUModelExtraFlags, to include pdpe1gb:

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: 'custom'
       NovaLibvirtCPUModels: 'Haswell-noTSX'
       NovaLibvirtCPUModelExtraFlags: 'vmx, pdpe1gb'

   Note

   • CPU feature flags do not need to be configured to allow instances to only request 2 MB huge pages.
   • You can only allocate 1G huge pages to an instance if the host supports 1G huge page allocation.
   • You only need to set NovaLibvirtCPUModelExtraFlags to pdpe1gb when NovaLibvirtCPUMode is set to host-model or custom.
   • If the host supports pdpe1gb, and host-passthrough is used as the NovaLibvirtCPUMode, then you do not need to set pdpe1gb as a NovaLibvirtCPUModelExtraFlags. The pdpe1gb flag is only included in Opteron_G4 and Opteron_G5 CPU models, it is not included in any of the Intel CPU models supported by QEMU.
   • To mitigate for CPU hardware issues, such as Microarchitectural Data Sampling (MDS), you might need to configure other CPU flags. For more information, see RHOS Mitigation for MDS ("Microarchitectural Data Sampling") Security Flaws.

5. To avoid loss of performance after applying Meltdown protection, configure the CPU feature flags, NovaLibvirtCPUModelExtraFlags, to include +pcid:

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: 'custom'
       NovaLibvirtCPUModels: 'Haswell-noTSX'
       NovaLibvirtCPUModelExtraFlags: 'vmx, pdpe1gb, +pcid'

   Tip

   For more information, see Reducing the performance impact of Meltdown CVE fixes for OpenStack guests with "PCID" CPU feature flag.

6. Add NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter, if not already present.

7. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files]  \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Open your Compute environment file.

2. Configure the amount of host disk space, in MiB, to make available for instance RAM, by adding the following parameter to your Compute environment file:

   parameter_defaults:
     NovaLibvirtFileBackedMemory: 102400

3. Optional: To configure the directory to store the memory backing files, set the QemuMemoryBackingDir parameter in your Compute environment file. If not set, the memory backing directory defaults to /var/lib/libvirt/qemu/ram/.

   Note

   You must locate your backing store in a directory at or above the default directory location, /var/lib/libvirt/qemu/ram/.

   You can also change the host disk for the backing store. For more information, see Changing the memory backing directory host disk.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Create an environment file to configure shared instance storage, for example, nfs_instance_disk_backend.yaml.

4. To configure an NFS backend for instance files, add the following configuration to nfs_instance_disk_backend.yaml:

   parameter_defaults:
     ...
     NovaNfsEnabled: True
     NovaNfsShare: <nfs_share>

   Replace <nfs_share> with the NFS share directory to mount for instance file storage, for example, '192.168.122.1:/export/nova' or '192.168.24.1:/var/nfs'. If using IPv6, use both double and single-quotes, e.g. "'[fdd0::1]:/export/nova'".

5. Optional: The default mount SELinux context for NFS storage when NFS backend storage is enabled is 'context=system_u:object_r:nova_var_lib_t:s0'. Add the following parameter to amend the mount options for the NFS instance file storage mount point:

   parameter_defaults:
     ...
     NovaNfsOptions: 'context=system_u:object_r:nova_var_lib_t:s0,<additional_nfs_mount_options>'

   Replace <additional_nfs_mount_options> with a comma-separated list of the mount options you want to use for NFS instance file storage. For more information on the available mount options, see the mount man page:

   $ man 8 mount.

6. Save the updates to your environment file.

7. Add your new environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/nfs_instance_disk_backend.yaml

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data_pci_passthrough.yaml that includes the Controller, Compute, and ComputePCI roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_pci_passthrough.yaml \
    Compute:ComputePCI Compute Controller

4. Open roles_data_pci_passthrough.yaml and edit or add the following parameters and sections:

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputePCI
   Role name	name: Compute	name: ComputePCI
   description	Basic Compute Node role	PCI Passthrough Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepci-%index%
   deprecated_nic_config_name	compute.yaml	compute-pci-passthrough.yaml

5. Register the PCI passthrough Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in the Director Installation and Usage guide.

7. Create the compute-pci-passthrough overcloud flavor for PCI passthrough Compute nodes:

   (undercloud)$ openstack flavor create --id auto \
    --ram <ram_size_mb> --disk <disk_size_gb> \
    --vcpus <no_vcpus> compute-pci-passthrough

   • Replace <ram_size_mb> with the RAM of the bare metal node, in MB.
   • Replace <disk_size_gb> with the size of the disk on the bare metal node, in GB.

   • Replace <no_vcpus> with the number of CPUs on the bare metal node.

     Note

     These properties are not used for scheduling instances. However, the Compute scheduler does use the disk size to determine the root partition size.

8. Retrieve a list of your nodes to identify their UUIDs:

   (undercloud)$ openstack baremetal node list

9. Tag each bare metal node that you want to designate for PCI passthrough with a custom PCI passthrough resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.PCI-PASSTHROUGH <node>

   Replace <node> with the ID of the bare metal node.

10. Associate the compute-pci-passthrough flavor with the custom PCI passthrough resource class:

    (undercloud)$ openstack flavor set \
     --property resources:CUSTOM_BAREMETAL_PCI_PASSTHROUGH=1 \
      compute-pci-passthrough

    To determine the name of a custom resource class that corresponds to a resource class of a Bare Metal service node, convert the resource class to uppercase, replace all punctuation with an underscore, and prefix with CUSTOM_.

    Note

    A flavor can request only one instance of a bare metal resource class.

11. Set the following flavor properties to prevent the Compute scheduler from using the bare metal flavor properties to schedule instances:

    (undercloud)$ openstack flavor set \
     --property resources:VCPU=0 --property resources:MEMORY_MB=0 \
     --property resources:DISK_GB=0 compute-pci-passthrough

12. Add the following parameters to the node-info.yaml file to specify the number of PCI passthrough Compute nodes, and the flavor to use for the PCI passthrough designated Compute nodes:

    parameter_defaults:
      OvercloudComputePCIFlavor: compute-pci-passthrough
      ComputePCICount: 3

13. To verify that the role was created, enter the following command:

    (undercloud)$ openstack overcloud profiles list

Procedure

1. Create an environment file to configure the Controller node on the overcloud for PCI passthrough, for example, pci_passthrough_controller.yaml.

2. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthrough_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

3. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthrough_controller.yaml:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"

   For more information about configuring the device_type field, see PCI passthrough device type field.

   Note

   If the nova-api service is running in a role different from the Controller role, replace ControllerExtraConfig with the user role in the format <Role>ExtraConfig.

4. Optional: To set a default NUMA affinity policy for PCI passthrough devices, add numa_policy to the nova::pci::aliases: configuration from step 3:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"
           numa_policy: "preferred"

5. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthrough_compute.yaml.

6. To specify the available PCIs for the devices on the Compute node, use the vendor_id and product_id options to add all matching PCI devices to the pool of PCI devices available for passthrough to instances. For example, to add all Intel® Ethernet Controller X710 devices to the pool of PCI devices available for passthrough to instances, add the following configuration to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIParameters:
       NovaPCIPassthrough:
         - vendor_id: "8086"
           product_id: "1572"

   For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

7. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the PCI passthrough Compute node, add the following to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"

   Note

   The Compute node aliases must be identical to the aliases on the Controller node. Therefore, if you added numa_affinity to nova::pci::aliases in pci_passthrough_controller.yaml, then you must also add it to nova::pci::aliases in pci_passthrough_compute.yaml.

8. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthrough_compute.yaml. For example, use the following KernalArgs settings to enable an Intel IOMMU:

   parameter_defaults:
     ...
     ComputePCIParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   To enable an AMD IOMMU, set KernelArgs to "amd_iommu=on iommu=pt".

9. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/pci_passthrough_controller.yaml \
     -e /home/stack/templates/pci_passthrough_compute.yaml \

10. Create and configure the flavors that your cloud users can use to request the PCI devices. The following example requests two devices, each with a vendor ID of 8086 and a product ID of 1572, using the alias defined in step 7:

    (overcloud)# openstack flavor set \
     --property "pci_passthrough:alias"="a1:2" device_passthrough

11. Optional: To override the default NUMA affinity policy for PCI passthrough devices, you can add the NUMA affinity policy property key to the flavor or the image:

    • To override the default NUMA affinity policy by using the flavor, add the hw:pci_numa_affinity_policy property key:

      (overcloud)# openstack flavor set \
       --property "hw:pci_numa_affinity_policy"="required" \
       device_passthrough

      For more information about the valid values for hw:pci_numa_affinity_policy, see Flavor metadata.

    • To override the default NUMA affinity policy by using the image, add the hw_pci_numa_affinity_policy property key:

      (overcloud)# openstack image set \
       --property hw_pci_numa_affinity_policy=required \
       device_passthrough_image

      Note

      If you set the NUMA affinity policy on both the image and the flavor then the property values must match. The flavor setting takes precedence over the image and default settings. Therefore, the configuration of the NUMA affinity policy on the image only takes effect if the property is not set on the flavor.

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor device_passthrough \
    --image <image> --wait test-pci

2. Log in to the instance as a cloud user. For more information, see Connecting to an instance.

3. To verify that the PCI device is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <device_name>

Procedure

1. Open your Compute environment file.

2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

   • AggregateInstanceExtraSpecsFilter: Add this value to filter Compute nodes by host aggregate metadata that match flavor extra specs.

     Note

     For this filter to perform as expected, you must scope the flavor extra specs by prefixing the extra_specs key with the aggregate_instance_extra_specs: namespace.

   • AggregateImagePropertiesIsolation: Add this value to filter Compute nodes by host aggregate metadata that match image metadata properties.

     Note

     To filter host aggregate metadata by using image metadata properties, the host aggregate metadata key must match a valid image metadata property. For information about valid image metadata properties, see Image metadata in the Creating and Managing Images guide.

   • AvailabilityZoneFilter: Add this value to filter by availability zone when launching an instance.

     Note

     Instead of using the AvailabilityZoneFilter Compute scheduler service filter, you can use the Placement service to process availability zone requests. For more information, see Filtering by availability zone using the Placement service.

3. Save the updates to your Compute environment file.

4. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. To create a host aggregate, enter the following command:

   (overcloud)# openstack aggregate create <aggregate_name>

   Replace <aggregate_name> with the name you want to assign to the host aggregate.

2. Add metadata to the host aggregate:

   (overcloud)# openstack aggregate set \
    --property <key=value> \
    --property <key=value> \
    <aggregate_name>

   • Replace <key=value> with the metadata key-value pair. If you are using the AggregateInstanceExtraSpecsFilter filter, the key can be any arbitrary string, for example, ssd=true. If you are using the AggregateImagePropertiesIsolation filter, the key must match a valid image metadata property. For more information about valid image metadata properties, see Image metadata.
   • Replace <aggregate_name> with the name of the host aggregate.

3. Add the Compute nodes to the host aggregate:

   (overcloud)# openstack aggregate add host \
    <aggregate_name> \
    <host_name>

   • Replace <aggregate_name> with the name of the host aggregate to add the Compute node to.
   • Replace <host_name> with the name of the Compute node to add to the host aggregate.

4. Create a flavor or image for the host aggregate:

   • Create a flavor:

     (overcloud)$ openstack flavor create \
       --ram <size_mb> \
       --disk <size_gb> \
       --vcpus <no_reserved_vcpus> \
       host-agg-flavor

   • Create an image:

     (overcloud)$ openstack image create host-agg-image

5. Set one or more key-value pairs on the flavor or image that match the key-value pairs on the host aggregate.

   • To set the key-value pairs on a flavor, use the scope aggregate_instance_extra_specs:

     (overcloud)# openstack flavor set \
      --property aggregate_instance_extra_specs:ssd=true \
      host-agg-flavor

   • To set the key-value pairs on an image, use valid image metadata properties as the key:

     (overcloud)# openstack image set \
      --property os_type=linux \
      host-agg-image

Procedure

1. To create an availability zone, you can create a new availability zone host aggregate, or make an existing host aggregate an availability zone:

   1. To create a new availability zone host aggregate, enter the following command:

      (overcloud)# openstack aggregate create \
       --zone <availability_zone> \
       <aggregate_name>

      • Replace <availability_zone> with the name you want to assign to the availability zone.
      • Replace <aggregate_name> with the name you want to assign to the host aggregate.

   2. To make an existing host aggregate an availability zone, enter the following command:

      (overcloud)# openstack aggregate set --zone <availability_zone> \
        <aggregate_name>

      • Replace <availability_zone> with the name you want to assign to the availability zone.
      • Replace <aggregate_name> with the name of the host aggregate.

2. Optional: Add metadata to the availability zone:

   (overcloud)# openstack aggregate set --property <key=value> \
     <aggregate_name>

   • Replace <key=value> with your metadata key-value pair. You can add as many key-value properties as required.
   • Replace <aggregate_name> with the name of the availability zone host aggregate.

3. Add Compute nodes to the availability zone host aggregate:

   (overcloud)# openstack aggregate add host <aggregate_name> \
     <host_name>

   • Replace <aggregate_name> with the name of the availability zone host aggregate to add the Compute node to.
   • Replace <host_name> with the name of the Compute node to add to the availability zone.

Procedure

1. To view a list of all the Compute nodes assigned to the host aggregate, enter the following command:

   (overcloud)# openstack aggregate show <aggregate_name>

2. To remove all assigned Compute nodes from the host aggregate, enter the following command for each Compute node:

   (overcloud)# openstack aggregate remove host <aggregate_name> \
     <host_name>

   • Replace <aggregate_name> with the name of the host aggregate to remove the Compute node from.
   • Replace <host_name> with the name of the Compute node to remove from the host aggregate.

3. After you remove all the Compute nodes from the host aggregate, enter the following command to delete the host aggregate:

   (overcloud)# openstack aggregate delete <aggregate_name>

Procedure

1. Open your Compute environment file.
2. To schedule project instances on the project-isolated host aggregate, set the NovaSchedulerLimitTenantsToPlacementAggregate parameter to True in the Compute environment file.

3. Optional: To ensure that only the projects that you assign to a host aggregate can create instances on your cloud, set the NovaSchedulerPlacementAggregateRequiredForTenants parameter to True.

   Note

   NovaSchedulerPlacementAggregateRequiredForTenants is False by default. When this parameter is False, projects that are not assigned to a host aggregate can create instances on any host aggregate.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml \

6. Create the host aggregate.

7. Retrieve the list of project IDs:

   (overcloud)# openstack project list

8. Use the filter_tenant_id<suffix> metadata key to assign projects to the host aggregate:

   (overcloud)# openstack aggregate set \
    --property filter_tenant_id<ID0>=<project_id0> \
    --property filter_tenant_id<ID1>=<project_id1> \
    ...
    --property filter_tenant_id<IDn>=<project_idn> \
    <aggregate_name>

   • Replace <ID0>, <ID1>, and all IDs up to <IDn> with unique values for each project filter that you want to create.
   • Replace <project_id0>, <project_id1>, and all project IDs up to <project_idn> with the ID of each project that you want to assign to the host aggregate.

   • Replace <aggregate_name> with the name of the project-isolated host aggregate.

     For example, use the following syntax to assign projects 78f1, 9d3t, and aa29 to the host aggregate project-isolated-aggregate:

     (overcloud)# openstack aggregate set \
      --property filter_tenant_id0=78f1 \
      --property filter_tenant_id1=9d3t \
      --property filter_tenant_id2=aa29 \
      project-isolated-aggregate

     Tip

     You can create a host aggregate that is available only to a single specific project by omitting the suffix from the filter_tenant_id metadata key:

     (overcloud)# openstack aggregate set \
      --property filter_tenant_id=78f1 \
      single-project-isolated-aggregate

Procedure

1. Open your Compute environment file.

2. Add the filters you want the scheduler to use to the NovaSchedulerDefaultFilters parameter, for example:

   parameter_defaults:
     NovaSchedulerDefaultFilters: AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter

3. Specify which attribute to use to calculate the weight of each Compute node, for example:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         filter_scheduler/weight_classes:
           value: nova.scheduler.weights.all_weighers

   For more information on the available attributes, see Compute scheduler weights.

4. Optional: Configure the multiplier to apply to each weigher. For example, to specify that the available RAM of a Compute node has a higher weight than the other default weighers, and that the Compute scheduler prefers Compute nodes with more available RAM over those nodes with less available RAM, use the following configuration:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         filter_scheduler/weight_classes:
           value: nova.scheduler.weights.all_weighers
         filter_scheduler/ram_weight_multiplier:
           value: 2.0

   Tip

   You can also set multipliers to a negative value. In the above example, to prefer Compute nodes with less available RAM over those nodes with more available RAM, set ram_weight_multiplier to -2.0.

5. Save the updates to your Compute environment file.

6. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

1. The scheduler normalizes each weight to a value between 0.0 and 1.0.
2. The scheduler multiplies the normalized weight by the weigher multiplier.

Procedure

1. Create a flavor that specifies the basic resources to make available to an instance:

   (overcloud)$ openstack flavor create --ram <size_mb> \
    --disk <size_gb> --vcpus <no_vcpus> \
    [--private --project <project_id>] <flavor_name>

   • Replace <size_mb> with the size of RAM to allocate to an instance created with this flavor.
   • Replace <size_gb> with the size of root disk to allocate to an instance created with this flavor.
   • Replace <no_vcpus> with the number of vCPUs to reserve for an instance created with this flavor.

   • Optional: Specify the --private and --project options to make the flavor accessible only by a particular project or group of users. Replace <project_id> with the ID of the project that can use this flavor to create instances. If you do not specify the accessibility, the flavor defaults to public, which means that it is available to all projects.

     Note

     You cannot make a public flavor private after it has been created.

   • Replace <flavor_name> with a unique name for your flavor.

     For more information about flavor arguments, see Flavor arguments.

2. Optional: To specify flavor metadata, set the required properties by using key-value pairs:

   (overcloud)$ openstack flavor set \
    --property <key=value> --property <key=value> ... <flavor_name>

   • Replace <key> with the metadata key of the property you want to allocate to an instance that is created with this flavor. For a list of available metadata keys, see Flavor metadata.
   • Replace <value> with the value of the metadata key you want to allocate to an instance that is created with this flavor.

   • Replace <flavor_name> with the name of your flavor.

     For example, an instance that is launched by using the following flavor has two CPU sockets, each with two CPUs:

     (overcloud)$ openstack flavor set \
      --property hw:cpu_sockets=2 \
      --property hw:cpu_cores=2 processor_topology_flavor

Procedure

1. Open your Compute environment file.

2. To always attach a config drive when launching an instance, set the following parameter to True:

   parameter_defaults:
     ComputeExtraConfig:
       nova::compute::force_config_drive: 'true'

3. Optional: To change the format of the config drive from the default value of iso9660 to vfat, add the config_drive_format parameter to your configuration:

   parameter_defaults:
     ComputeExtraConfig:
       nova::compute::force_config_drive: 'true'
       nova::compute::config_drive_format: vfat

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

Verification

1. Create an instance:

   (overcloud)$ openstack server create --flavor m1.tiny \
    --image cirros test-config-drive-instance

2. Log in to the instance.

3. Mount the config drive:

   • If the instance OS uses udev:

     # mkdir -p /mnt/config
     # mount /dev/disk/by-label/config-2 /mnt/config

   • If the instance OS does not use udev, you need to first identify the block device that corresponds to the config drive:

     # blkid -t LABEL="config-2" -odevice
     /dev/vdb
     # mkdir -p /mnt/config
     # mount /dev/vdb /mnt/config

4. Inspect the files in the mounted config drive directory, mnt/config/openstack/{version}/, for your metadata.

Procedure

1. Create the JSON file for the metadata.
2. Open your Compute environment file.

3. Add the path to the JSON file to your environment file:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_jsonfile_path:
           value: <path_to_the_JSON_file>

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

Procedure

1. Open your Compute environment file.

2. Add DynamicJSON to the vendordata provider module:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_providers:
           value: StaticJSON,DynamicJSON

3. Specify the REST services to contact to generate the metadata. You can specify as many target REST services as required, for example:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_providers:
           value: StaticJSON,DynamicJSON
         api/vendordata_dynamic_targets:
           value: target1@http://127.0.0.1:125
         api/vendordata_dynamic_targets:
           value: target2@http://127.0.0.1:126

   The Compute service generates the JSON file, vendordata2.json, to contain the metadata retrieved from the configured target services, and stores it in the config drive directory.

   Note

   Do not use the same name for a target service more than once.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file that includes the ComputeAMDSEV role, along with any other roles that you need for the overcloud. The following example generates the roles data file roles_data_amd_sev.yaml, which includes the roles Controller and ComputeAMDSEV:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_amd_sev.yaml \
    Compute:ComputeAMDSEV Controller

4. Open roles_data_amd_sev.yaml and edit or add the following parameters and sections:

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputeAMDSEV
   Role name	name: Compute	name: ComputeAMDSEV
   description	Basic Compute Node role	AMD SEV Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputeamdsev-%index%
   deprecated_nic_config_name	compute.yaml	compute-amd-sev.yaml

5. Register the AMD SEV Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in the Director Installation and Usage guide.

7. Create the compute-amd-sev overcloud flavor for AMD SEV Compute nodes:

   (undercloud)$ openstack flavor create --id auto \
    --ram <ram_size_mb> --disk <disk_size_gb> \
    --vcpus <no_vcpus> compute-amd-sev

   • Replace <ram_size_mb> with the RAM of the bare metal node, in MB.
   • Replace <disk_size_gb> with the size of the disk on the bare metal node, in GB.

   • Replace <no_vcpus> with the number of CPUs on the bare metal node.

     Note

     These properties are not used for scheduling instances. However, the Compute scheduler does use the disk size to determine the root partition size.

8. Retrieve a list of your nodes to identify their UUIDs:

   (undercloud)$ openstack baremetal node list

9. Tag each bare metal node that you want to designate for memory encryption with a custom AMD SEV resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.AMD-SEV <node>

   Replace <node> with the ID of the bare metal node.

10. Associate the compute-amd-sev flavor with the custom AMD SEV resource class:

    (undercloud)$ openstack flavor set \
     --property resources:CUSTOM_BAREMETAL_AMD_SEV=1 \
      compute-amd-sev

    To determine the name of a custom resource class that corresponds to a resource class of a Bare Metal service node, convert the resource class to uppercase, replace each punctuation mark with an underscore, and prefix with CUSTOM_.

    Note

    A flavor can request only one instance of a bare metal resource class.

11. Set the following flavor properties to prevent the Compute scheduler from using the bare metal flavor properties to schedule instances:

    (undercloud)$ openstack flavor set \
     --property resources:VCPU=0 --property resources:MEMORY_MB=0 \
     --property resources:DISK_GB=0 compute-amd-sev

12. Optional: If the network topology of the ComputeAMDSEV role is different from the network topology of your Compute role, then create a custom network interface template. For more information, see Custom network interface templates in the Advanced Overcloud Customization guide.

    If the network topology of the ComputeAMDSEV role is the same as the Compute role, then you can use the default network topology defined in compute.yaml.

13. Register the Net::SoftwareConfig of the ComputeAMDSEV role in your network-environment.yaml file:

    resource_registry:
      OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
      OS::TripleO::ComputeCPUPinning::Net::SoftwareConfig: /home/stack/templates/nic-configs/<amd_sev_net_top>.yaml
      OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml

    Replace <amd_sev_net_top> with the name of the file that contains the network topology of the ComputeAMDSEV role, for example, compute.yaml to use the default network topology.

14. Add the following parameters to the node-info.yaml file to specify the number of AMD SEV Compute nodes, and the flavor that you want to use for the AMD SEV designated Compute nodes:

    parameter_defaults:
      OvercloudComputeAMDSEVFlavor: compute-amd-sev
      ComputeAMDSEVCount: 3

15. To verify that the role was created, enter the following command:

    (undercloud)$ openstack baremetal node list --long -c "UUID" \
     -c "Instance UUID" -c "Resource Class" -c "Provisioning State" \
     -c "Power State" -c "Last Error" -c "Fault" -c "Name" -f json

    Example output:

    [
      {
        "Fault": null,
        "Instance UUID": "e8e60d37-d7c7-4210-acf7-f04b245582ea",
        "Last Error": null,
        "Name": "compute-0",
        "Power State": "power on",
        "Provisioning State": "active",
        "Resource Class": "baremetal.AMD-SEV",
        "UUID": "b5a9ac58-63a7-49ba-b4ad-33d84000ccb4"
      },
      {
        "Fault": null,
        "Instance UUID": "3ec34c0b-c4f5-4535-9bd3-8a1649d2e1bd",
        "Last Error": null,
        "Name": "compute-1",
        "Power State": "power on",
        "Provisioning State": "active",
        "Resource Class": "compute",
        "UUID": "432e7f86-8da2-44a6-9b14-dfacdf611366"
      },
      {
        "Fault": null,
        "Instance UUID": "4992c2da-adde-41b3-bef1-3a5b8e356fc0",
        "Last Error": null,
        "Name": "controller-0",
        "Power State": "power on",
        "Provisioning State": "active",
        "Resource Class": "controller",
        "UUID": "474c2fc8-b884-4377-b6d7-781082a3a9c0"
      }
    ]

Procedure

1. Open your Compute environment file.

2. Optional: Add the following configuration to your Compute environment file to specify the maximum number of memory-encrypted instances that the AMD SEV Compute nodes can host concurrently:

   parameter_defaults:
     ComputeAMDSEVExtraConfig:
       nova::config::nova_config:
         libvirt/num_memory_encrypted_guests:
           value: 15

   Note

   The default value of the libvirt/num_memory_encrypted_guests parameter is none. If you do not set a custom value, the AMD SEV Compute nodes do not impose a limit on the number of memory-encrypted instances that the nodes can host concurrently. Instead, the hardware determines the maximum number of memory-encrypted instances that the AMD SEV Compute nodes can host concurrently, which might cause some memory-encrypted instances to fail to launch.

3. Optional: To specify that all x86_64 images use the q35 machine type by default, add the following configuration to your Compute environment file:

   parameter_defaults:
     ComputeAMDSEVParameters:
       NovaHWMachineType: x86_64=q35

   If you specify this parameter value, you do not need to set the hw_machine_type property to q35 on every AMD SEV instance image.

4. To ensure that the AMD SEV Compute nodes reserve enough memory for host-level services to function, add 16MB for each potential AMD SEV instance:

   parameter_defaults:
     ComputeAMDSEVParameters:
       ...
       NovaReservedHostMemory: <libvirt/num_memory_encrypted_guests * 16>

5. Configure the kernel parameters for the AMD SEV Compute nodes:

   parameter_defaults:
     ComputeAMDSEVParameters:
       ...
       KernelArgs: "hugepagesz=1GB hugepages=32 default_hugepagesz=1GB mem_encrypt=on kvm_amd.sev=1"

6. Save the updates to your Compute environment file.

7. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Create a new image for memory encryption:

   (overcloud)$ openstack image create ...  \
    --property hw_firmware_type=uefi amd-sev-image

   Note

   If you use an existing image, the image must have the hw_firmware_type property set to uefi.

2. Optional: Add the property hw_mem_encryption=True to the image to enable AMD SEV memory encryption on the image:

   (overcloud)$ openstack image set  \
    --property hw_mem_encryption=True amd-sev-image

   Tip

   You can enable memory encryption on the flavor. For more information, see Creating a flavor for memory encryption.

3. Optional: Set the machine type to q35, if not already set in the Compute node configuration:

   (overcloud)$ openstack image set  \
    --property hw_machine_type=q35 amd-sev-image

4. Optional: To schedule memory-encrypted instances on a SEV-capable host aggregate, add the following trait to the image extra specs:

   (overcloud)$ openstack image set  \
    --property trait:HW_CPU_X86_AMD_SEV=required amd-sev-image

   Tip

   You can also specify this trait on the flavor. For more information, see Creating a flavor for memory encryption.

Procedure

1. Create a flavor for memory encryption:

   (overcloud)$ openstack flavor create --vcpus 1 --ram 512 --disk 2  \
    --property hw:mem_encryption=True m1.small-amd-sev

2. To schedule memory-encrypted instances on a SEV-capable host aggregate, add the following trait to the flavor extra specs:

   (overcloud)$ openstack flavor set  \
    --property trait:HW_CPU_X86_AMD_SEV=required m1.small-amd-sev

Procedure

1. Create an instance by using an AMD SEV flavor or image. The following example creates an instance by using the flavor created in Creating a flavor for memory encryption and the image created in Creating an image for memory encryption:

   (overcloud)$ openstack server create --flavor m1.small-amd-sev \
    --image amd-sev-image amd-sev-instance

2. Log in to the instance as a cloud user.

3. To verify that the instance uses memory encryption, enter the following command from the instance:

   $ dmesg | grep -i sev
   AMD Secure Encrypted Virtualization (SEV) active

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data_pmem.yaml that includes the Controller, Compute, and ComputePMEM roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_pmem.yaml \
    Compute:ComputePMEM Compute Controller

4. Open roles_data_pmem.yaml and edit or add the following parameters and sections:

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputePMEM
   Role name	name: Compute	name: ComputePMEM
   description	Basic Compute Node role	PMEM Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepmem-%index%
   deprecated_nic_config_name	compute.yaml	compute-pmem.yaml

5. Register the NVDIMM Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in the Director Installation and Usage guide.

7. Create the compute-pmem overcloud flavor for PMEM Compute nodes:

   (undercloud)$ openstack flavor create --id auto \
    --ram <ram_size_mb> --disk <disk_size_gb> \
    --vcpus <no_vcpus> compute-pmem

   • Replace <ram_size_mb> with the RAM of the bare metal node, in MB.
   • Replace <disk_size_gb> with the size of the disk on the bare metal node, in GB.

   • Replace <no_vcpus> with the number of CPUs on the bare metal node.

     Note

     These properties are not used for scheduling instances. However, the Compute scheduler does use the disk size to determine the root partition size.

8. Retrieve a list of your nodes to identify their UUIDs:

   (undercloud)$ openstack baremetal node list

9. Tag each bare metal node that you want to designate for PMEM workloads with a custom PMEM resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.PMEM <node>

   Replace <node> with the ID of the bare metal node.

10. Associate the compute-pmem flavor with the custom PMEM resource class:

    (undercloud)$ openstack flavor set \
     --property resources:CUSTOM_BAREMETAL_PMEM=1 \
      compute-pmem

    To determine the name of a custom resource class that corresponds to a resource class of a Bare Metal service node, convert the resource class to uppercase, replace all punctuation with an underscore, and prefix with CUSTOM_.

    Note

    A flavor can request only one instance of a bare metal resource class.

11. Set the following flavor properties to prevent the Compute scheduler from using the bare metal flavor properties to schedule instances:

    (undercloud)$ openstack flavor set \
     --property resources:VCPU=0 --property resources:MEMORY_MB=0 \
     --property resources:DISK_GB=0 compute-pmem

12. Add the following parameters to the node-info.yaml file to specify the number of PMEM Compute nodes, and the flavor to use for the PMEM-designated Compute nodes:

    parameter_defaults:
      OvercloudComputePMEMFlavor: compute-pmem
      ComputePMEMCount: 3 #set to the no of NVDIMM devices you have

13. To verify that the role was created, enter the following command:

    (undercloud)$ openstack overcloud profiles list

Procedure

1. Create a new Compute environment file for configuring NVDIMM Compute nodes, for example, env_pmem.yaml.

2. To partition the NVDIMM regions into PMEM namespaces that the instances can use, add the NovaPMEMNamespaces role-specific parameter to the PMEM role in your Compute environment file, and set the value using the following format:

   <size>:<namespace_name>[,<size>:<namespace_name>]

   Use the following suffixes to represent the size:

   • "k" or "K" for KiB
   • "m" or "M" for MiB
   • "G" or "G" for GiB

   • "t" or "T" for TiB

     For example, the following configuration creates four namespaces, three of size 6 GiB, and one of size 100 GiB:

     parameter_defaults:
       ComputePMEMParameters:
         NovaPMEMNamespaces: "6G:ns0,6G:ns1,6G:ns2,100G:ns3"

3. To map the PMEM namespaces to labels that can be used in a flavor, add the NovaPMEMMappings role-specific parameter to the PMEM role in your Compute environment file, and set the value using the following format:

   <label>:<namespace_name>[|<namespace_name>][,<label>:<namespace_name>[|<namespace_name>]].

   For example, the following configuration maps the three 6 GiB namespaces to the label "6GB", and the 100 GiB namespace to the label "LARGE":

   parameter_defaults:
     ComputePMEMParameters:
       NovaPMEMNamespaces: "6G:ns0,6G:ns1,6G:ns2,100G:ns3"
       NovaPMEMMappings: "6GB:ns0|ns1|ns2,LARGE:ns3"

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -r /home/stack/templates/roles_data_pmem.yaml \
    -e /home/stack/templates/node-info.yaml \
    -e [your environment files] \
    -e /home/stack/templates/env_pmem.yaml

6. Create and configure the flavors that your cloud users can use to launch instances that have vPMEM. The following example creates a flavor that requests a small PMEM device, 6GB, as mapped in Step 3:

   (overcloud)$ openstack flavor create --vcpus 1 --ram 512 --disk 2  \
    --property hw:pmem='6GB' small_pmem_flavor

Verification

1. Create an instance using one of the PMEM flavors:

   (overcloud)$ openstack flavor list
   (overcloud)$ openstack server create --flavor small_pmem_flavor \
    --image rhel8 pmem_instance

2. Log in to the instance as a cloud user. For more information, see Connecting to an instance.

3. List all the disk devices attached to the instance:

   $ sudo fdisk -l /dev/pmem0

   The instance has vPMEM if one of the devices listed is NVDIMM.

1. Prepare the GPU role, profile, and flavor for designating Compute nodes for vGPU.
2. Configure the Compute node for vGPU.
3. Deploy the overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the overcloudrc credential file:

   $ source ~/overcloudrc

3. Create an instance with the hardware and software profile that your vGPU instances require:

   (overcloud)$ openstack server create --flavor <flavor> \
    --image <image> temp_vgpu_instance

   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require. For information about creating a vGPU flavor, see Creating a vGPU flavor for instances.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information about downloading RHEL cloud images, see Image service.
4. Log in to the instance as a cloud user. For more information, see Connecting to an instance.
5. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

6. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances. For more information about supported drivers, see Image metadata.

7. Create an image snapshot of the instance:

   (overcloud)$ openstack server image create \
    --name vgpu_image temp_vgpu_instance

8. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor, for example:

   (overcloud)$ openstack flavor create --vcpus 6 \
    --ram 8192 --disk 100 m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

2. Assign a vGPU resource to the flavor that you created. You can assign only one vGPU for each instance.

   (overcloud)$ openstack flavor set m1.small-gpu \
    --property "resources:VGPU=1"
   
   (overcloud)$ openstack flavor show m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | access_project_ids         | None                                 |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 | resources:VGPU='1'                   |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

Procedure

1. Create an instance using a GPU flavor and image, for example:

   (overcloud)$ openstack server create --flavor m1.small-gpu \
    --image vgpu_image --security-group web --nic net-id=internal0 \
    --key-name lambda vgpu-instance

2. Log in to the instance as a cloud user. For more information, see Connecting to an instance.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

Procedure

1. To determine the vendor ID and product ID for each passthrough device type, enter the following command on the physical server that has the PCI cards:

   # lspci -nn | grep -i <gpu_name>

   For example, to determine the vendor and product ID for an NVIDIA GPU, enter the following command:

   # lspci -nn | grep -i nvidia
   3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
   d8:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1db4] (rev a1)

2. To determine if each PCI device has Single Root I/O Virtualization (SR-IOV) capabilities, enter the following command on the physical server that has the PCI cards:

   # lspci -v -s 3b:00.0
   3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
             ...
             Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
             ...

3. To configure the Controller node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_controller.yaml.

4. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthru_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

5. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthru_controller.yaml:

   • If the PCI device has SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     For more information on configuring the device_type field, see PCI passthrough device type field.

     Note

     If the nova-api service is running in a role other than the Controller, then replace ControllerExtraConfig with the user role, in the format <Role>ExtraConfig.

6. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_compute.yaml.

7. To specify the available PCIs for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   parameter_defaults:
     NovaPCIPassthrough:
       - vendor_id: "10de"
         product_id: "1eb8"

8. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   • If the PCI device has SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     Note

     The Compute node aliases must be identical to the aliases on the Controller node.

9. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthru_compute.yaml:

   parameter_defaults:
     ...
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

10. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
      -e [your environment files] \
      -e /home/stack/templates/pci_passthru_controller.yaml \
      -e /home/stack/templates/pci_passthru_compute.yaml

11. Configure a flavor to request the PCI devices. The following example requests two devices, each with a vendor ID of 10de and a product ID of 13f2:

    # openstack flavor set m1.large \
     --property "pci_passthrough:alias"="t4:2"

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor m1.large \
    --image <custom_gpu> --wait test-pci

   Replace <custom_gpu> with the name of your custom instance image that has the required GPU driver installed.

2. Log in to the instance as a cloud user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

4. To check the NVIDIA System Management Interface status, enter the following command from the instance:

   $ nvidia-smi

   Example output:

   -----------------------------------------------------------------------------
   | NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
   |---------------------------------------------------------------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |===========================================================================|
   |   0  Tesla T4            Off  | 00000000:01:00.0 Off |                    0 |
   | N/A   43C    P0    20W /  70W |      0MiB / 15109MiB |      0%      Default |
   ---------------------------------------------------------------------------
   
   -----------------------------------------------------------------------------
   | Processes:                                                       GPU Memory |
   |  GPU       PID   Type   Process name                             Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   -----------------------------------------------------------------------------

Procedure

1. To build the real-time overcloud image, you must enable the rhel-8-for-x86_64-nfv-rpms repository for RT-KVM. To check which packages will be installed from the repository, enter the following command:

   $ dnf repo-pkgs rhel-8-for-x86_64-nfv-rpms list
   Loaded plugins: product-id, search-disabled-repos, subscription-manager
   Available Packages
   kernel-rt.x86_64                   4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-debug.x86_64             4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-debug-devel.x86_64       4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-debug-kvm.x86_64         4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-devel.x86_64             4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-doc.noarch               4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-kvm.x86_64               4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   [ output omitted…]

2. To build the overcloud image for Real-time Compute nodes, install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud)$ sudo dnf install libguestfs-tools

   Important

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

3. Extract the images:

   (undercloud)$ tar -xf /usr/share/rhosp-director-images/overcloud-full.tar
   (undercloud)$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent.tar

4. Copy the default image:

   (undercloud)$ cp overcloud-full.qcow2 overcloud-realtime-compute.qcow2

5. Register the image and configure the required subscriptions:

   (undercloud)$ virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager register --username=<username> --password=<password>'
   [  0.0] Examining the guest ...
   [ 10.0] Setting a random seed
   [ 10.0] Running: subscription-manager register --username=<username> --password=<password>
   [ 24.0] Finishing off

   Replace the username and password values with your Red Hat customer account details.

   For general information about building a Real-time overcloud image, see the knowledgebase article Modifying the Red Hat Enterprise Linux OpenStack Platform Overcloud Image with virt-customize.

6. Find the SKU of the Red Hat OpenStack Platform for Real Time subscription. The SKU might be located on a system that is already registered to the Red Hat Subscription Manager with the same account and credentials:

   $ sudo subscription-manager list

7. Attach the Red Hat OpenStack Platform for Real Time subscription to the image:

   (undercloud)$  virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager attach --pool [subscription-pool]'

8. Create a script to configure rt on the image:

   (undercloud)$ cat rt.sh
     #!/bin/bash
   
     set -eux
   
     subscription-manager repos --enable=[REPO_ID]
     dnf -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     dnf -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
   
     # END OF SCRIPT

9. Run the script to configure the real-time image:

   (undercloud)$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

10. Re-label SELinux:

    (undercloud)$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

11. Extract vmlinuz and initrd. For example:

    (undercloud)$ mkdir image
    (undercloud)$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud)$ cp image/boot/vmlinuz-4.18.0-80.7.1.rt9.153.el8_0.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud)$ cp image/boot/initramfs-4.18.0-80.7.1.rt9.153.el8_0.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud)$ guestunmount image

    Note

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

12. Upload the image:

    (undercloud)$ openstack overcloud image upload \
     --update-existing --os-image-name
     overcloud-realtime-compute.qcow2

    You now have a real-time image you can use with the ComputeRealTime composable role on select Compute nodes.

13. To reduce latency on your Real-time Compute nodes, you must modify the BIOS settings in the Compute nodes. You should disable all options for the following components in your Compute node BIOS settings:

    • Power Management
    • Hyper-Threading
    • CPU sleep states

    • Logical processors

      See Setting BIOS parameters for descriptions of these settings and the impact of disabling them. See your hardware manufacturer documentation for complete details on how to change BIOS settings.

Procedure

1. Based on the /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml file, create a compute-real-time.yaml environment file that sets the parameters for the ComputeRealTime role.

   cp /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml /home/stack/templates/compute-real-time.yaml

   The file must include values for the following parameters:

   • IsolCpusList and NovaComputeCpuDedicatedSet: List of isolated CPU cores and virtual CPU pins to reserve for real-time workloads. This value depends on the CPU hardware of your real-time Compute nodes.
   • NovaComputeCpuSharedSet: List of host CPUs to reserve for emulator threads.
   • KernelArgs: Arguments to pass to the kernel of the Real-time Compute nodes. For example, you can use default_hugepagesz=1G hugepagesz=1G hugepages=<number_of_1G_pages_to_reserve> hugepagesz=2M hugepages=<number_of_2M_pages> to define the memory requirements of guests that have huge pages with multiple sizes. In this example, the default size is 1GB but you can also reserve 2M huge pages.
   • NovaComputeDisableIrqBalance: Ensure that this parameter is set to true for Real-time Compute nodes, because the tuned service manages IRQ balancing for real-time deployments instead of the irqbalance service.

2. Add the ComputeRealTime role to your roles data file and regenerate the file. For example:

   $ openstack overcloud roles generate -o /home/stack/templates/rt_roles_data.yaml Controller Compute ComputeRealTime

   This command generates a ComputeRealTime role with contents similar to the following example, and also sets the ImageDefault option to overcloud-realtime-compute.

   - name: ComputeRealTime
     description: |
       Compute role that is optimized for real-time behaviour. When using this role
       it is mandatory that an overcloud-realtime-compute image is available and
       the role specific parameters IsolCpusList, NovaComputeCpuDedicatedSet and
       NovaComputeCpuSharedSet are set accordingly to the hardware of the real-time compute nodes.
     CountDefault: 1
     networks:
       InternalApi:
         subnet: internal_api_subnet
       Tenant:
         subnet: tenant_subnet
       Storage:
         subnet: storage_subnet
     HostnameFormatDefault: '%stackname%-computerealtime-%index%'
     ImageDefault: overcloud-realtime-compute
     RoleParametersDefault:
       TunedProfileName: "realtime-virtual-host"
       KernelArgs: ""      # these must be set in an environment file
       IsolCpusList: ""    # or similar according to the hardware
       NovaComputeCpuDedicatedSet: ""  # of real-time nodes
       NovaComputeCpuSharedSet: ""     #
       NovaLibvirtMemStatsPeriodSeconds: 0
     ServicesDefault:
       - OS::TripleO::Services::Aide
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::BootParams
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::ComputeCeilometerAgent
       - OS::TripleO::Services::ComputeNeutronCorePlugin
       - OS::TripleO::Services::ComputeNeutronL3Agent
       - OS::TripleO::Services::ComputeNeutronMetadataAgent
       - OS::TripleO::Services::ComputeNeutronOvsAgent
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::IpaClient
       - OS::TripleO::Services::Ipsec
       - OS::TripleO::Services::Iscsid
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::LoginDefs
       - OS::TripleO::Services::MetricsQdr
       - OS::TripleO::Services::MySQLClient
       - OS::TripleO::Services::NeutronBgpVpnBagpipe
       - OS::TripleO::Services::NeutronLinuxbridgeAgent
       - OS::TripleO::Services::NeutronVppAgent
       - OS::TripleO::Services::NovaCompute
       - OS::TripleO::Services::NovaLibvirt
       - OS::TripleO::Services::NovaLibvirtGuests
       - OS::TripleO::Services::NovaMigrationTarget
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::OpenDaylightOvs
       - OS::TripleO::Services::Podman
       - OS::TripleO::Services::Rhsm
       - OS::TripleO::Services::RsyslogSidecar
       - OS::TripleO::Services::Securetty
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::SkydiveAgent
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Sshd
       - OS::TripleO::Services::Timesync
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Vpp
       - OS::TripleO::Services::OVNController
       - OS::TripleO::Services::OVNMetadataAgent

   For general information about custom roles and about the roles-data.yaml, see Roles.

3. Create the compute-realtime flavor to tag nodes that you want to designate for real-time workloads. For example:

   $ source ~/stackrc
   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-realtime
   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute-realtime" compute-realtime

4. Tag each node that you want to designate for real-time workloads with the compute-realtime profile.

   $ openstack baremetal node set --property capabilities='profile:compute-realtime,boot_option:local' <node_uuid>

5. Map the ComputeRealTime role to the compute-realtime flavor by creating an environment file with the following content:

   parameter_defaults:
     OvercloudComputeRealTimeFlavor: compute-realtime

6. Add your environment files and the new roles file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -r /home/stack/templates/rt~/my_roles_data.yaml \
    -e home/stack/templates/compute-real-time.yaml

Procedure

1. Create the compute-real-time.yaml file with the following parameters:

   parameter_defaults:
     ComputeRealTimeParameters:
       IsolCpusList: "1"
       NovaComputeCpuDedicatedSet: "1"
       NovaComputeCpuSharedSet: "0"
       KernelArgs: "default_hugepagesz=1G hugepagesz=1G hugepages=16"
       NovaComputeDisableIrqBalance: true

2. Create a new rt_roles_data.yaml file with the ComputeRealTime role:

   $ openstack overcloud roles generate \
    -o ~/rt_roles_data.yaml Controller ComputeRealTime

3. Add compute-real-time.yaml and rt_roles_data.yaml to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -r /home/stack/rt_roles_data.yaml \
     -e [your environment files] \
     -e /home/stack/templates/compute-real-time.yaml

   This command deploys one Controller node and one Real-time Compute node.

4. Log into the Real-time Compute node and check the following parameters:

   [root@overcloud-computerealtime-0 ~]# uname -a
   Linux overcloud-computerealtime-0 4.18.0-80.7.1.rt9.153.el8_0.x86_64 #1 SMP PREEMPT RT Wed Dec 13 13:37:53 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
   [root@overcloud-computerealtime-0 ~]# cat /proc/cmdline
   BOOT_IMAGE=/boot/vmlinuz-4.18.0-80.7.1.rt9.153.el8_0.x86_64 root=UUID=45ae42d0-58e7-44fe-b5b1-993fe97b760f ro console=tty0 crashkernel=auto console=ttyS0,115200 default_hugepagesz=1G hugepagesz=1G hugepages=16
   [root@overcloud-computerealtime-0 ~]# tuned-adm active
   Current active profile: realtime-virtual-host
   [root@overcloud-computerealtime-0 ~]# grep ^isolated_cores /etc/tuned/realtime-virtual-host-variables.conf
   isolated_cores=1
   [root@overcloud-computerealtime-0 ~]# cat /usr/lib/tuned/realtime-virtual-host/lapic_timer_adv_ns
   4000 # The returned value must not be 0
   [root@overcloud-computerealtime-0 ~]# cat /sys/module/kvm/parameters/lapic_timer_advance_ns
   4000 # The returned value must not be 0
   # To validate hugepages at a host level:
   [root@overcloud-computerealtime-0 ~]# cat /proc/meminfo | grep -E HugePages_Total|Hugepagesize
   HugePages_Total:      64
   Hugepagesize:    1048576 kB
   # To validate hugepages on a per NUMA level (below example is a two NUMA compute host):
   [root@overcloud-computerealtime-0 ~]# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
   32
   [root@overcloud-computerealtime-0 ~]# cat /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages
   32
   [root@overcloud-computerealtime-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_dedicated_set
   1
   [root@overcloud-computerealtime-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_shared_set
   0
   [root@overcloud-computerealtime-0 ~]# systemctl status irqbalance
   ● irqbalance.service - irqbalance daemon
      Loaded: loaded (/usr/lib/systemd/system/irqbalance.service; enabled; vendor preset: enabled)
      Active: inactive (dead) since Tue 2021-03-30 13:36:31 UTC; 2s ago

Procedure

1. Launch the real-time instance:

   # openstack server create  --image <rhel> \
    --flavor r1.small --nic net-id=<dpdk_net> test-rt

2. Optional: Verify that the instance uses the assigned emulator threads:

   # virsh dumpxml <instance_id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>