Procedure

1. Copy the /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml file to your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml /home/stack/templates/enable-swap.yaml

2. Configure the swap size by adding the following parameters to your enable-swap.yaml file:

   parameter_defaults:
     swap_size_megabytes: <swap size in MB>
     swap_path: <full path to location of swap, default: /swap>

3. Add the enable_swap.yaml environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/enable-swap.yaml

Procedure

1. Reserve physical CPU cores for the dedicated instances by setting the NovaComputeCpuDedicatedSet configuration in the Compute environment file for each Compute node:

   NovaComputeCpuDedicatedSet=1,3,5,7

2. Reserve physical CPU cores for the shared instances by setting the NovaComputeCpuSharedSet configuration in the Compute environment file for each Compute node:

   NovaComputeCpuSharedSet=2,6

3. Set the NovaReservedHostMemory option in the same files to the amount of RAM to reserve for host processes. For example, if you want to reserve 512 MB, use:

   NovaReservedHostMemory=512

4. To ensure that host processes do not run on the CPU cores reserved for instances, set the parameter IsolCpusList in each Compute environment file to the CPU cores you have reserved for instances. Specify the value of the IsolCpusList parameter using a list, or ranges, of CPU indices separated by a whitespace.

   IsolCpusList=1 2 3 5 6 7

5. To filter out hosts based on its NUMA topology, add NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter in each Compute environment file.

6. Add your Compute environment file(s) to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Open your Compute environment file.

2. Configure the amount of huge page memory to reserve on each NUMA node for processes that are not instances:

   parameter_defaults:
     NovaReservedHugePages: ["node:0,size:2048,count:64","node:1,size:1GB,count:1"]

   • Replace the size value for each node with the size of the allocated huge page. Set to one of the following valid values:

     • 2048 (for 2MB)
     • 1GB
   • Replace the count value for each node with the number of huge pages used by OVS per NUMA node. For example, for 4096 of socket memory used by Open vSwitch, set this to 2.

3. Optional: To allow instances to allocate 1GB huge pages, configure the CPU feature flags, NovaLibvirtCPUModelExtraFlags, to include pdpe1gb:

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: 'custom'
       NovaLibvirtCPUModels: 'Haswell-noTSX'
       NovaLibvirtCPUModelExtraFlags: 'vmx, pdpe1gb'

   Note

   • CPU feature flags do not need to be configured to allow instances to only request 2 MB huge pages.
   • You can only allocate 1G huge pages to an instance if the host supports 1G huge page allocation.
   • You only need to set NovaLibvirtCPUModelExtraFlags to pdpe1gb when NovaLibvirtCPUMode is set to host-model or custom.
   • If the host supports pdpe1gb, and host-passthrough is used as the NovaLibvirtCPUMode, then you do not need to set pdpe1gb as a NovaLibvirtCPUModelExtraFlags. The pdpe1gb flag is only included in Opteron_G4 and Opteron_G5 CPU models, it is not included in any of the Intel CPU models supported by QEMU.
   • To mitigate for CPU hardware issues, such as Microarchitectural Data Sampling (MDS), you might need to configure other CPU flags. For more information, see RHOS Mitigation for MDS ("Microarchitectural Data Sampling") Security Flaws.

4. To avoid loss of performance after applying Meltdown protection, configure the CPU feature flags, NovaLibvirtCPUModelExtraFlags, to include +pcid:

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: 'custom'
       NovaLibvirtCPUModels: 'Haswell-noTSX'
       NovaLibvirtCPUModelExtraFlags: 'vmx, pdpe1gb, +pcid'

   Tip

   For more information, see Reducing the performance impact of Meltdown CVE fixes for OpenStack guests with "PCID" CPU feature flag.

5. Add NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter, if not already present.

6. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files]  \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Open your Compute environment file.

2. Configure the amount of host disk space, in MiB, to make available for instance RAM, by adding the following parameter to your Compute environment file:

   parameter_defaults:
     NovaLibvirtFileBackedMemory: 102400

3. Optional: To configure the directory to store the memory backing files, set the QemuMemoryBackingDir parameter in your Compute environment file. If not set, the memory backing directory defaults to /var/lib/libvirt/qemu/ram/.

   Note

   You must locate your backing store in a directory at or above the default directory location, /var/lib/libvirt/qemu/ram/.

   You can also change the host disk for the backing store. For more information, see Changing the memory backing directory host disk.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Generate a new roles data file named roles_data_pci_passthrough.yaml that includes the Controller, Compute, and ComputePCI roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_pci_passthrough.yaml \
    Compute:ComputePCI Compute Controller

2. Open roles_data_pci_passthrough.yaml and edit or add the following parameters and sections:

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputePCI
   Role name	name: Compute	name: ComputePCI
   description	Basic Compute Node role	PCI Passthrough Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepci-%index%
   deprecated_nic_config_name	compute.yaml	compute-pci-passthrough.yaml

3. Register the PCI passthrough Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.

4. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Inspecting the hardware of nodes in the Director Installation and Usage guide.

5. Create the compute-pci-passthrough bare metal flavor to use to tag nodes that you want to designate for PCI passthrough:

   (undercloud)$ openstack flavor create --id auto \
    --ram <ram_size_mb> --disk <disk_size_gb> \
    --vcpus <no_vcpus> compute-pci-passthrough

   • Replace <ram_size_mb> with the RAM of the bare metal node, in MB.
   • Replace <disk_size_gb> with the size of the disk on the bare metal node, in GB.

   • Replace <no_vcpus> with the number of CPUs on the bare metal node.

     Note

     These properties are not used for scheduling instances. However, the Compute scheduler does use the disk size to determine the root partition size.

6. Tag each bare metal node that you want to designate for PCI passthrough with a custom PCI passthrough resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.PCI-PASSTHROUGH <node>

   Replace <node> with the ID of the bare metal node.

7. Associate the compute-pci-passthrough flavor with the custom PCI passthrough resource class:

   (undercloud)$ openstack flavor set \
    --property resources:CUSTOM_BAREMETAL_PCI_PASSTHROUGH=1 \
     compute-pci-passthrough

   To determine the name of a custom resource class that corresponds to a resource class of a Bare Metal service node, convert the resource class to uppercase, replace all punctuation with an underscore, and prefix with CUSTOM_.

   Note

   A flavor can request only one instance of a bare metal resource class.

8. Set the following flavor properties to prevent the Compute scheduler from using the bare metal flavor properties to schedule instances:

   (undercloud)$ openstack flavor set \
    --property resources:VCPU=0 --property resources:MEMORY_MB=0 \
    --property resources:DISK_GB=0 compute-pci-passthrough

9. Add the following parameters to the node-info.yaml file to specify the number of PCI passthrough Compute nodes, and the flavor to use for the PCI passthrough designated Compute nodes:

   parameter_defaults:
     OvercloudComputePCIFlavor: compute-pci-passthrough
     ComputePCICount: 3

10. To verify that the role was created, enter the following command:

    (undercloud)$ openstack overcloud profiles list

Procedure

1. Create an environment file to configure the Controller node on the overcloud for PCI passthrough, for example, pci_passthrough_controller.yaml.

2. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthrough_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

3. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthrough_controller.yaml:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type": "type-PF"

   For more information about configuring the device_type field, see PCI passthrough device type field.

   Note

   If the nova-api service is running in a role different from the Controller role, replace ControllerExtraConfig with the user role in the format <Role>ExtraConfig.

4. Optional: To set a default NUMA affinity policy for PCI passthrough devices, add numa_policy to the nova::pci::aliases: configuration from step 3:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"
           numa_policy: "preferred"

5. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthrough_compute.yaml.

6. To specify the available PCIs for the devices on the Compute node, use the vendor_id and product_id options to add all matching PCI devices to the pool of PCI devices available for passthrough to instances. For example, to add all Intel® Ethernet Controller X710 devices to the pool of PCI devices available for passthrough to instances, add the following configuration to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIParameters:
       NovaPCIPassthrough:
         - vendor_id: "8086"
           product_id: "1572"

   For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

7. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the PCI passthrough Compute node, add the following to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type": "type-PF"

   Note

   The Compute node aliases must be identical to the aliases on the Controller node. Therefore, if you added numa_affinity to nova::pci::aliases in pci_passthrough_controller.yaml, then you must also add it to nova::pci::aliases in pci_passthrough_compute.yaml.

8. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthrough_compute.yaml. For example, use the following KernalArgs settings to enable an Intel IOMMU:

   parameter_defaults:
     ...
     ComputePCIParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   To enable an AMD IOMMU, set KernelArgs to "amd_iommu=on iommu=pt".

9. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/pci_passthrough_controller.yaml \
     -e /home/stack/templates/pci_passthrough_compute.yaml \

10. Create and configure the flavors that your cloud users can use to request the PCI devices. The following example requests two devices, each with a vendor ID of 8086 and a product ID of 1572, using the alias defined in step 7:

    (overcloud)# openstack flavor set \
     --property "pci_passthrough:alias"="a1:2" device_passthrough

11. Optional: To override the default NUMA affinity policy for PCI passthrough devices, you can add the NUMA affinity policy property key to the flavor or the image:

    • To override the default NUMA affinity policy by using the flavor, add the hw:pci_numa_affinity_policy property key:

      (overcloud)# openstack flavor set \
       --property "hw:pci_numa_affinity_policy"="required" \
       device_passthrough

      For more information about the valid values for hw:pci_numa_affinity_policy, see Flavor metadata.

    • To override the default NUMA affinity policy by using the image, add the hw_pci_numa_affinity_policy property key:

      (overcloud)# openstack image set \
       --property hw_pci_numa_affinity_policy=required \
       device_passthrough_image

      Note

      If you set the NUMA affinity policy on both the image and the flavor then the property values must match. The flavor setting takes precedence over the image and default settings. Therefore, the configuration of the NUMA affinity policy on the image only takes effect if the property is not set on the flavor.

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor device_passthrough \
    --image <image> --wait test-pci

2. Log in to the instance as a cloud user. For more information, see Log in to an Instance.

3. To verify that the PCI device is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <device_name>

Procedure

1. Open your Compute environment file.

2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

   • AggregateInstanceExtraSpecsFilter: Add this value to filter Compute nodes by host aggregate metadata that match flavor extra specs.

     Note

     For this filter to perform as expected, you must scope the flavor extra specs by prefixing the extra_specs key with the aggregate_instance_extra_specs: namespace.

   • AggregateImagePropertiesIsolation: Add this value to filter Compute nodes by host aggregate metadata that match image metadata properties.

     Note

     To filter host aggregate metadata by using image metadata properties, the host aggregate metadata key must match a valid image metadata property. For information about valid image metadata properties, see Image metadata.

   • AvailabilityZoneFilter: Add this value to filter by availability zone when launching an instance.

     Note

     Instead of using the AvailabilityZoneFilter Compute scheduler service filter, you can use the Placement service to process availability zone requests. For more information, see Filtering by availability zone using the Placement service.

3. Save the updates to your Compute environment file.

4. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. To create a host aggregate, enter the following command:

   (overcloud)# openstack aggregate create <aggregate_name>

   Replace <aggregate_name> with the name you want to assign to the host aggregate.

2. Add metadata to the host aggregate:

   (overcloud)# openstack aggregate set \
    --property <key=value> \
    --property <key=value> \
    <aggregate_name>

   • Replace <key=value> with the metadata key-value pair. If you are using the AggregateInstanceExtraSpecsFilter filter, the key can be any arbitrary string, for example, ssd=true. If you are using the AggregateImagePropertiesIsolation filter, the key must match a valid image metadata property. For more information about valid image metadata properties, see Image metadata.
   • Replace <aggregate_name> with the name of the host aggregate.

3. Add the Compute nodes to the host aggregate:

   (overcloud)# openstack aggregate add host \
    <aggregate_name> \
    <host_name>

   • Replace <aggregate_name> with the name of the host aggregate to add the Compute node to.
   • Replace <host_name> with the name of the Compute node to add to the host aggregate.

4. Create a flavor or image for the host aggregate:

   • Create a flavor:

     (overcloud)$ openstack flavor create \
       --ram <size-mb> \
       --disk <size-gb> \
       --vcpus <no_reserved_vcpus> \
       host-agg-flavor

   • Create an image:

     (overcloud)$ openstack image create host-agg-image

5. Set one or more key-value pairs on the flavor or image that match the key-value pairs on the host aggregate.

   • To set the key-value pairs on a flavor, use the scope aggregate_instance_extra_specs:

     (overcloud)# openstack flavor set \
      --property aggregate_instance_extra_specs:ssd=true \
      host-agg-flavor

   • To set the key-value pairs on an image, use valid image metadata properties as the key:

     (overcloud)# openstack image set \
      --property os_type=linux \
      host-agg-image

Procedure

1. To create an availability zone, you can create a new availability zone host aggregate, or make an existing host aggregate an availability zone:

   1. To create a new availability zone host aggregate, enter the following command:

      (overcloud)# openstack aggregate create \
       --zone <availability_zone> \
       <aggregate_name>

      • Replace <availability_zone> with the name you want to assign to the availability zone.
      • Replace <aggregate_name> with the name you want to assign to the host aggregate.

   2. To make an existing host aggregate an availability zone, enter the following command:

      (overcloud)# openstack aggregate set --zone <availability_zone> \
        <aggregate_name>

      • Replace <availability_zone> with the name you want to assign to the availability zone.
      • Replace <aggregate_name> with the name of the host aggregate.

2. Optional: Add metadata to the availability zone:

   (overcloud)# openstack aggregate set --property <key=value> \
     <aggregate_name>

   • Replace <key=value> with your metadata key-value pair. You can add as many key-value properties as required.
   • Replace <aggregate_name> with the name of the availability zone host aggregate.

3. Add Compute nodes to the availability zone host aggregate:

   (overcloud)# openstack aggregate add host <aggregate_name> \
     <host_name>

   • Replace <aggregate_name> with the name of the availability zone host aggregate to add the Compute node to.
   • Replace <host_name> with the name of the Compute node to add to the availability zone.

Procedure

1. To view a list of all the Compute nodes assigned to the host aggregate, enter the following command:

   (overcloud)# openstack aggregate show <aggregate_name>

2. To remove all assigned Compute nodes from the host aggregate, enter the following command for each Compute node:

   (overcloud)# openstack aggregate remove host <aggregate_name> \
     <host_name>

   • Replace <aggregate_name> with the name of the host aggregate to remove the Compute node from.
   • Replace <host_name> with the name of the Compute node to remove from the host aggregate.

3. After you remove all the Compute nodes from the host aggregate, enter the following command to delete the host aggregate:

   (overcloud)# openstack aggregate delete <aggregate_name>

Procedure

1. Open your Compute environment file.
2. To schedule project instances on the project-isolated host aggregate, set the NovaSchedulerLimitTenantsToPlacementAggregate parameter to True in the Compute environment file.

3. Optional: To ensure that only the projects that you assign to a host aggregate can create instances on your cloud, set the NovaSchedulerPlacementAggregateRequiredForTenants parameter to True.

   Note

   NovaSchedulerPlacementAggregateRequiredForTenants is False by default. When this parameter is False, projects that are not assigned to a host aggregate can create instances on any host aggregate.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml \

6. Create the host aggregate.

7. Retrieve the list of project IDs:

   (overcloud)# openstack project list

8. Use the filter_tenant_id<suffix> metadata key to assign projects to the host aggregate:

   (overcloud)# openstack aggregate set \
    --property filter_tenant_id<ID0>=<project_id0> \
    --property filter_tenant_id<ID1>=<project_id1> \
    ...
    --property filter_tenant_id<IDn>=<project_idn> \
    <aggregate_name>

   • Replace <ID0>, <ID1>, and all IDs up to <IDn> with unique values for each project filter that you want to create.
   • Replace <project_id0>, <project_id1>, and all project IDs up to <project_idn> with the ID of each project that you want to assign to the host aggregate.

   • Replace <aggregate_name> with the name of the project-isolated host aggregate.

     For example, use the following syntax to assign projects 78f1, 9d3t, and aa29 to the host aggregate project-isolated-aggregate:

     (overcloud)# openstack aggregate set \
      --property filter_tenant_id0=78f1 \
      --property filter_tenant_id1=9d3t \
      --property filter_tenant_id2=aa29 \
      project-isolated-aggregate

     Tip

     You can create a host aggregate that is available only to a single specific project by omitting the suffix from the filter_tenant_id metadata key:

     (overcloud)# openstack aggregate set \
      --property filter_tenant_id=78f1 \
      single-project-isolated-aggregate

Procedure

1. Open your Compute environment file.

2. Add the filters you want the scheduler to use to the NovaSchedulerDefaultFilters parameter, for example:

   parameter_defaults:
     NovaSchedulerDefaultFilters: AggregateInstanceExtraSpecsFilter,RetryFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter

3. Specify which attribute to use to calculate the weight of each Compute node, for example:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         DEFAULT/scheduler_weight_classes:
           value: nova.scheduler.weights.all_weighers

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Create a flavor that specifies the basic resources to make available to an instance:

   (overcloud)$ openstack flavor create --ram <size-mb> \
    --disk <size-gb> --vcpus <no_reserved_vcpus> <flavor-name>

   • Replace <size-mb> with the size of RAM to allocate to an instance created with this flavor.
   • Replace <size-gb> with the size of root disk to allocate to an instance created with this flavor.
   • Replace <no-vcpus> with the number of vCPUs to reserve for an instance created with this flavor.

   • Replace <flavor-name> with a unique name for your flavor.

     For more information about the arguments, see Flavor arguments.

2. Optional: To specify flavor metadata, set the required properties by using key-value pairs:

   (overcloud)$ openstack flavor set \
    --property <key=value> --property <key=value> ... <flavor-name>

   • Replace <key> with the metadata key of the property you want to allocate to an instance that is created with this flavor. For a list of available metadata keys, see Flavor metadata.
   • Replace <value> with the value of the metadata key you want to allocate to an instance that is created with this flavor.

   • Replace <flavor-name> with the name of your flavor.

     For example, an instance that is launched by using the following flavor has two CPU sockets, each with two CPUs:

     (overcloud)$ openstack flavor set \
      --property hw:cpu_sockets=2 \
      --property hw:cpu_cores=2 processor_topology_flavor

3. Optional: To make the flavor accessible only by a particular project or group of users, set the following attributes:

   (overcloud)$ openstack flavor set --private --project <project-id> <flavor-name>

   • Replace <project-id> with the ID of the project that can use this flavor to create instances.
   • Replace <flavor-name> with the name of your flavor.

Procedure

1. Open your Compute environment file.

2. To always attach a config drive when launching an instance, set the following parameter to True:

   parameter_defaults:
     ComputeExtraConfig:
       nova::compute::force_config_drive: 'true'

3. Optional: To change the format of the config drive from the default value of iso9660 to vfat, add the config_drive_format parameter to your configuration:

   parameter_defaults:
     ComputeExtraConfig:
       nova::compute::force_config_drive: 'true'
       nova::compute::config_drive_format: vfat

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

Verification

1. Create an instance:

   (overcloud)$ openstack server create --flavor m1.tiny \
    --image cirros test-config-drive-instance

2. Log in to the instance.

3. Mount the config drive:

   • If the instance OS uses udev:

     # mkdir -p /mnt/config
     # mount /dev/disk/by-label/config-2 /mnt/config

   • If the instance OS does not use udev, you need to first identify the block device that corresponds to the config drive:

     # blkid -t LABEL="config-2" -odevice
     /dev/vdb
     # mkdir -p /mnt/config
     # mount /dev/vdb /mnt/config

4. Inspect the files in the mounted config drive directory, mnt/config/openstack/{version}/, for your metadata.

Procedure

1. Create the JSON file for the metadata.
2. Open your Compute environment file.

3. Add the path to the JSON file to your environment file:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_jsonfile_path:
           value: <path_to_the_JSON_file>

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

Procedure

1. Open your Compute environment file.

2. Add DynamicJSON to the vendordata provider module:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_providers:
           value: StaticJSON,DynamicJSON

3. Specify the REST services to contact to generate the metadata. You can specify as many target REST services as required, for example:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_providers:
           value: StaticJSON,DynamicJSON
         api/vendordata_dynamic_targets:
           value: target1@http://127.0.0.1:125
         api/vendordata_dynamic_targets:
           value: target2@http://127.0.0.1:126

   The Compute service generates the JSON file, vendordata2.json, to contain the metadata retrieved from the configured target services, and stores it in the config drive directory.

   Note

   Do not use the same name for a target service more than once.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Open your Compute environment file.

2. Optional: Add the following configuration to your Compute environment file to specify the maximum number of memory-encrypted instances the SEV-capable Compute node can host concurrently:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         libvirt/num_memory_encrypted_guests:
           value: 15

   Note

   If not set, libvirt/num_memory_encrypted_guests defaults to none, which means the SEV-capable Compute node does not impose a limit on the number of memory-encrypted instances that can be hosted concurrently. Instead, the hardware determines the maximum number of memory-encrypted instances the SEV-capable Compute node can host concurrently, which might cause some memory-encrypted instances to fail to launch.

3. Optional: To specify that all x86_64 images use the q35 machine type by default, add the NovaHWMachineType parameter to the Compute environment file, and set it to x86_64=q35.

   This configuration removes the need to set the hw_machine_type property to q35 on every SEV-enabled instance image.

4. To prevent memory overcommit, set the NovaRAMAllocationRatio parameter to 1.0 in the Compute environment file.
5. To ensure that the SEV-capable Compute nodes reserve enough memory for host-level services to function, add 16MB for each potential SEV instance (the maximum number of concurrent SEV instances), to your value for NovaReservedHostMemory in the Compute environment file.

6. Add the following configuration to your Compute environment file to schedule memory-encrypted instances on a SEV-capable Compute node aggregate:

   parameter_defaults:
     ControllerExtraConfig:
       nova::config::nova_config:
         scheduler/enable_isolated_aggregate_filtering:
           value: 'True'

7. Save the updates to your Compute environment file.

8. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

9. Create a host aggregate for SEV Compute nodes to ensure that instances that do not request memory encryption are not created on SEV-capable hosts:

   (undercloud)$ source ~/overcloudrc
   (overcloud)$ openstack aggregate create sev_agg
   (overcloud)$ openstack aggregate add host sev_agg hostA
   (overcloud)$ openstack aggregate add host sev_agg hostB
   (overcloud)$ openstack --os-compute-api-version 2.53 aggregate set --property trait:HW_CPU_X86_AMD_SEV=required sev_agg

Procedure

1. Create a new image for SEV:

   (overcloud)$ openstack image create ...  \
    --property hw_firmware_type=uefi sev-image

   Note

   If you use an existing image, the image must have the hw_firmware_type property set to uefi.

2. Optional: Add the property hw_mem_encryption=True to the image to enable SEV memory encryption on the image:

   (overcloud)$ openstack image set  \
    --property hw_mem_encryption=True sev-image

   Tip

   You can enable SEV memory encryption on the flavor. For more information, see Creating a SEV-enabled flavor for instances.

3. Optional: Set the machine type to q35, if not already set in the Compute node configuration:

   (overcloud)$ openstack image set  \
    --property hw_machine_type=q35 sev-image

4. Optional: To schedule memory-encrypted instances on a SEV-capable host aggregate, add the following trait to the image extra specs:

   (overcloud)$ openstack image set  \
    --property trait:HW_CPU_X86_AMD_SEV=required sev-image

   Tip

   You can also specify this trait on the flavor. For more information, see Creating a SEV-enabled flavor for instances.

Procedure

1. Create a flavor for SEV:

   (overcloud)$ openstack flavor create --vcpus 1 --ram 512 --disk 2  \
    --property hw:mem_encryption=True m1.small-sev

2. To schedule memory-encrypted instances on a SEV-capable host aggregate, add the following trait to the flavor extra specs:

   (overcloud)$ openstack flavor set  \
    --property trait:HW_CPU_X86_AMD_SEV=required m1.small-sev

1. Build a custom GPU-enabled overcloud image.
2. Prepare the GPU role, profile, and flavor for designating Compute nodes for vGPU.
3. Configure the Compute node for vGPU.
4. Deploy the overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the overcloudrc credential file:

   $ source ~/overcloudrc

3. Create an instance with the hardware and software profile that your vGPU instances require:

   (overcloud)$ openstack server create --flavor <flavor> \
    --image <image> temp_vgpu_instance

   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require. For information about creating a vGPU flavor, see Creating a vGPU flavor for instances.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information about downloading RHEL cloud images, see Image service.
4. Log in to the instance as a cloud-user. For more information, see Log in to an Instance.
5. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

6. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances. For more information about supported drivers, see Image metadata.

7. Create an image snapshot of the instance:

   (overcloud)$ openstack server image create \
    --name vgpu_image temp_vgpu_instance

8. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor, for example:

   (overcloud)$ openstack flavor create --vcpus 6 \
    --ram 8192 --disk 100 m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

2. Assign a vGPU resource to the flavor that you created. You can assign only one vGPU for each instance.

   (overcloud)$ openstack flavor set m1.small-gpu \
    --property "resources:VGPU=1"
   
   (overcloud)$ openstack flavor show m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | access_project_ids         | None                                 |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 | resources:VGPU='1'                   |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

Procedure

1. Create an instance using a GPU flavor and image, for example:

   (overcloud)$ openstack server create --flavor m1.small-gpu \
    --image vgpu_image --security-group web --nic net-id=internal0 \
    --key-name lambda vgpu-instance

2. Log in to the instance as a cloud-user. For more information, see Log in to an Instance.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

Procedure

1. To determine the vendor ID and product ID for each passthrough device type, enter the following command on the physical server that has the PCI cards:

   # lspci -nn | grep -i <gpu_name>

   For example, to determine the vendor and product ID for an NVIDIA GPU, enter the following command:

   # lspci -nn | grep -i nvidia
   3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
   d8:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1db4] (rev a1)

2. To determine if each PCI device has Single Root I/O Virtualization (SR-IOV) capabilities, enter the following command on the physical server that has the PCI cards:

   # lspci -v -s 3b:00.0
   3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
             ...
             Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
             ...

3. To configure the Controller node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_controller.yaml.

4. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthru_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['RetryFilter','AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

5. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthru_controller.yaml:

   • If the PCI device has SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     For more information on configuring the device_type field, see PCI passthrough device type field.

     Note

     If the nova-api service is running in a role other than the Controller, then replace ControllerExtraConfig with the user role, in the format <Role>ExtraConfig.

6. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_compute.yaml.

7. To specify the available PCIs for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   parameter_defaults:
     NovaPCIPassthrough:
       - vendor_id: "10de"
         product_id: "1eb8"

8. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   • If the PCI device has SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     Note

     The Compute node aliases must be identical to the aliases on the Controller node.

9. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthru_compute.yaml:

   parameter_defaults:
     ...
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

10. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
      -e [your environment files] \
      -e /home/stack/templates/pci_passthru_controller.yaml \
      -e /home/stack/templates/pci_passthru_compute.yaml

11. Configure a flavor to request the PCI devices. The following example requests two devices, each with a vendor ID of 10de and a product ID of 13f2:

    # openstack flavor set m1.large \
     --property "pci_passthrough:alias"="t4:2"

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor m1.large \
    --image <custom_gpu> --wait test-pci

   Replace <custom_gpu> with the name of your custom instance image that has the required GPU driver installed.

2. Log in to the instance as a cloud user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

4. To check the NVIDIA System Management Interface status, enter the following command from the instance:

   $ nvidia-smi

   Example output:

   -----------------------------------------------------------------------------
   | NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
   |---------------------------------------------------------------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |===========================================================================|
   |   0  Tesla T4            Off  | 00000000:01:00.0 Off |                    0 |
   | N/A   43C    P0    20W /  70W |      0MiB / 15109MiB |      0%      Default |
   ---------------------------------------------------------------------------
   
   -----------------------------------------------------------------------------
   | Processes:                                                       GPU Memory |
   |  GPU       PID   Type   Process name                             Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   -----------------------------------------------------------------------------

Procedure

1. To build the real-time overcloud image, you must enable the rhel-8-for-x86_64-nfv-rpms repository for RT-KVM. To check which packages will be installed from the repository, enter the following command:

   $ dnf repo-pkgs rhel-8-for-x86_64-nfv-rpms list
   Loaded plugins: product-id, search-disabled-repos, subscription-manager
   Available Packages
   kernel-rt.x86_64                   4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-debug.x86_64             4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-debug-devel.x86_64       4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-debug-kvm.x86_64         4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-devel.x86_64             4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-doc.noarch               4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   kernel-rt-kvm.x86_64               4.18.0-80.7.1.rt9.153.el8_0               rhel-8-for-x86_64-nfv-rpms
   [ output omitted…]

2. To build the overcloud image for Real-time Compute nodes, install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud)$ sudo dnf install libguestfs-tools

   Important

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

3. Extract the images:

   (undercloud)$ tar -xf /usr/share/rhosp-director-images/overcloud-full.tar
   (undercloud)$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent.tar

4. Copy the default image:

   (undercloud)$ cp overcloud-full.qcow2 overcloud-realtime-compute.qcow2

5. Register the image and configure the required subscriptions:

   (undercloud)$ virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager register --username=<username> --password=<password>'
   [  0.0] Examining the guest ...
   [ 10.0] Setting a random seed
   [ 10.0] Running: subscription-manager register --username=<username> --password=<password>
   [ 24.0] Finishing off

   Replace the username and password values with your Red Hat customer account details.

   For general information about building a Real-time overcloud image, see the knowledgebase article Modifying the Red Hat Enterprise Linux OpenStack Platform Overcloud Image with virt-customize.

6. Find the SKU of the Red Hat OpenStack Platform for Real Time subscription. The SKU might be located on a system that is already registered to the Red Hat Subscription Manager with the same account and credentials:

   $ sudo subscription-manager list

7. Attach the Red Hat OpenStack Platform for Real Time subscription to the image:

   (undercloud)$  virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager attach --pool [subscription-pool]'

8. Create a script to configure rt on the image:

   (undercloud)$ cat rt.sh
     #!/bin/bash
   
     set -eux
   
     subscription-manager repos --enable=[REPO_ID]
     dnf -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     dnf -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
   
     # END OF SCRIPT

9. Run the script to configure the real-time image:

   (undercloud)$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

10. Re-label SELinux:

    (undercloud)$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

11. Extract vmlinuz and initrd. For example:

    (undercloud)$ mkdir image
    (undercloud)$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud)$ cp image/boot/vmlinuz-4.18.0-80.7.1.rt9.153.el8_0.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud)$ cp image/boot/initramfs-4.18.0-80.7.1.rt9.153.el8_0.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud)$ guestunmount image

    Note

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

12. Upload the image:

    (undercloud)$ openstack overcloud image upload \
     --update-existing --os-image-name
     overcloud-realtime-compute.qcow2

    You now have a real-time image you can use with the ComputeRealTime composable role on select Compute nodes.

13. To reduce latency on your Real-time Compute nodes, you must modify the BIOS settings in the Compute nodes. You should disable all options for the following components in your Compute node BIOS settings:

    • Power Management
    • Hyper-Threading
    • CPU sleep states

    • Logical processors

      See Setting BIOS parameters for descriptions of these settings and the impact of disabling them. See your hardware manufacturer documentation for complete details on how to change BIOS settings.

Procedure

1. Based on the /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml file, create a compute-real-time.yaml environment file that sets the parameters for the ComputeRealTime role.

   cp /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml /home/stack/templates/compute-real-time.yaml

   The file must include values for the following parameters:

   • IsolCpusList and NovaComputeCpuDedicatedSet: List of isolated CPU cores and virtual CPU pins to reserve for real-time workloads. This value depends on the CPU hardware of your real-time Compute nodes.
   • NovaComputeCpuSharedSet: List of host CPUs to reserve for emulator threads.
   • KernelArgs: Arguments to pass to the kernel of the Real-time Compute nodes. For example, you can use default_hugepagesz=1G hugepagesz=1G hugepages=<number_of_1G_pages_to_reserve> hugepagesz=2M hugepages=<number_of_2M_pages> to define the memory requirements of guests that have huge pages with multiple sizes. In this example, the default size is 1GB but you can also reserve 2M huge pages.

2. Add the ComputeRealTime role to your roles data file and regenerate the file. For example:

   $ openstack overcloud roles generate -o /home/stack/templates/rt_roles_data.yaml Controller Compute ComputeRealTime

   This command generates a ComputeRealTime role with contents similar to the following example, and also sets the ImageDefault option to overcloud-realtime-compute.

   - name: ComputeRealTime
     description: |
       Compute role that is optimized for real-time behaviour. When using this role
       it is mandatory that an overcloud-realtime-compute image is available and
       the role specific parameters IsolCpusList, NovaComputeCpuDedicatedSet and
       NovaComputeCpuSharedSet are set accordingly to the hardware of the real-time compute nodes.
     CountDefault: 1
     networks:
       InternalApi:
         subnet: internal_api_subnet
       Tenant:
         subnet: tenant_subnet
       Storage:
         subnet: storage_subnet
     HostnameFormatDefault: '%stackname%-computerealtime-%index%'
     ImageDefault: overcloud-realtime-compute
     RoleParametersDefault:
       TunedProfileName: "realtime-virtual-host"
       KernelArgs: ""      # these must be set in an environment file
       IsolCpusList: ""    # or similar according to the hardware
       NovaComputeCpuDedicatedSet: ""  # of real-time nodes
       NovaComputeCpuSharedSet: ""     #
       NovaLibvirtMemStatsPeriodSeconds: 0
     ServicesDefault:
       - OS::TripleO::Services::Aide
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::BootParams
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::ComputeCeilometerAgent
       - OS::TripleO::Services::ComputeNeutronCorePlugin
       - OS::TripleO::Services::ComputeNeutronL3Agent
       - OS::TripleO::Services::ComputeNeutronMetadataAgent
       - OS::TripleO::Services::ComputeNeutronOvsAgent
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::IpaClient
       - OS::TripleO::Services::Ipsec
       - OS::TripleO::Services::Iscsid
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::LoginDefs
       - OS::TripleO::Services::MetricsQdr
       - OS::TripleO::Services::MySQLClient
       - OS::TripleO::Services::NeutronBgpVpnBagpipe
       - OS::TripleO::Services::NeutronLinuxbridgeAgent
       - OS::TripleO::Services::NeutronVppAgent
       - OS::TripleO::Services::NovaCompute
       - OS::TripleO::Services::NovaLibvirt
       - OS::TripleO::Services::NovaLibvirtGuests
       - OS::TripleO::Services::NovaMigrationTarget
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::OpenDaylightOvs
       - OS::TripleO::Services::Podman
       - OS::TripleO::Services::Rhsm
       - OS::TripleO::Services::RsyslogSidecar
       - OS::TripleO::Services::Securetty
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::SkydiveAgent
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Sshd
       - OS::TripleO::Services::Timesync
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Vpp
       - OS::TripleO::Services::OVNController
       - OS::TripleO::Services::OVNMetadataAgent

   For general information about custom roles and about the roles-data.yaml, see Roles.

3. Create the compute-realtime flavor to tag nodes that you want to designate for real-time workloads. For example:

   $ source ~/stackrc
   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-realtime
   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute-realtime" compute-realtime

4. Tag each node that you want to designate for real-time workloads with the compute-realtime profile.

   $ openstack baremetal node set --property capabilities='profile:compute-realtime,boot_option:local' <NODE UUID>

5. Map the ComputeRealTime role to the compute-realtime flavor by creating an environment file with the following content:

   parameter_defaults:
     OvercloudComputeRealTimeFlavor: compute-realtime

6. Add your environment files and the new roles file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -r /home/stack/templates/rt~/my_roles_data.yaml \
    -e home/stack/templates/compute-real-time.yaml

Procedure

1. Create the compute-real-time.yaml file with the following parameters:

   parameter_defaults:
     ComputeRealTimeParameters:
       IsolCpusList: "1"
       NovaComputeCpuDedicatedSet: "1"
       NovaComputeCpuSharedSet: "0"
       KernelArgs: "default_hugepagesz=1G hugepagesz=1G hugepages=16"

2. Create a new rt_roles_data.yaml file with the ComputeRealTime role:

   $ openstack overcloud roles generate \
    -o ~/rt_roles_data.yaml Controller ComputeRealTime

3. Deploy the overcloud, adding both your new real-time roles data file and your real-time environment file to the stack along with your other environment files:

   (undercloud)$ openstack overcloud deploy --templates \
     -r /home/stack/rt_roles_data.yaml
     -e [your environment files]
     -e /home/stack/templates/compute-real-time.yaml

   This command deploys one Controller node and one Real-time Compute node.

4. Log into the Real-time Compute node and check the following parameters. Replace <...> with the values of the relevant parameters from the compute-real-time.yaml.

   [root@overcloud-computerealtime-0 ~]# uname -a
   Linux overcloud-computerealtime-0 4.18.0-80.7.1.rt9.153.el8_0.x86_64 #1 SMP PREEMPT RT Wed Dec 13 13:37:53 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
   [root@overcloud-computerealtime-0 ~]# cat /proc/cmdline
   BOOT_IMAGE=/boot/vmlinuz-4.18.0-80.7.1.rt9.153.el8_0.x86_64 root=UUID=45ae42d0-58e7-44fe-b5b1-993fe97b760f ro console=tty0 crashkernel=auto console=ttyS0,115200 default_hugepagesz=1G hugepagesz=1G hugepages=16
   [root@overcloud-computerealtime-0 ~]# tuned-adm active
   Current active profile: realtime-virtual-host
   [root@overcloud-computerealtime-0 ~]# grep ^isolated_cores /etc/tuned/realtime-virtual-host-variables.conf
   isolated_cores=<IsolCpusList>
   [root@overcloud-computerealtime-0 ~]# cat /usr/lib/tuned/realtime-virtual-host/lapic_timer_adv_ns
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# cat /sys/module/kvm/parameters/lapic_timer_advance_ns
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_dedicated_set
   <NovaComputeCpuDedicatedSet>
   [root@overcloud-computerealtime-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_shared_set
   <NovaComputeCpuSharedSet>

Procedure

1. Launch the real-time instance:

   # openstack server create  --image <rhel> \
    --flavor r1.small --nic net-id=<dpdk-net> test-rt

2. Optional: Verify that the instance uses the assigned emulator threads:

   # virsh dumpxml <instance-id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>

1. Extract parameter information from the default first cell in the basic overcloud. This cell becomes the global Controller after you redeploy the overcloud.
2. Configure a custom role and flavor for each cell.
3. Create environment files to configure cell-specific parameters.
4. Deploy the overcloud with the new cell stack.

Procedure: Extract parameter information from the overcloud

1. Create a new directory for the new cell, and export the directory to the DIR environment variable. The example used throughout this procedure is cell1:

   $ source ~/stackrc
   (undercloud)$ mkdir cell1
   (undercloud)$ export DIR=cell1

2. Export the default cell configuration and password information from the overcloud to a new environment file for the cell:

   (undercloud)$ openstack overcloud cell export \
    --output-file cell1/cell1-ctrl-input.yaml cell1

   Note

   If the environment file already exists, enter the command with the --force-overwrite or -f option.

   This command exports the EndpointMap, HostsEntry, AllNodesConfig, GlobalConfig parameters, and the password information, to the new environment file for the cell, cell1/cell1-ctrl-input.yaml.

Procedure: Configure a custom role for a cell

1. Generate a new roles data file named cell_roles_data.yaml that includes the Compute and CellController roles:

   (undercloud)$ openstack overcloud roles generate \
     --roles-path /usr/share/openstack-tripleo-heat-templates/roles \
     -o $DIR/cell_roles_data.yaml Compute CellController

2. Optional: To divide your network between the global Controller and the cells, configure network access in the roles file that you created using segmented network role parameters:

   name: Compute
    description: |
      Basic Compute Node role
    CountDefault: 1
    # Create external Neutron bridge (unset if using ML2/OVS without DVR)
    tags:
      - external_bridge
    networks:
      InternalApi:
        subnet: internal_api_cell1
      Tenant:
        subnet: tenant_subnet
      Storage:
        subnet: storage_cell1
   ...
   - name: CellController
      description: |
        CellController role for the nova cell_v2 controller services
      CountDefault: 1
      tags:
        - primary
        - controller
      networks:
        External:
          subnet: external_cell1
        InternalApi:
          subnet: internal_api_cell1
        Storage:
          subnet: storage_cell1
        StorageMgmt:
          subnet: storage_mgmt_cell1
        Tenant:
          subnet: tenant_subnet

Procedure: Configure the cell flavor and tag nodes to a cell

1. Create the cellcontroller flavor to use to tag nodes that you want to allocate to the cell. For example:

   (undercloud)$ openstack flavor create --id auto --ram 4096 \
     --disk 40 --vcpus 1 cellcontroller
   (undercloud)$ openstack flavor set --property "cpu_arch"="x86_64" \
    --property "capabilities:boot_option"="local" \
    --property "capabilities:profile"="cellcontroller" \
    --property "resources:CUSTOM_BAREMETAL=1" \
    --property "resources:DISK_GB=0" \
    --property "resources:MEMORY_MB=0" \
    --property "resources:VCPU=0" \
    cellcontroller

2. Tag each node that you want to assign to the cell with the cellcontroller profile.

   (undercloud)$ openstack baremetal node set --property \
    capabilities='profile:cellcontroller,boot_option:local' <node_uuid>

   Replace <node_uuid> with the ID of the Compute node that you want to assign to the cell.

Procedure: Create environment files with cell parameters

1. Create a new environment file in the cell directory cell for cell-specific parameters, for example, /cell1/cell1.yaml.

2. Add the following parameters, updating the parameter values for your deployment:

   resource_registry:
     # since the same networks are used in this example, the
     # creation of the different networks is omitted
     OS::TripleO::Network::External: OS::Heat::None
     OS::TripleO::Network::InternalApi: OS::Heat::None
     OS::TripleO::Network::Storage: OS::Heat::None
     OS::TripleO::Network::StorageMgmt: OS::Heat::None
     OS::TripleO::Network::Tenant: OS::Heat::None
     OS::TripleO::Network::Management: OS::Heat::None
     OS::TripleO::Network::Ports::OVNDBsVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
     OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
   
   parameter_defaults:
     # CELL Parameter to reflect that this is an additional CELL
     NovaAdditionalCell: True
   
     # mapping of the CellController flavor to the CellController role
     CellControllerFlavor: cellcontroller
   
     # The DNS names for the VIPs for the cell
     CloudName: cell1.ooo.test
     CloudNameInternal: cell1.internalapi.ooo.test
     CloudNameStorage: cell1.storage.ooo.test
     CloudNameStorageManagement: cell1.storagemgmt.ooo.test
     CloudNameCtlplane: cell1.ctlplane.ooo.test
   
     # Flavors used for the cell controller and computes
     OvercloudCellControllerFlavor: cellcontroller
     OvercloudComputeFlavor: compute
   
     # Number of controllers/computes in the cell
     CellControllerCount: 1
     ComputeCount: 1
   
     # Compute node name (must be unique across all cells)
     ComputeHostnameFormat: 'cell1-compute-%index%'
   
     # default gateway
     ControlPlaneStaticRoutes:
       - ip_netmask: 0.0.0.0/0
         next_hop: 192.168.24.1
         default: true
     DnsServers:
       - x.x.x.x

3. Optional: To allocate a network resource to the cell and register cells to the network, add the following parameter to your environment file:

   resource_registry:
     OS::TripleO::CellController::Net::SoftwareConfig: single-nic-vlans/controller.yaml
     OS::TripleO::Compute::Net::SoftwareConfig: single-nic-vlans/compute.yaml

4. Optional: If you divide your network between the global Controller and the cells and want to run the Compute metadata API in each cell instead of in the global Controller, add the following parameter:

   parameter_defaults:
     NovaLocalMetadataPerCell: True

   Note

   • The parameters in this file restrict the overcloud to use a single network for all cells.
   • The Compute host names must be unique across all cells.

5. Make a copy of the network_data.yaml file and rename the copy to include the cell name, for example:

   (undercloud)$ cp /usr/share/openstack-tripleo-heat-templates/network_data.yaml cell1/network_data-ctrl.yaml

6. Retrieve the UUIDs of the network components that you want to reuse for the cells:

   (undercloud)$ openstack network show <network>
   (undercloud)$ openstack subnet show <subnet>
   (undercloud)$ openstack network segment show <network_segment>
   (undercloud)$ openstack port show <port>

   • Replace <network> with the name of the network whose UUID you want to retrieve.
   • Replace <subnet> with the name of the subnet whose UUID you want to retrieve.
   • Replace <network_segment> with the name of the segment whose UUID you want to retrieve.
   • Replace <port> with the name of the port whose UUID you want to retrieve.

7. To reuse the network components for the cells, add the following configuration to your new cell network_data.yaml file:

   external_resource_network_id: <network_uuid>
   external_resource_subnet_id: <subnet_uuid>
   external_resource_segment_id: <segment_uuid>
   external_resource_vip_id: <port_uuid>

   • Replace <network_uuid> with the UUID of the network you retrieved in the previous step.
   • Replace <subnet_uuid> with the UUID of the subnet you retrieved in the previous step.
   • Replace <segment_uuid> with the UUID of the segment you retrieved in the previous step.
   • Replace <port_uuid> with the UUID of the port you retrieved in the previous step.

8. Optional: To configure segmented networks for the global Controller cell and the Compute cells, create an environment file and add the routing information and virtual IP address (VIP) information for the cell. For example, you can add the following configuration for cell1 to the file cell1_routes.yaml:

   parameter_defaults:
     InternalApiInterfaceRoutes:
       - destination: 172.17.2.0/24
         nexthop: 172.16.2.254
     StorageInterfaceRoutes:
       - destination: 172.17.1.0/24
         nexthop: 172.16.1.254
     StorageMgmtInterfaceRoutes:
       - destination: 172.17.3.0/24
         nexthop: 172.16.3.254
   
   parameter_defaults:
     VipSubnetMap:
       InternalApi: internal_api_cell1
       Storage: storage_cell1
       StorageMgmt: storage_mgmt_cell1
       External: external_cell1

9. Add the environment files to the stack with your other environment files and deploy the overcloud, for example:

   (undercloud)$ openstack overcloud deploy --templates \
    --stack cell1 \
    -e [your environment files] \
    -r $HOME/$DIR/cell_roles_data.yaml \
    -e $HOME/$DIR/cell1_routes.yaml \
    -e $HOME/$DIR/network_data-ctrl.yaml \
    -e $HOME/$DIR/cell1-ctrl-input.yaml \
    -e $HOME/$DIR/cell1.yaml

   Note

   If you deploy Compute cells in Edge sites, enter the overcloud deploy command in each site with the environment files and configuration for each Compute cell in that site.

Procedure

1. Get the IP addresses of the control plane and cell controller.

   $ CTRL_IP=$(openstack server list -f value -c Networks --name overcloud-controller-0 | sed 's/ctlplane=//')
   $ CELL_CTRL_IP=$(openstack server list -f value -c Networks --name cellcontroller-0 | sed 's/ctlplane=//')

2. Add the cell information to all Controller nodes. This information is used to connect to the cell endpoint from the undercloud.

   (undercloud)$ CELL_INTERNALAPI_INFO=$(ssh heat-admin@${CELL_CTRL_IP} egrep \
    cellcontrol.*\.internalapi /etc/hosts)
   (undercloud)$ ansible -i /usr/bin/tripleo-ansible-inventory Controller -b \
    -m lineinfile -a "dest=/etc/hosts line=\"$CELL_INTERNALAPI_INFO\""

3. Get the message queue endpoint for the controller cell from the transport_url parameter, and the database connection for the controller cell from the database.connection parameter:

   (undercloud)$ CELL_TRANSPORT_URL=$(ssh heat-admin@${CELL_CTRL_IP} sudo \
    crudini --get /var/lib/config-data/nova/etc/nova/nova.conf DEFAULT transport_url)
   (undercloud)$ CELL_MYSQL_VIP=$(ssh heat-admin@${CELL_CTRL_IP} sudo \
    crudini --get /var/lib/config-data/nova/etc/nova/nova.conf database connection \
    | perl -nle'/(\d+\.\d+\.\d+\.\d+)/ && print $1')

4. Log in to one of the global Controller nodes to create the cell:

   $ export CONTAINERCLI='podman'
   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI}  \
    exec -i -u root nova_api \
    nova-manage cell_v2 create_cell --name computecell1 \
    --database_connection "{scheme}://{username}:{password}@$CELL_MYSQL_VIP/nova?{query}" \
    --transport-url "$CELL_TRANSPORT_URL"

5. Check that the cell is created and appears in the cell list.

   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI}  \
    exec -i -u root nova_api \
    nova-manage cell_v2 list_cells --verbose

6. Restart the Compute services on the Controller nodes.

   $ ansible -i /usr/bin/tripleo-ansible-inventory Controller -b -a \
   "systemctl restart tripleo_nova_api tripleo_nova_conductor tripleo_nova_scheduler"

7. Check that the cell controller services are provisioned.

   (overcloud)$ nova service-list

Procedure

1. Log into one of the Controller nodes.

2. Get the IP address of the control plane for the cell and enter the host discovery command to expose and assign Compute hosts to the cell.

   $ CTRL=overcloud-controller-0
   $ CTRL_IP=$(openstack server list -f value -c Networks --name $CTRL | sed 's/ctlplane=//')
   
   $ export CONTAINERCLI='podman'
   
   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI} exec -i -u root nova_api \
     nova-manage cell_v2 discover_hosts --by-service --verbose

3. Verify that the Compute hosts were assigned to the cell.

   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI} exec -i -u root nova_api \
     nova-manage cell_v2 list_hosts

Procedure

1. Delete all instances from the Compute nodes in the cell.

   Note

   Migrating instances between cells is not supported. You must delete the instances and re-create them in another cell.

2. On one of the global Controllers, delete all Compute nodes from the cell.

   $ CTRL=overcloud-controller-0
   $ CTRL_IP=$(openstack server list -f value -c Networks --name $CTRL | sed 's/ctlplane=//')
   
   $ export CONTAINERCLI='podman'
   
   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI}  \
    exec -i -u root nova_api \
    nova-manage cell_v2 list_hosts
   
   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI}  \
    exec -i -u root nova_api \
    nova-manage cell_v2 delete_host --cell_uuid <uuid> --host <compute>

3. Delete the resource providers for the cell from the Placement service, to ensure that the host name is available in case you want to add Compute nodes with the same host name to another cell later. For example:

   (undercloud)$ source ~/overcloudrc
   
   (overcloud)$ openstack resource provider list
   +--------------------------------------+---------------------------------------+------------+
   | uuid                                 | name                                  | generation |
   +--------------------------------------+---------------------------------------+------------+
   | 9cd04a8b-5e6c-428e-a643-397c9bebcc16 | computecell1-novacompute-0.site1.test |         11 |
   +--------------------------------------+---------------------------------------+------------+
   
   (overcloud)$ openstack resource provider  \
    delete 9cd04a8b-5e6c-428e-a643-397c9bebcc16

Procedure

1. On one of the global Controllers, delete the cell.

   $ CTRL=overcloud-controller-0
   $ CTRL_IP=$(openstack server list -f value -c Networks --name $CTRL | sed 's/ctlplane=//')
   
   $ export CONTAINERCLI='podman'
   
   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI}  \
    exec -i -u root nova_api \
    nova-manage cell_v2 list_cells
   
   $ ssh heat-admin@${CTRL_IP} sudo ${CONTAINERCLI}  \
    exec -i -u root nova_api \
    nova-manage cell_v2 delete_cell --cell_uuid <uuid>

2. Delete the cell stack from the overcloud.

   $ openstack stack delete <stack name> --wait --yes && openstack  \
    overcloud plan delete <STACK_NAME>

   Note

   If you deployed separate cell stacks for a Controller and Compute cell, delete the Compute cell stack first and then the Controller cell stack.