Procedure

1. Copy the /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml file to your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml /home/stack/templates/enable-swap.yaml

2. Configure the swap size by adding the following parameters to your enable-swap.yaml file:

   parameter_defaults:
     swap_size_megabytes: <swap size in MB>
     swap_path: <full path to location of swap, default: /swap>

3. To apply this configuration, add the enable_swap.yaml environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud) $ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/enable-swap.yaml \

1. Cache the Image service’s backing image as the libvirt base.
2. Convert the base image to the raw format (if configured).
3. Resize the base image to match the VM’s flavor specifications.
4. Use the base image to create the libvirt instance disk.

Procedure

1. Create an environment file to configure the service tokens, such as service_tokens.yaml.

2. Add the following configuration parameters to the service token environment file:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         service_user/send_service_user_token:
           value: true
         service_user/username:
           value: nova
         service_user/auth_strategy:
           value: keystone
         service_user/auth_type:
           value: password
         service_user/password:
           value: "%{hiera('nova::placement::password')}"
         service_user/auth_url:
           value: "%{hiera('nova::placement::auth_url')}"
         service_user/user_domain_name:
           value: "Default"
         service_user/project_name:
           value: "%{hiera('nova::placement::project_name')}"
         service_user/project_default_name:
           value: "Default"
   
     ControllerExtraConfig:
       nova::config::nova_config:
         keystone_authtoken/service_token_roles_required:
           value: true
         keystone_authtoken/service_token_roles:
           value: admin
         service_user/send_service_user_token:
           value: true
         service_user/username:
           value: nova
         service_user/auth_strategy:
           value: keystone
         service_user/auth_type:
           value: password
         service_user/password:
           value: "%{hiera('nova::keystone::authtoken::password')}"
         service_user/auth_url:
           value: "%{hiera('nova::keystone::authtoken::auth_url')}"
         service_user/user_domain_name:
           value: "%{hiera('nova::keystone::authtoken::user_domain_name')}"
         service_user/project_name:
           value: "%{hiera('nova::keystone::authtoken::project_name')}"
         service_user/project_domain_name:
           value: "%{hiera('nova::keystone::authtoken::project_domain_name')}"
   
       cinder::config::cinder_config:
         keystone_authtoken/service_token_roles_required:
           value: true
         keystone_authtoken/service_token_roles:
           value: admin
         service_user/send_service_user_token:
           value: true
         service_user/username:
           value: cinder
         service_user/auth_strategy:
           value: keystone
         service_user/auth_type:
           value: password
         service_user/password:
           value: "%{hiera('cinder::keystone::authtoken::password')}"
         service_user/auth_url:
           value: "%{hiera('cinder::keystone::authtoken::auth_url')}"
         service_user/user_domain_name:
           value: "%{hiera('cinder::keystone::authtoken::user_domain_name')}"
         service_user/project_name:
           value: "%{hiera('cinder::keystone::authtoken::project_name')}"
         service_user/project_domain_name:
           value: "%{hiera('cinder::keystone::authtoken::project_domain_name')}"
   
     BlockStorageExtraConfig:
       cinder::config::cinder_config:
         keystone_authtoken/service_token_roles_required:
           value: true
         keystone_authtoken/service_token_roles:
           value: admin
         service_user/send_service_user_token:
           value: true
         service_user/username:
           value: cinder
         service_user/auth_strategy:
           value: keystone
         service_user/auth_type:
           value: password
         service_user/password:
           value: "%{hiera('cinder::keystone::authtoken::password')}"
         service_user/auth_url:
           value: "%{hiera('cinder::keystone::authtoken::auth_url')}"
         service_user/user_domain_name:
           value: "%{hiera('cinder::keystone::authtoken::user_domain_name')}"
         service_user/project_name:
           value: "%{hiera('cinder::keystone::authtoken::project_name')}"
         service_user/project_domain_name:
           value: "%{hiera('cinder::keystone::authtoken::project_domain_name')}"

3. Add the service token environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/service_tokens.yaml \

Procedure

1. Identify the source Compute node host name and the destination Compute node host name:

   (undercloud)$ source ~/overcloudrc
   (overcloud)$ openstack compute service list

2. List the instances on the source Compute node and locate the ID of the instance or instances that you want to migrate:

   (overcloud)$ openstack server list --host <source> --all-projects

   Replace <source> with the name or ID of the source Compute node.

3. Optional: To shut down the source Compute node for maintenance, disable the source Compute node from the undercloud to ensure that the scheduler does not attempt to assign new instances to the source Compute node during maintenance:

   (overcloud)$ source ~/stackrc
   (undercloud)$ openstack compute service set <source> nova-compute --disable

   Replace <source> with the name or ID of the source Compute node.

Procedure

1. If the destination Compute node for NUMA, CPU-pinned or DPDK instances is not disabled, disable it to prevent the scheduler from assigning instances to the node:

   (overcloud)$ openstack compute service set <dest> nova-compute --disable

   Replace <dest> with the name or ID of the destination Compute node.

2. Ensure that the destination Compute node has no instances, except for instances that you previously migrated from the source Compute node when you migrated multiple DPDK or NUMA instances:

   (overcloud)$ openstack server list --host <dest> --all-projects

   Replace <dest> with the name or ID of the destination Compute node.

3. Ensure that the destination Compute node has sufficient resources to run the NUMA, CPU-pinned or DPDK instance:

   (overcloud)$ openstack host show <dest>
   $ ssh <dest>
   $ numactl --hardware
   $ exit

   Replace <dest> with the name or ID of the destination Compute node.

4. To discover NUMA information about the source or destination Compute nodes, run the following commands:

   $ ssh root@overcloud-compute-n
   # lscpu && lscpu | grep NUMA
   # virsh nodeinfo
   # virsh capabilities
   # exit

   Use ssh to connect to overcloud-compute-n where overcloud-compute-n is the source or destination Compute node.

5. If you do not know if an instance uses NUMA, check the flavor of the instance:

   (overcloud)$ openstack server list -c Name -c Flavor --name <vm>
   (overcloud)$ openstack flavor show <flavor>

   • Replace <vm> with the name or ID of the instance.

   • Replace <flavor> with the name or ID of the flavor.

     • If the properties field includes hw:mem_page_size with a value other than any, such as 2MB, 2048 or 1GB, the instance has a NUMA topology.
     • If the properties field includes aggregate_instance_extra_specs:pinned='true', the instance uses CPU pinning.
     • If the properties field includes hw:numa_nodes, the OpenStack Compute (nova) service restricts the instance to a specific NUMA node.

6. For each instance that uses NUMA, you can retrieve information about the NUMA topology from the underlying Compute node so that you can verify that the NUMA topology on the destination Compute node reflects the NUMA topology of the source Compute node after migration is complete. You can use the following commands to perform this check:

   • To view details about NUMA and CPU pinning, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh vcpuinfo <vm>

     Replace <vm> with the name of the instance.

   • To view details about which NUMA node the instance is using, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh numatune <vm>

     Replace <vm> with the name of the instance.

Procedure

1. To cold migrate an instance, run the following command to power off and move the instance:

   (overcloud)$ openstack server migrate <vm> --wait

   • Replace <vm> with the name or ID of the instance to migrate.
   • Specify the --block-migration flag if migrating a locally stored volume.
2. Wait for migration to complete. See Checking migration status to check the status of the migration.

3. Check the status of the instance:

   (overcloud)$ openstack server list --all-projects

   A status of "VERIFY_RESIZE" indicates you need to confirm or revert the migration:

   • If the migration worked as expected, confirm it:

     (overcloud)$ openstack server resize --confirm <vm>`

     Replace <vm> with the name or ID of the instance to migrate. A status of "ACTIVE" indicates that the instance is ready to use.

   • If the migration did not work as expected, revert it:

     (overcloud)$ openstack server resize --revert <vm>`

     Replace <vm> with the name or ID of the instance.

4. Restart the instance:

   (overcloud)$ openstack server start <vm>

   Replace <vm> with the name or ID of the instance.

Procedure

1. To live migrate an instance, specify the instance and the destination Compute node:

   (overcloud)$ openstack server migrate <vm> --live <dest> --wait

   • Replace <vm> with the name or ID of the instance.

   • Replace <dest> with the name or ID of the destination Compute node.

     注記

     The openstack server migrate command covers migrating instances with shared storage, which is the default. Specify the --block-migration flag to migrate a locally stored volume:

     (overcloud)$ openstack server migrate <vm> --live <dest> --wait --block-migration

2. Confirm that the instance is migrating:

   $ openstack server show <vm>
   
   +----------------------+--------------------------------------+
   | Field                | Value                                |
   +----------------------+--------------------------------------+
   | ...                  | ...                                  |
   | status               | MIGRATING                            |
   | ...                  | ...                                  |
   +----------------------+--------------------------------------+

3. Wait for migration to complete. See Checking migration status to check the status of the migration.

4. Check the status of the instance to confirm if the migration was successful:

   (overcloud)$ openstack server list --host <dest> --all-projects

   Replace <dest> with the name or ID of the destination Compute node.

5. Optional: For instances that use NUMA, CPU-pinning, or DPDK, retrieve information about the NUMA topology from a Compute node to compare it with the NUMA topology that you retrieved during the preparing to migrate procedure. Comparing the NUMA topologies of the source and destination Compute nodes ensures that the source and destination Compute nodes use the same NUMA topology.

   • To view details about NUMA and CPU pinning, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh vcpuinfo <vm>

     • Replace overcloud-compute-n with the host name of the Compute node.
     • Replace <vm> with the name of the instance.

   • To view details about which NUMA node the instance is using, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh numatune <vm>

     • Replace overcloud-compute-n with the host name of the Compute node.
     • Replace <vm> with the name or ID of the instance.

1. Queued: The Compute service has accepted the request to migrate an instance, and migration is pending.
2. Preparing: The Compute service is preparing to migrate the instance.
3. Running: The Compute service is migrating the instance.
4. Post-migrating: The Compute service has built the instance on the destination Compute node and is releasing resources on the source Compute node.
5. Completed: The Compute service has completed migrating the instance and finished releasing resources on the source Compute node.

Procedure

1. Retrieve the list of migration IDs for the instance:

   $ nova server-migration-list <vm>
   
   +----+-------------+-----------  (...)
   | Id | Source Node | Dest Node | (...)
   +----+-------------+-----------+ (...)
   | 2  | -           | -         | (...)
   +----+-------------+-----------+ (...)

   Replace <vm> with the name or ID of the instance.

2. Show the status of the migration:

   $ nova server-migration-show <vm> <migration-id>

   • Replace <vm> with the name or ID of the instance.

   • Replace <migration-id> with the ID of the migration.

     Running the nova server-migration-show command returns the following example output:

     +------------------------+--------------------------------------+
     | Property               | Value                                |
     +------------------------+--------------------------------------+
     | created_at             | 2017-03-08T02:53:06.000000           |
     | dest_compute           | controller                           |
     | dest_host              | -                                    |
     | dest_node              | -                                    |
     | disk_processed_bytes   | 0                                    |
     | disk_remaining_bytes   | 0                                    |
     | disk_total_bytes       | 0                                    |
     | id                     | 2                                    |
     | memory_processed_bytes | 65502513                             |
     | memory_remaining_bytes | 786427904                            |
     | memory_total_bytes     | 1091379200                           |
     | server_uuid            | d1df1b5a-70c4-4fed-98b7-423362f2c47c |
     | source_compute         | compute2                             |
     | source_node            | -                                    |
     | status                 | running                              |
     | updated_at             | 2017-03-08T02:53:47.000000           |
     +------------------------+--------------------------------------+

     ヒント

     The OpenStack Compute service measures progress of the migration by the number of remaining memory bytes to copy. If this number does not decrease over time, the migration might be unable to complete, and the Compute service might abort it.

Procedure

1. Re-enable the source Compute node:

   (overcloud)$ source ~/stackrc
   (undercloud)$ openstack compute service set <source> nova-compute --enable

   Replace <source> with the host name of the source Compute node.

2. Optional: For instances that use DPDK, re-enable the destination Compute node from the undercloud:

   (undercloud)$ openstack compute service set <dest> nova-compute --enable

   Replace <dest> with the host name of the destination Compute node.

Procedure

1. Generate a new roles data file named roles_data_pci_passthrough.yaml that includes the Controller, Compute, and ComputePCI roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_pci_passthrough.yaml \
    Compute:ComputePCI Compute Controller

2. Open roles_data_pci_passthrough.yaml and edit or add the following parameters and sections:

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputePCI
   Role name	name: Compute	name: ComputePCI
   description	Basic Compute Node role	PCI Passthrough Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepci-%index%
   deprecated_nic_config_name	compute.yaml	compute-pci-passthrough.yaml

3. Register the PCI passthrough Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.

4. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Inspecting the hardware of nodes in the Director Installation and Usage guide.

5. Create the compute-pci-passthrough bare metal flavor to use to tag nodes that you want to designate for PCI passthrough:

   (undercloud)$ openstack flavor create --id auto \
    --ram <ram_size_mb> --disk <disk_size_gb> \
    --vcpus <no_vcpus> compute-pci-passthrough

   • Replace <ram_size_mb> with the RAM of the bare metal node, in MB.
   • Replace <disk_size_gb> with the size of the disk on the bare metal node, in GB.

   • Replace <no_vcpus> with the number of CPUs on the bare metal node.

     注記

     These properties are not used for scheduling instances. However, the Compute scheduler does use the disk size to determine the root partition size.

6. Tag each bare metal node that you want to designate for PCI passthrough with a custom PCI passthrough resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.PCI-PASSTHROUGH <node>

   Replace <node> with the ID of the bare metal node.

7. Associate the compute-pci-passthrough flavor with the custom PCI passthrough resource class:

   (undercloud)$ openstack flavor set \
    --property resources:CUSTOM_BAREMETAL_PCI_PASSTHROUGH=1 \
     compute-pci-passthrough

   To determine the name of a custom resource class that corresponds to a resource class of a Bare Metal service node, convert the resource class to uppercase, replace all punctuation with an underscore, and prefix with CUSTOM_.

   注記

   A flavor can request only one instance of a bare metal resource class.

8. Set the following flavor properties to prevent the Compute scheduler from using the bare metal flavor properties to schedule instances:

   (undercloud)$ openstack flavor set \
    --property resources:VCPU=0 --property resources:MEMORY_MB=0 \
    --property resources:DISK_GB=0 compute-pci-passthrough

9. Add the following parameters to the node-info.yaml file to specify the number of PCI passthrough Compute nodes, and the flavor to use for the PCI passthrough designated Compute nodes:

   parameter_defaults:
     OvercloudComputePCIFlavor: compute-pci-passthrough
     ComputePCICount: 3

10. To verify that the role was created, enter the following command:

    (undercloud)$ openstack overcloud profiles list

Procedure

1. Create an environment file to configure the Controller node on the overcloud for PCI passthrough, for example, pci_passthrough_controller.yaml.

2. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthrough_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

3. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthrough_controller.yaml:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"

   For more information about configuring the device_type field, see PCI passthrough device type field.

   注記

   If the nova-api service is running in a role different from the Controller role, replace ControllerExtraConfig with the user role in the format <Role>ExtraConfig.

4. Optional: To set a default NUMA affinity policy for PCI passthrough devices, add numa_policy to the nova::pci::aliases: configuration from step 3:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"
           numa_policy: "preferred"

5. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthrough_compute.yaml.

6. To specify the available PCIs for the devices on the Compute node, use the vendor_id and product_id options to add all matching PCI devices to the pool of PCI devices available for passthrough to instances. For example, to add all Intel® Ethernet Controller X710 devices to the pool of PCI devices available for passthrough to instances, add the following configuration to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIParameters:
       NovaPCIPassthrough:
         - vendor_id: "8086"
           product_id: "1572"

   For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

7. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the PCI passthrough Compute node, add the following to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"

   注記

   The Compute node aliases must be identical to the aliases on the Controller node. Therefore, if you added numa_affinity to nova::pci::aliases in pci_passthrough_controller.yaml, then you must also add it to nova::pci::aliases in pci_passthrough_compute.yaml.

8. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthrough_compute.yaml. For example, use the following KernalArgs settings to enable an Intel IOMMU:

   parameter_defaults:
     ...
     ComputePCIParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   To enable an AMD IOMMU, set KernelArgs to "amd_iommu=on iommu=pt".

9. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/pci_passthrough_controller.yaml \
     -e /home/stack/templates/pci_passthrough_compute.yaml \

10. Create and configure the flavors that your cloud users can use to request the PCI devices. The following example requests two devices, each with a vendor ID of 8086 and a product ID of 1572, using the alias defined in step 7:

    (overcloud)# openstack flavor set \
     --property "pci_passthrough:alias"="a1:2" device_passthrough

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor device_passthrough \
    --image <image> --wait test-pci

2. Log in to the instance as a cloud user. For more information, see Log in to an Instance.

3. To verify that the PCI device is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <device_name>

Procedure

1. Open your Compute environment file.

2. Add the heat parameter that controls the cron job that you want to add or modify. For example, to purge the shadow tables immediately after they are archived, set the following parameter to "True":

   parameter_defaults:
     …
     NovaCronArchiveDeleteRowsPurge: True

   For a complete list of the heat parameters to manage database cron jobs, see Configuration options for Openstack Compute (nova) automated database management.

3. Save the updates to your Compute environment file.

4. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Configure the amount of huge page memory to reserve on each NUMA node for processes that are not instances:

   parameter_defaults:
     NovaReservedHugePages: ["node:0,size:2048,count:64","node:1,size:1GB,count:1"]

   Where:

   Attribute	Description
   size	The size of the allocated huge page. Valid values: * 2048 (for 2MB) * 1GB
   count	The number of huge pages used by OVS per NUMA node. For example, for 4096 of socket memory used by Open vSwitch, set this to 2.

2. (Optional) To allow instances to allocate 1GB huge pages, configure the CPU feature flags, cpu_model_extra_flags, to include "pdpe1gb":

   parameter_defaults:
      ComputeExtraConfig:
        nova::compute::libvirt::libvirt_cpu_mode: 'custom'
        nova::compute::libvirt::libvirt_cpu_model: 'Haswell-noTSX'
        nova::compute::libvirt::libvirt_cpu_model_extra_flags: 'vmx, pdpe1gb'

   注記

   • CPU feature flags do not need to be configured to allow instances to only request 2 MB huge pages.
   • You can only allocate 1G huge pages to an instance if the host supports 1G huge page allocation.
   • You only need to set cpu_model_extra_flags to pdpe1gb when cpu_mode is set to host-model or custom.
   • If the host supports pdpe1gb, and host-passthrough is used as the cpu_mode, then you do not need to set pdpe1gb as a cpu_model_extra_flags. The pdpe1gb flag is only included in Opteron_G4 and Opteron_G5 CPU models, it is not included in any of the Intel CPU models supported by QEMU.
   • To mitigate for CPU hardware issues, such as Microarchitectural Data Sampling (MDS), you might need to configure other CPU flags. For more information, see RHOS Mitigation for MDS ("Microarchitectural Data Sampling") Security Flaws.

3. To avoid loss of performance after applying Meltdown protection, configure the CPU feature flags, cpu_model_extra_flags, to include "+pcid":

   parameter_defaults:
      ComputeExtraConfig:
        nova::compute::libvirt::libvirt_cpu_mode: 'custom'
        nova::compute::libvirt::libvirt_cpu_model: 'Haswell-noTSX'
        nova::compute::libvirt::libvirt_cpu_model_extra_flags: 'vmx, pdpe1gb, +pcid'

   ヒント

   For more information, see Reducing the performance impact of Meltdown CVE fixes for OpenStack guests with "PCID" CPU feature flag.

4. Add NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter in each Compute environment file, if not already present.

5. Apply this huge page configuration by adding the environment file(s) to your deployment command and deploying the overcloud:

   (undercloud) $ openstack overcloud deploy --templates \
     -e [your environment files]
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Open your Compute environment file.

2. To always attach a config drive when launching an instance, set the following parameter to True:

   parameter_defaults:
     ComputeExtraConfig:
       nova::compute::force_config_drive: 'true'

3. Optional: To change the format of the config drive from the default value of iso9660 to vfat, add the config_drive_format parameter to your configuration:

   parameter_defaults:
     ComputeExtraConfig:
       nova::compute::force_config_drive: 'true'
       nova::compute::config_drive_format: vfat

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

Verification

1. Create an instance:

   (overcloud)$ openstack server create --flavor m1.tiny \
    --image cirros test-config-drive-instance

2. Log in to the instance.

3. Mount the config drive:

   • If the instance OS uses udev:

     # mkdir -p /mnt/config
     # mount /dev/disk/by-label/config-2 /mnt/config

   • If the instance OS does not use udev, you need to first identify the block device that corresponds to the config drive:

     # blkid -t LABEL="config-2" -odevice
     /dev/vdb
     # mkdir -p /mnt/config
     # mount /dev/vdb /mnt/config

4. Inspect the files in the mounted config drive directory, mnt/config/openstack/{version}/, for your metadata.

Procedure

1. Create the JSON file for the metadata.
2. Open your Compute environment file.

3. Add the path to the JSON file to your environment file:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_jsonfile_path:
           value: <path_to_the_JSON_file>

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

Procedure

1. Open your Compute environment file.

2. Add DynamicJSON to the vendordata provider module:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_providers:
           value: StaticJSON,DynamicJSON

3. Specify the REST services to contact to generate the metadata. You can specify as many target REST services as required, for example:

   parameter_defaults:
     ComputeExtraConfig:
       nova::config::nova_config:
         ...
         api/vendordata_providers:
           value: StaticJSON,DynamicJSON
         api/vendordata_dynamic_targets:
           value: target1@http://127.0.0.1:125
         api/vendordata_dynamic_targets:
           value: target2@http://127.0.0.1:126

   The Compute service generates the JSON file, vendordata2.json, to contain the metadata retrieved from the configured target services, and stores it in the config drive directory.

   注記

   Do not use the same name for a target service more than once.

4. Save the updates to your Compute environment file.

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml \

1. Install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud) [stack@undercloud-0 ~]$ sudo yum install libguestfs-tools

   重要

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

2. Extract the images:

   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/overcloud-full.tar
   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent.tar

3. Copy the default image:

   (undercloud) [stack@undercloud-0 ~]$ cp overcloud-full.qcow2 overcloud-realtime-compute.qcow2

4. Register the image and configure the required subscriptions:

   (undercloud) [stack@undercloud-0 ~]$  virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager register --username=[username] --password=[password]'
   [  0.0] Examining the guest ...
   [ 10.0] Setting a random seed
   [ 10.0] Running: subscription-manager register --username=[username] --password=[password]
   [ 24.0] Finishing off

   Replace the username and password values with your Red Hat customer account details. For general information about building a Real-time overcloud image, see the Modifying the Red Hat Enterprise Linux OpenStack Platform Overcloud Image with virt-customize knowledgebase article.

5. Find the SKU of the Red Hat OpenStack Platform for Real Time subscription. The SKU might be located on a system that is already registered to the Red Hat Subscription Manager with the same account and credentials. For example:

   $ sudo subscription-manager list

6. Attach the Red Hat OpenStack Platform for Real Time subscription to the image:

   (undercloud) [stack@undercloud-0 ~]$  virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager attach --pool [subscription-pool]'

7. Create a script to configure rt on the image:

   (undercloud) [stack@undercloud-0 ~]$ cat rt.sh
     #!/bin/bash
   
     set -eux
   
     subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-openstack-13-rpms --enable=rhel-7-server-nfv-rpms
     yum -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     yum -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
   
     # END OF SCRIPT

8. Run the script to configure the real-time image:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

9. Re-label SELinux:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

10. Extract vmlinuz and initrd:

    (undercloud) [stack@undercloud-0 ~]$ mkdir image
    (undercloud) [stack@undercloud-0 ~]$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/vmlinuz-3.10.0-862.rt56.804.el7.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/initramfs-3.10.0-862.rt56.804.el7.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud) [stack@undercloud-0 ~]$ guestunmount image

    注記

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

11. Upload the image:

    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud image upload --update-existing --os-image-name overcloud-realtime-compute.qcow2

1. Based on the /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml file, create a compute-real-time.yaml environment file that sets the parameters for the ComputeRealTime role.

   cp /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml /home/stack/templates/compute-real-time.yaml

   The file must include values for the following parameters:

   • IsolCpusList and NovaVcpuPinSet. List of isolated CPU cores and virtual CPU pins to reserve for real-time workloads. This value depends on the CPU hardware of your Real-time Compute nodes.
   • KernelArgs. Arguments to pass to the kernel of the Real-time Compute nodes. For example, you can use default_hugepagesz=1G hugepagesz=1G hugepages=<number_of_1G_pages_to_reserve> hugepagesz=2M hugepages=<number_of_2M_pages> to define the memory requirements of guests that have huge pages with multiple sizes. In this example, the default size is 1GB but you can also reserve 2M huge pages.

2. Add the ComputeRealTime role to your roles data file and regenerate the file. For example:

   $ openstack overcloud roles generate -o /home/stack/templates/rt_roles_data.yaml Controller Compute ComputeRealTime

   This command generates a ComputeRealTime role with contents similar to the following example, and also sets the ImageDefault option to overcloud-realtime-compute.

   ###############################################################
   # Role: ComputeRealTime                                                               #
   ###############################################################
   
   - name: ComputeRealTime
     description: |
       Compute role that is optimized for real-time behaviour. When using this role
       it is mandatory that an overcloud-realtime-compute image is available and
       the role specific parameters IsolCpusList and NovaVcpuPinSet are set
       accordingly to the hardware of the real-time compute nodes.
     CountDefault: 1
     networks:
       - InternalApi
       - Tenant
       - Storage
     HostnameFormatDefault: '%stackname%-computerealtime-%index%'
     disable_upgrade_deployment: True
     ImageDefault: overcloud-realtime-compute
     RoleParametersDefault:
       TunedProfileName: "realtime-virtual-host"
       KernelArgs: ""      # these must be set in an environment file or similar
       IsolCpusList: ""    # according to the hardware of real-time nodes
       NovaVcpuPinSet: ""  #
     ServicesDefault:
       - OS::TripleO::Services::Aide
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::ComputeCeilometerAgent
       - OS::TripleO::Services::ComputeNeutronCorePlugin
       - OS::TripleO::Services::ComputeNeutronL3Agent
       - OS::TripleO::Services::ComputeNeutronMetadataAgent
       - OS::TripleO::Services::ComputeNeutronOvsAgent
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::Ipsec
       - OS::TripleO::Services::Iscsid
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::LoginDefs
       - OS::TripleO::Services::MySQLClient
       - OS::TripleO::Services::NeutronBgpVpnBagpipe
       - OS::TripleO::Services::NeutronLinuxbridgeAgent
       - OS::TripleO::Services::NeutronVppAgent
       - OS::TripleO::Services::NovaCompute
       - OS::TripleO::Services::NovaLibvirt
       - OS::TripleO::Services::NovaMigrationTarget
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::OpenDaylightOvs
       - OS::TripleO::Services::Rhsm
       - OS::TripleO::Services::RsyslogSidecar
       - OS::TripleO::Services::Securetty
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::SkydiveAgent
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Sshd
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Vpp
       - OS::TripleO::Services::OVNController
       - OS::TripleO::Services::OVNMetadataAgent
       - OS::TripleO::Services::Ptp

   For general information about custom roles and about the roles-data.yaml, see the Roles section.

3. Create the compute-realtime flavor to tag nodes that you want to designate for real-time workloads. For example:

   $ source ~/stackrc
   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-realtime
   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute-realtime" compute-realtime

4. Tag each node that you want to designate for real-time workloads with the compute-realtime profile.

   $ openstack baremetal node set --property capabilities='profile:compute-realtime,boot_option:local' <NODE UUID>

5. Map the ComputeRealTime role to the compute-realtime flavor by creating an environment file with the following content:

   parameter_defaults:
     OvercloudComputeRealTimeFlavor: compute-realtime

6. Run the openstack overcloud deploy command with the -e option and specify all the environment files that you created, as well as the new roles file. For example:

   $ openstack overcloud deploy -r /home/stack/templates/rt~/my_roles_data.yaml  -e /home/stack/templates/compute-real-time.yaml <FLAVOR_ENV_FILE>

   注記

   If you want to run additional real-time instances on the same Compute node, you can change the priority of the instances in the realtime_schedule_priority paremeter in the nova.conf file.

1. Create the compute-real-time.yaml file with the following parameters:

   parameter_defaults:
     ComputeRealTimeParameters:
       IsolCpusList: "1"
       NovaVcpuPinSet: "1"
       KernelArgs: "default_hugepagesz=1G hugepagesz=1G hugepages=16"

2. Create a new rt_roles_data.yaml file with the ComputeRealTime role.

   $ openstack overcloud roles generate -o ~/rt_roles_data.yaml Controller ComputeRealTime

3. Deploy the overcloud, adding both your new real-time roles data file and your real-time environment file to the stack along with your other environment files:

   (undercloud) $ openstack overcloud deploy --templates \
     -r /home/stack/rt_roles_data.yaml
     -e [your environment files]
     -e /home/stack/templates/compute-real-time.yaml

   This command deploys one Controller node and one Real-time Compute node.

4. Log into the Real-time Compute node and check the following parameters. Replace <...> with the values of the relevant parameters from the compute-real-time.yaml.

   [root@overcloud-computerealtime-0 ~]# uname -a
   Linux overcloud-computerealtime-0 3.10.0-693.11.1.rt56.632.el7.x86_64 #1 SMP PREEMPT RT Wed Dec 13 13:37:53 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
   [root@overcloud-computerealtime-0 ~]# cat /proc/cmdline
   BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.11.1.rt56.632.el7.x86_64 root=UUID=45ae42d0-58e7-44fe-b5b1-993fe97b760f ro console=tty0 crashkernel=auto console=ttyS0,115200 default_hugepagesz=1G hugepagesz=1G hugepages=16
   [root@overcloud-computerealtime-0 ~]# tuned-adm active
   Current active profile: realtime-virtual-host
   [root@overcloud-computerealtime-0 ~]# grep ^isolated_cores /etc/tuned/realtime-virtual-host-variables.conf
   isolated_cores=<IsolCpusList>
   [root@overcloud-computerealtime-0 ~]# cat /usr/lib/tuned/realtime-virtual-host/lapic_timer_adv_ns
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# cat /sys/module/kvm/parameters/lapic_timer_advance_ns
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# grep ^vcpu_pin_set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf
   vcpu_pin_set=<NovaVcpuPinSet>

Launching a real-time instance

1. Make sure that the compute-realtime flavor exists on the overcloud, as described in the Deploying the Real-time Compute Role section.

2. Launch the real-time instance.

   # openstack server create  --image <rhel> --flavor r1.small --nic net-id=<dpdk-net> test-rt

3. If you have administrator access to the Compute host, you can optionally verify that the instance uses the assigned emulator threads.

   # virsh dumpxml <instance-id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>

1. Set the emulator_thread_policy parameter to isolate. For example:

1. Configure your flavor to use a dedicated CPU policy. To do so, set the hw:cpu_policy parameter to dedicated on the flavor. For example:

1. Build a custom GPU-enabled overcloud image.
2. Prepare the GPU role, profile, and flavor for designating Compute nodes for vGPU.
3. Configure the Compute node for vGPU.
4. Deploy the overcloud.

Procedure

1. To determine the vendor ID and product ID for each passthrough device type, run the following command on the physical server that has the PCI cards:

   # lspci -nn | grep -i <gpu_name>

   For example, to determine the vendor and product ID for an NVIDIA GPU, run the following command:

   # lspci -nn | grep -i nvidia
   3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
   d8:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1db4] (rev a1)

2. To configure the Controller node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_controller.yaml.

3. Add PciPassthroughFilter to the NovaSchedulerDefaultFilters parameter in pci_passthru_controller.yaml:

   parameter_defaults:
     NovaSchedulerDefaultFilters: ['RetryFilter','AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

4. To specify the PCI alias for the devices on the Controller node, add the following to pci_passthru_controller.yaml:

   ControllerExtraConfig:
       nova::pci::aliases:
         -  name: "t4"
            product_id: "1eb8"
            vendor_id: "10de"
         -  name: "v100"
            product_id: "1db4"
            vendor_id: "10de"

   注記

   If the nova-api service is running in a role other than the Controller, then replace ControllerExtraConfig with the user role, in the format <Role>ExtraConfig.

5. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_compute.yaml.

6. To specify the available PCIs for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   parameter_defaults:
     NovaPCIPassthrough:
       - vendor_id: "10de"
         product_id: "1eb8"

7. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthru_compute.yaml:

      parameter_defaults:
         ...
         ComputeParameters:
           KernelArgs: "intel_iommu=on iommu=pt"

8. Deploy the overcloud, adding your custom environment files to the stack along with your other environment files:

   (undercloud) $ openstack overcloud deploy --templates \
     -e [your environment files]
     -e /home/stack/templates/pci_passthru_controller.yaml
     -e /home/stack/templates/pci_passthru_compute.yaml

9. Configure a flavor to request the PCI devices. The following example requests two devices, each with a vendor ID of 10de and a product ID of 13f2:

   # openstack flavor set m1.large --property "pci_passthrough:alias"="t4:2"

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor m1.large --image rhelgpu --wait test-pci

2. Log in to the instance as a cloud user.

3. Install the GPU driver on the instance. For example, run the following script to install an NVIDIA driver:

   $ sh NVIDIA-Linux-x86_64-430.24-grid.run

4. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

5. To check the NVIDIA System Management Interface status, run the following command from the instance:

   $ nvidia-smi

   Example output:

   -----------------------------------------------------------------------------
   | NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
   |---------------------------------------------------------------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |===========================================================================|
   |   0  Tesla T4            Off  | 00000000:01:00.0 Off |                    0 |
   | N/A   43C    P0    20W /  70W |      0MiB / 15109MiB |      0%      Default |
   ---------------------------------------------------------------------------
   
   -----------------------------------------------------------------------------
   | Processes:                                                       GPU Memory |
   |  GPU       PID   Type   Process name                             Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   -----------------------------------------------------------------------------