Chapter 4. Configuring CPUs on Compute nodes

As a cloud administrator, you can configure the scheduling and placement of instances for optimal performance by creating customized flavors to target specialized workloads, including NFV and High Performance Computing (HPC).

Use the following features to tune your instances for optimal CPU performance:

  • CPU pinning: Pin virtual CPUs to physical CPUs.
  • Emulator threads: Pin emulator threads associated with the instance to physical CPUs.
  • CPU feature flags: Configure the standard set of CPU feature flags that are applied to instances to improve live migration compatibility across Compute nodes.

4.1. Configuring CPU pinning on Compute nodes

You can configure each instance CPU process to run on a dedicated host CPU by enabling CPU pinning on the Compute nodes. When an instance uses CPU pinning, each instance vCPU process is allocated its own host pCPU that no other instance vCPU process can use. Instances that run on Compute nodes with CPU pinning enabled have a NUMA topology. Each NUMA node of the instance NUMA topology maps to a NUMA node on the host Compute node.

You can configure the Compute scheduler to schedule instances with dedicated (pinned) CPUs and instances with shared (floating) CPUs on the same Compute node. To configure CPU pinning on Compute nodes that have a NUMA topology, you must complete the following:

  1. Designate Compute nodes for CPU pinning.
  2. Configure the Compute nodes to reserve host cores for pinned instance vCPU processes, floating instance vCPU processes, and host processes.
  3. Deploy the overcloud.
  4. Create a flavor for launching instances that require CPU pinning.
  5. Create a flavor for launching instances that use shared, or floating, CPUs.
Note

Configuring CPU pinning creates an implicit NUMA topology on the instance even if a NUMA topology is not requested.

4.1.1. Prerequisites

4.1.2. Designating Compute nodes for CPU pinning

To designate Compute nodes for instances with pinned CPUs, you must create a new role file to configure the CPU pinning role, and configure the bare metal nodes with a CPU pinning resource class to use to tag the Compute nodes for CPU pinning.

Note

The following procedure applies to new overcloud nodes that have not yet been provisioned. To assign a resource class to an existing overcloud node that has already been provisioned, you must use the scale down procedure to unprovision the node, then use the scale up procedure to reprovision the node with the new resource class assignment. For more information, see Scaling overcloud nodes.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    [stack@director ~]$ source ~/stackrc
  3. Generate a new roles data file named roles_data_cpu_pinning.yaml that includes the Controller, Compute, and ComputeCPUPinning roles, along with any other roles that you need for the overcloud:

    (undercloud)$ openstack overcloud roles \
     generate -o /home/stack/templates/roles_data_cpu_pinning.yaml \
     Compute:ComputeCPUPinning Compute Controller
  4. Open roles_data_cpu_pinning.yaml and edit or add the following parameters and sections:

    Section/ParameterCurrent valueNew value

    Role comment

    Role: Compute

    Role: ComputeCPUPinning

    Role name

    name: Compute

    name: ComputeCPUPinning

    description

    Basic Compute Node role

    CPU Pinning Compute Node role

    HostnameFormatDefault

    %stackname%-novacompute-%index%

    %stackname%-novacomputepinning-%index%

    deprecated_nic_config_name

    compute.yaml

    compute-cpu-pinning.yaml

  5. Register the CPU pinning Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.
  6. Inspect the node hardware:

    (undercloud)$ openstack overcloud node introspect \
     --all-manageable --provide

    For more information, see Creating an inventory of the bare-metal node hardware in the Installing and managing Red Hat OpenStack Platform with director guide.

  7. Tag each bare metal node that you want to designate for CPU pinning with a custom CPU pinning resource class:

    (undercloud)$ openstack baremetal node set \
     --resource-class baremetal.CPU-PINNING <node>

    Replace <node> with the ID of the bare metal node.

  8. Add the ComputeCPUPinning role to your node definition file, overcloud-baremetal-deploy.yaml, and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes:

    - name: Controller
      count: 3
    - name: Compute
      count: 3
    - name: ComputeCPUPinning
      count: 1
      defaults:
        resource_class: baremetal.CPU-PINNING
        network_config:
          template: /home/stack/templates/nic-config/myRoleTopology.j2 1
    1
    You can reuse an existing network topology or create a new custom network interface template for the role. For more information, see Custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

    For more information about the properties you can use to configure node attributes in your node definition file, see Bare metal node provisioning attributes. For an example node definition file, see Example node definition file.

  9. Run the provisioning command to provision the new nodes for your role:

    (undercloud)$ openstack overcloud node provision \
    --stack <stack> \
    [--network-config \]
    --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
    /home/stack/templates/overcloud-baremetal-deploy.yaml
    • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
    • Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used.
  10. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list
  11. If you did not run the provisioning command with the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files:

    parameter_defaults:
       ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2
       ComputeCPUPinningNetworkConfigTemplate: /home/stack/templates/nic-configs/<cpu_pinning_net_top>.j2
       ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2

    Replace <cpu_pinning_net_top> with the name of the file that contains the network topology of the ComputeCPUPinning role, for example, compute.yaml to use the default network topology.

4.1.3. Configuring Compute nodes for CPU pinning

Configure CPU pinning on your Compute nodes based on the NUMA topology of the nodes. Reserve some CPU cores across all the NUMA nodes for the host processes for efficiency. Assign the remaining CPU cores to managing your instances.

This procedure uses the following NUMA topology, with eight CPU cores spread across two NUMA nodes, to illustrate how to configure CPU pinning:

Table 4.1. Example of NUMA Topology

NUMA Node 0

NUMA Node 1

Core 0

Core 1

Core 2

Core 3

Core 4

Core 5

Core 6

Core 7

The procedure reserves cores 0 and 4 for host processes, cores 1, 3, 5 and 7 for instances that require CPU pinning, and cores 2 and 6 for floating instances that do not require CPU pinning.

Procedure

  1. Create an environment file to configure Compute nodes to reserve cores for pinned instances, floating instances, and host processes, for example, cpu_pinning.yaml.
  2. To schedule instances with a NUMA topology on NUMA-capable Compute nodes, add NUMATopologyFilter to the NovaSchedulerEnabledFilters parameter in your Compute environment file, if not already present:

    parameter_defaults:
      NovaSchedulerEnabledFilters:
        - AvailabilityZoneFilter
        - ComputeFilter
        - ComputeCapabilitiesFilter
        - ImagePropertiesFilter
        - ServerGroupAntiAffinityFilter
        - ServerGroupAffinityFilter
        - PciPassthroughFilter
        - NUMATopologyFilter

    For more information on NUMATopologyFilter, see Compute scheduler filters .

  3. To reserve physical CPU cores for the dedicated instances, add the following configuration to cpu_pinning.yaml:

    parameter_defaults:
      ComputeCPUPinningParameters:
        NovaComputeCpuDedicatedSet: 1,3,5,7
  4. To reserve physical CPU cores for the shared instances, add the following configuration to cpu_pinning.yaml:

    parameter_defaults:
      ComputeCPUPinningParameters:
        ...
        NovaComputeCpuSharedSet: 2,6
  5. If you are not using file-backed memory, specify the amount of RAM to reserve for host processes:

    parameter_defaults:
      ComputeCPUPinningParameters:
        ...
        NovaReservedHugePages: <ram>

    Replace <ram> with the amount of RAM to reserve in MB.

  6. To ensure that host processes do not run on the CPU cores reserved for instances, set the parameter IsolCpusList to the CPU cores you have reserved for instances:

    parameter_defaults:
      ComputeCPUPinningParameters:
        ...
        IsolCpusList: 1-3,5-7

    Specify the value of the IsolCpusList parameter using a list, or ranges, of CPU indices separated by a comma.

  7. Add your new files to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
      -e [your environment files] \
      -r /home/stack/templates/roles_data_cpu_pinning.yaml \
      -e /home/stack/templates/network-environment.yaml \
      -e /home/stack/templates/cpu_pinning.yaml \
      -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
      -e /home/stack/templates/node-info.yaml

4.1.4. Creating a dedicated CPU flavor for instances

To enable your cloud users to create instances that have dedicated CPUs, you can create a flavor with a dedicated CPU policy for launching instances.

Prerequisites

Procedure

  1. Source the overcloudrc file:

    (undercloud)$ source ~/overcloudrc
  2. Create a flavor for instances that require CPU pinning:

    (overcloud)$ openstack flavor create --ram <size_mb> \
     --disk <size_gb> --vcpus <no_reserved_vcpus> pinned_cpus
  3. To request pinned CPUs, set the hw:cpu_policy property of the flavor to dedicated:

    (overcloud)$ openstack flavor set \
     --property hw:cpu_policy=dedicated pinned_cpus
  4. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation:

    (overcloud)$ openstack flavor set \
     --property hw:mem_page_size=<page_size> pinned_cpus
    • Replace <page_size> with one of the following valid values:

      • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
      • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
      • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
      • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.
    Note

    To set hw:mem_page_size to small or any, you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes.

  5. To place each vCPU on thread siblings, set the hw:cpu_thread_policy property of the flavor to require:

    (overcloud)$ openstack flavor set \
     --property hw:cpu_thread_policy=require pinned_cpus
    Note
    • If the host does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling fails. To prevent this, set hw:cpu_thread_policy to prefer instead of require. The prefer policy is the default policy that ensures that thread siblings are used when available.
    • If you use hw:cpu_thread_policy=isolate, you must have SMT disabled or use a platform that does not support SMT.

Verification

  1. To verify the flavor creates an instance with dedicated CPUs, use your new flavor to launch an instance:

    (overcloud)$ openstack server create --flavor pinned_cpus \
     --image <image> pinned_cpu_instance

4.1.5. Creating a shared CPU flavor for instances

To enable your cloud users to create instances that use shared, or floating, CPUs, you can create a flavor with a shared CPU policy for launching instances.

Prerequisites

Procedure

  1. Source the overcloudrc file:

    (undercloud)$ source ~/overcloudrc
  2. Create a flavor for instances that do not require CPU pinning:

    (overcloud)$ openstack flavor create --ram <size_mb> \
     --disk <size_gb> --vcpus <no_reserved_vcpus> floating_cpus
  3. To request floating CPUs, set the hw:cpu_policy property of the flavor to shared:

    (overcloud)$ openstack flavor set \
     --property hw:cpu_policy=shared floating_cpus
  4. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation:

    (overcloud)$ openstack flavor set \
     --property hw:mem_page_size=<page_size> pinned_cpus
    • Replace <page_size> with one of the following valid values:

      • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
      • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
      • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
      • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.
    Note

    To set hw:mem_page_size to small or any, you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes.

4.1.6. Creating a mixed CPU flavor for instances

To enable your cloud users to create instances that have a mix of dedicated and shared CPUs, you can create a flavor with a mixed CPU policy for launching instances.

Procedure

  1. Source the overcloudrc file:

    (undercloud)$ source ~/overcloudrc
  2. Create a flavor for instances that require a mixed of dedicated and shared CPUs:

    (overcloud)$ openstack flavor create --ram <size_mb> \
     --disk <size_gb> --vcpus <number_of_reserved_vcpus> \
     --property hw:cpu_policy=mixed mixed_CPUs_flavor
  3. Specify which CPUs must be dedicated or shared:

    (overcloud)$ openstack flavor set \
     --property hw:cpu_dedicated_mask=<CPU_number> \
     mixed_CPUs_flavor
    • Replace <CPU_number> with the CPUs that must be either dedicated or shared:

      • To specify dedicated CPUs, specify the CPU number or CPU range. For example, set the property to 2-3 to specify that CPUs 2 and 3 are dedicated and all the remaining CPUs are shared.
      • To specify shared CPUs, prepend the CPU number or CPU range with a caret (^). For example, set the property to ^0-1 to specify that CPUs 0 and 1 are shared and all the remaining CPUs are dedicated.
  4. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation:

    (overcloud)$ openstack flavor set \
     --property hw:mem_page_size=<page_size> pinned_cpus
    • Replace <page_size> with one of the following valid values:

      • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
      • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
      • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
      • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.
    Note

    To set hw:mem_page_size to small or any, you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes.

4.1.7. Configuring CPU pinning on Compute nodes with simultaneous multithreading (SMT)

If a Compute node supports simultaneous multithreading (SMT), group thread siblings together in either the dedicated or the shared set. Thread siblings share some common hardware which means it is possible for a process running on one thread sibling to impact the performance of the other thread sibling.

For example, the host identifies four logical CPU cores in a dual core CPU with SMT: 0, 1, 2, and 3. Of these four, there are two pairs of thread siblings:

  • Thread sibling 1: logical CPU cores 0 and 2
  • Thread sibling 2: logical CPU cores 1 and 3

In this scenario, do not assign logical CPU cores 0 and 1 as dedicated and 2 and 3 as shared. Instead, assign 0 and 2 as dedicated and 1 and 3 as shared.

The files /sys/devices/system/cpu/cpuN/topology/thread_siblings_list, where N is the logical CPU number, contain the thread pairs. You can use the following command to identify which logical CPU cores are thread siblings:

# grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u

The following output indicates that logical CPU core 0 and logical CPU core 2 are threads on the same core:

/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,2
/sys/devices/system/cpu/cpu2/topology/thread_siblings_list:1,3

4.1.8. Additional resources

4.2. Configuring emulator threads

Compute nodes have overhead tasks associated with the hypervisor for each instance, known as emulator threads. By default, emulator threads run on the same CPUs as the instance, which impacts the performance of the instance.

You can configure the emulator thread policy to run emulator threads on separate CPUs to those the instance uses.

Note
  • To avoid packet loss, you must never preempt the vCPUs in an NFV deployment.

Prerequisites

  • CPU pinning must be enabled.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Open your Compute environment file.
  3. To reserve physical CPU cores for instances that require CPU pinning, configure the NovaComputeCpuDedicatedSet parameter in the Compute environment file. For example, the following configuration sets the dedicated CPUs on a Compute node with a 32-core CPU:

    parameter_defaults:
      ...
      NovaComputeCpuDedicatedSet: 2-15,18-31
      ...

    For more information, see Configuring CPU pinning on the Compute nodes.

  4. To reserve physical CPU cores for the emulator threads, configure the NovaComputeCpuSharedSet parameter in the Compute environment file. For example, the following configuration sets the shared CPUs on a Compute node with a 32-core CPU:

    parameter_defaults:
      ...
      NovaComputeCpuSharedSet: 0,1,16,17
      ...
    Note

    The Compute scheduler also uses the CPUs in the shared set for instances that run on shared, or floating, CPUs. For more information, see Configuring CPU pinning on Compute nodes

  5. Add the Compute scheduler filter NUMATopologyFilter to the NovaSchedulerEnabledFilters parameter, if not already present.
  6. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml
  7. Configure a flavor that runs emulator threads for the instance on a dedicated CPU, which is selected from the shared CPUs configured using NovaComputeCpuSharedSet:

    (overcloud)$ openstack flavor set --property hw:cpu_policy=dedicated \
     --property hw:emulator_threads_policy=share \
     dedicated_emulator_threads

    For more information about configuration options for hw:emulator_threads_policy, see Emulator threads policy in Flavor metadata.

4.3. Configuring CPU feature flags for instances

You can enable or disable CPU feature flags for an instance without changing the settings on the host Compute node and rebooting the Compute node. By configuring the standard set of CPU feature flags that are applied to instances, you are helping to achieve live migration compatibility across Compute nodes. You are also helping to manage the performance and security of the instances, by disabling flags that have a negative impact on the security or performance of the instances with a particular CPU model, or enabling flags that provide mitigation from a security problem or alleviates performance problems.

4.3.1. Prerequisites

  • The CPU model and feature flags must be supported by the hardware and software of the host Compute node:

    • To check the hardware your host supports, enter the following command on the Compute node:

      $ cat /proc/cpuinfo
    • To check the CPU models supported on your host, enter the following command on the Compute node:

      $ sudo podman exec -it nova_libvirt virsh cpu-models <arch>

      Replace <arch> with the name of the architecture, for example, x86_64.

4.3.2. Configuring CPU feature flags for instances

Configure the Compute service to apply CPU feature flags to instances with specific vCPU models.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    [stack@director ~]$ source ~/stackrc
  3. Open your Compute environment file.
  4. Configure the instance CPU mode:

    parameter_defaults:
      ComputeParameters:
        NovaLibvirtCPUMode: <cpu_mode>

    Replace <cpu_mode> with the CPU mode of each instance on the Compute node. Set to one of the following valid values:

    • host-model: (Default) Use the CPU model of the host Compute node. Use this CPU mode to automatically add critical CPU flags to the instance to provide mitigation from security flaws.
    • custom: Use to configure the specific CPU models each instance should use.

      Note

      You can also set the CPU mode to host-passthrough to use the same CPU model and feature flags as the Compute node for the instances hosted on that Compute node.

  5. Optional: If you set NovaLibvirtCPUMode to custom, configure the instance CPU models that you want to customise:

    parameter_defaults:
      ComputeParameters:
        NovaLibvirtCPUMode: 'custom'
        NovaLibvirtCPUModels: <cpu_model>

    Replace <cpu_model> with a comma-separated list of the CPU models that the host supports. List the CPU models in order, placing the more common and less advanced CPU models first in the list, and the more feature-rich CPU models last, for example, SandyBridge,IvyBridge,Haswell,Broadwell. For a list of model names, see /usr/share/libvirt/cpu_map.xml, or enter the following command on the host Compute node:

    $ sudo podman exec -it nova_libvirt virsh cpu-models <arch>

    Replace <arch> with the name of the architecture of the Compute node, for example, x86_64.

  6. Configure the CPU feature flags for instances with the specified CPU models:

    parameter_defaults:
      ComputeParameters:
        ...
        NovaLibvirtCPUModelExtraFlags: <cpu_feature_flags>

    Replace <cpu_feature_flags> with a comma-separated list of feature flags to enable or disable. Prefix each flag with "+" to enable the flag, or "-" to disable it. If a prefix is not specified, the flag is enabled. For a list of the available feature flags for a given CPU model, see /usr/share/libvirt/cpu_map/*.xml.

    The following example enables the CPU feature flags pcid and ssbd for the IvyBridge and Cascadelake-Server models, and disables the feature flag mtrr.

    parameter_defaults:
      ComputeParameters:
        NovaLibvirtCPUMode: 'custom'
        NovaLibvirtCPUModels: 'IvyBridge','Cascadelake-Server'
        NovaLibvirtCPUModelExtraFlags: 'pcid,+ssbd,-mtrr'
  7. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml