CPU pinning overlap after live or offline migration in Red Hat OpenStack Platform 13

Solution Verified - Updated -

Issue

  • Multiple instances on the same host are scheduled to use the same pCPUs, even though hw:cpu_policy='dedicated' is set – and consequently, CPU steal time occurs. See below, instances 97,99,105, for example.
[root@overcloud-compute-0 ~]# ./get_instance_cpu_placement.sh
Instance 15 CPU placement:
8,59,68,78,0
<vcpupin vcpu='0' cpuset='8'/>, <vcpupin vcpu='1' cpuset='59'/>, <vcpupin vcpu='2' cpuset='68'/>, <vcpupin vcpu='3' cpuset='78'/>, <emulatorpin cpuset='0'/>

Instance 70 CPU placement:
44,54,29,76,11
<vcpupin vcpu='0' cpuset='44'/>, <vcpupin vcpu='1' cpuset='54'/>, <vcpupin vcpu='2' cpuset='29'/>, <vcpupin vcpu='3' cpuset='76'/>, <emulatorpin cpuset='11'/>

Instance 71 CPU placement:
6,17,24,39,14
<vcpupin vcpu='0' cpuset='6'/>, <vcpupin vcpu='1' cpuset='17'/>, <vcpupin vcpu='2' cpuset='24'/>, <vcpupin vcpu='3' cpuset='39'/>, <emulatorpin cpuset='14'/>

Instance 73 CPU placement:
2,13,67,32,18
<vcpupin vcpu='0' cpuset='2'/>, <vcpupin vcpu='1' cpuset='13'/>, <vcpupin vcpu='2' cpuset='67'/>, <vcpupin vcpu='3' cpuset='32'/>, <emulatorpin cpuset='18'/>

Instance 86 CPU placement:
1,41,46,6,2,42,47,7,55,15,17,57,13,53,16,56,19
<vcpupin vcpu='0' cpuset='1'/>, <vcpupin vcpu='1' cpuset='41'/>, <vcpupin vcpu='2' cpuset='46'/>, <vcpupin vcpu='3' cpuset='6'/>, <vcpupin vcpu='4' cpuset='2'/>, <vcpupin vcpu='5' cpuset='42'/>, <vcpupin vcpu='6' cpuset='47'/>, <vcpupin vcpu='7' cpuset='7'/>, <vcpupin vcpu='8' cpuset='55'/>, <vcpupin vcpu='9' cpuset='15'/>, <vcpupin vcpu='10' cpuset='17'/>, <vcpupin vcpu='11' cpuset='57'/>, <vcpupin vcpu='12' cpuset='13'/>, <vcpupin vcpu='13' cpuset='53'/>, <vcpupin vcpu='14' cpuset='16'/>, <vcpupin vcpu='15' cpuset='56'/>, <emulatorpin cpuset='19'/>

Instance 97 CPU placement:
9,12,26,35,20
<vcpupin vcpu='0' cpuset='9'/>, <vcpupin vcpu='1' cpuset='12'/>, <vcpupin vcpu='2' cpuset='26'/>, <vcpupin vcpu='3' cpuset='35'/>, <emulatorpin cpuset='20'/>

Instance 99 CPU placement:
9,12,26,35,21
<vcpupin vcpu='0' cpuset='9'/>, <vcpupin vcpu='1' cpuset='12'/>, <vcpupin vcpu='2' cpuset='26'/>, <vcpupin vcpu='3' cpuset='35'/>, <emulatorpin cpuset='21'/>

Instance 105 CPU placement:
9,12,26,35,22
<vcpupin vcpu='0' cpuset='9'/>, <vcpupin vcpu='1' cpuset='12'/>, <vcpupin vcpu='2' cpuset='26'/>, <vcpupin vcpu='3' cpuset='35'/>, <emulatorpin cpuset='22'/>
  • Here are the flavors being used:
(overcloud) [stack@undercloud ~]$ openstack flavor show flavor_one
+----------------------------+---------------------------------------------------------------------------------------+                                                                                            
| Field                      | Value                                                                                 |                                                                                            
+----------------------------+---------------------------------------------------------------------------------------+                                                                                            
| OS-FLV-DISABLED:disabled   | False                                                                                 |                                                                                            
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                     |                                                                                            
| access_project_ids         | None                                                                                  |                                                                                            
| disk                       | 80                                                                                    |                                                                                            
| id                         | 96c2631c-f54c-4018-9d93-4eff9bd2c232                                                  |                                                                                            
| name                       | flavor_one                                                                      |                                                                                            
| os-flavor-access:is_public | True                                                                                  |                                                                                            
| properties                 | hw:cpu_policy='dedicated', hw:numa_nodes='4', hw:pci_numa_affinity_policy='preferred' |                                                                                            
| ram                        | 8192                                                                                  |                                                                                            
| rxtx_factor                | 1.0                                                                                   |                                                                                            
| swap                       |                                                                                       |                                                                                            
| vcpus                      | 4                                                                                     |                                                                                            
+----------------------------+---------------------------------------------------------------------------------------+                                                                                            
(overcloud) [stack@undercloud ~]$ openstack flavor show flavor_two
+----------------------------+---------------------------------------------------------------------------------------+                                                                                            
| Field                      | Value                                                                                 |                                                                                            
+----------------------------+---------------------------------------------------------------------------------------+                                                                                            
| OS-FLV-DISABLED:disabled   | False                                                                                 |                                                                                            
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                     |                                                                                            
| access_project_ids         | None                                                                                  |                                                                                            
| disk                       | 80                                                                                    |                                                                                            
| id                         | 18a35106-500d-4939-b0c3-5c922eb40234                                                  |                                                                                            
| name                       | flavor_two                                                                  |                                                                                            
| os-flavor-access:is_public | True                                                                                  |                                                                                            
| properties                 | hw:cpu_policy='dedicated', hw:numa_nodes='4', hw:pci_numa_affinity_policy='preferred' |                                                                                            
| ram                        | 16384                                                                                 |                                                                                            
| rxtx_factor                | 1.0                                                                                   |                                                                                            
| swap                       |                                                                                       |                                                                                            
| vcpus                      | 4                                                                                     |                                                                                            
+----------------------------+---------------------------------------------------------------------------------------+
  • /var/log/containers/nova/nova-compute.log contains errors similar to this one:
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager [req-c7bef2a3-8f36-4a0b-a922-5e17428459bb - - - - -] Error updating resources for node overcloud-compute-0: CPUPinningInvalid: CPU set to pin [6] must be a subset of free CPU set [3, 4, 5, 9, 43, 45, 48, 49]
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager Traceback (most recent call last):
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7426, in update_available_resource_for_node
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     return f(*args, **kwargs)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 730, in _update_available_resource
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     self._update_usage_from_instances(context, instances, nodename)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1233, in _update_usage_from_instances
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     require_allocation_refresh=require_allocation_refresh)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1126, in _update_usage_from_instance
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     sign=sign)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 937, in _update_usage
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     cn, usage, free)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1868, in get_host_numa_usage_from_instance
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     host_numa_topology, instance_numa_topology, free=free))
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1724, in numa_usage_from_instances
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     newcell.pin_cpus(pinned_cpus)
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/objects/numa.py", line 91, in pin_cpus
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager     self.pinned_cpus))
2019-10-15 00:00:41.720 1 ERROR nova.compute.manager CPUPinningInvalid: CPU set to pin [6] must be a subset of free CPU set [3, 4, 5, 9, 43, 45, 48, 49]

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content