Numa affinity for SRIOV PCI devices not following policy

Solution In Progress - Updated -

Issue

  • When using the nova flavor spec "hw:pci_numa_affinity_policy": "preferred", we expect the VM to be spawned on the Numa node with PCI devices first and then fall back to Numa nodes with no PCI device.

  • Device with SRIOV VF's is on Numa 1:

[root@overcloud-compute-1 ~]# lspci -vvv -s af:00.0 | grep NUMA
    NUMA node: 1
  • Flavor policy is set to preferred
(overcloud) [stack@undercloud ~]$ nova flavor-show m1.iperf  | grep extra
| extra_specs                | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "large", "hw:pci_numa_affinity_policy": "preferred"} |
  • Node CPU set up:
[root@overcloud-compute-1 ~]# lscpu |  grep NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0-19,40-59
NUMA node1 CPU(s):     20-39,60-79
  • Nova vcpu_pin_set:
[root@overcloud-compute-1 ~]# cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep pin_set | grep -v ^#
vcpu_pin_set=2-12,22-32
  • Spawning VM using VF's (switchdev) VM is spawned on Numa 0.
[root@overcloud-compute-1 ~]# numastat -c qemu-kvm

Per-node process memory usage (in MBs) for PID 13714 (qemu-kvm)
         Node 0 Node 1 Total
         ------ ------ -----
Huge       8192      0  8192
Heap         28      0    28
Stack         1      0     1
Private      23      7    30
-------  ------ ------ -----
Total      8243      7  8250
  • Emulatorpin is set on NUMA0 too:
[root@overcloud-compute-1 ~]# virsh dumpxml 3 | grep emulatorpin
    <emulatorpin cpuset='2-6,8-9,12'/>
  • Preferred should spawn on Numa with PCI device first then to non PCI devices as fallback.

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In