Numa affinity for SRIOV PCI devices not following policy

Issue

• When using the nova flavor spec "hw:pci_numa_affinity_policy": "preferred", we expect the VM to be spawned on the Numa node with PCI devices first and then fall back to Numa nodes with no PCI device.

• Device with SRIOV VF's is on Numa 1:

[root@overcloud-compute-1 ~]# lspci -vvv -s af:00.0 | grep NUMA
    NUMA node: 1

• Flavor policy is set to preferred

(overcloud) [stack@undercloud ~]$ nova flavor-show m1.iperf  | grep extra
| extra_specs                | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "large", "hw:pci_numa_affinity_policy": "preferred"} |

• Node CPU set up:

[root@overcloud-compute-1 ~]# lscpu |  grep NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0-19,40-59
NUMA node1 CPU(s):     20-39,60-79

• Nova vcpu_pin_set:

[root@overcloud-compute-1 ~]# cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep pin_set | grep -v ^#
vcpu_pin_set=2-12,22-32

• Spawning VM using VF's (switchdev) VM is spawned on Numa 0.

[root@overcloud-compute-1 ~]# numastat -c qemu-kvm

Per-node process memory usage (in MBs) for PID 13714 (qemu-kvm)
         Node 0 Node 1 Total
         ------ ------ -----
Huge       8192      0  8192
Heap         28      0    28
Stack         1      0     1
Private      23      7    30
-------  ------ ------ -----
Total      8243      7  8250

• Emulatorpin is set on NUMA0 too:

[root@overcloud-compute-1 ~]# virsh dumpxml 3 | grep emulatorpin
    <emulatorpin cpuset='2-6,8-9,12'/>

• Preferred should spawn on Numa with PCI device first then to non PCI devices as fallback.