After deployments ports are reported as down and no communication possible

Issue

• Customer moved from RHOSP13 to 16.2 and deployed the VMs again with SRIOV as before. However there are no problems when deploying the VMs but the VMs can not talk via SRIOV. Openstack port list shows the ports in down state but on compute they are not, neither on the guest. Seems all SRIOV ports have issues.

• As you can see, some of the ports are down:

(overcloud) [stack@undercloud-0 ~]$ openstack port list | grep "port-"
| 0209dc4a-8532-45e5-8e2e-de20c9bd9d23 | port-ns2-jtsk3vcdsn2p          | 00:00:00:e1:0b:16 | ip_address='10.10.10.46', subnet_id='574f4260-b94e-4e64-b62e-f3c7d15f8380'   | DOWN   |
| 20f34ebd-a9c9-4ca1-8152-cd6477c6c47f | port-ns3-xu7cqw5d54qx          | 00:00:00:e7:77:3a | ip_address='10.10.10.79', subnet_id='6091d127-778e-4a3e-a742-067d22615574'   | DOWN   |
| 20ff61de-5723-42e4-a006-056db06f5bbf | port-ns1-2xoujswozclx          | 00:00:00:6d:e5:62 | ip_address='10.10.10.24', subnet_id='a8de0a7b-0734-4a33-919c-437dbb789ada'   | DOWN   |
| 4143522e-34a1-43f7-bbf8-df63464d47f7 | port-base2-jadks52enjg4        | 00:00:00:7a:68:c8 | ip_address='10.10.4.85', subnet_id='99c0bde1-326a-4514-9666-702391e223e5'  | ACTIVE |
| 4187600a-6fb4-49f8-853f-9f8779cab75b | port-ns1-bi6oaibr7esv          | 00:00:00:06:08:72 | ip_address='10.10.10.23', subnet_id='a8de0a7b-0734-4a33-919c-437dbb789ada'   | DOWN   |
| 68274fa3-b7a8-4a1f-a3b9-00b433185dc6 | port-mgmt-rb56z76s7mmx         | 00:00:00:59:d0:a9 | ip_address='10.12.60.194', subnet_id='fff83f5d-7883-46d5-891b-8a86e3e54f47' | ACTIVE |
| 6c6f7a34-aad4-4a5e-9d00-c07ff4423e15 | port-ns4-p3uxuwae2ls6          | 00:00:00:16:d9:fc | ip_address='10.10.10.121', subnet_id='ea6456f2-93fb-47eb-9fa6-f119a6f40730'  | DOWN   |
| 797a621a-74e5-4e2f-8559-86ad45bfbd33 | port-base1-wxdycmkcds7o        | 00:00:00:a9:10:63 | ip_address='10.10.3.165', subnet_id='2d2b549c-a47f-47ed-b338-277441cd000b' | ACTIVE |
| 7ad7f01a-7333-4621-b11f-d82bfe0cda37 | port-base2-rryxvo2mtozk        | 00:00:00:61:ff:ae | ip_address='10.10.4.40', subnet_id='99c0bde1-326a-4514-9666-702391e223e5'  | ACTIVE |
| 7af30710-d1f9-4fa1-8004-06be2334a58d | port-ew1-eihh76dz53to          | 00:00:00:da:27:9a | ip_address='10.11.5.209', subnet_id='2438255a-45f8-4720-8d54-93d1f15439a9' | DOWN   |
| 890194c2-f89d-4018-86a3-0b469d594b1d | port-mgmt-x4nncn6m46iu         | 00:00:00:b5:69:bb | ip_address='10.12.60.195', subnet_id='fff83f5d-7883-46d5-891b-8a86e3e54f47' | ACTIVE |
| 902ac9e2-4801-4d81-a352-5e0d92a0b27e | port-ns4-jomxxxggxbte          | 00:00:00:71:6e:d6 | ip_address='10.10.10.101', subnet_id='ea6456f2-93fb-47eb-9fa6-f119a6f40730'  | DOWN   |
| 926d9b63-f2f0-4edd-bfc6-349678cb65bd | port-ns3-nghujhzeclgy          | 00:00:00:ac:1c:5e | ip_address='10.10.10.82', subnet_id='6091d127-778e-4a3e-a742-067d22615574'   | DOWN   |
| 9888549e-2bb7-4326-b917-f31433d32471 | port-ew2-a2orvb6jjbp7          | 00:00:00:9f:2f:20 | ip_address='10.11.6.214', subnet_id='22822d3c-051b-4ff8-97db-8aedaba69b75' | DOWN   |
| ab63e5f1-9d34-4e67-bc1b-67f321c92d52 | port-ns2-wvds3k5awjyd          | 00:00:00:b2:f6:c3 | ip_address='10.10.10.61', subnet_id='574f4260-b94e-4e64-b62e-f3c7d15f8380'   | DOWN   |
| b6ac7165-b989-4166-bd67-066be7eb51a2 | port-ew2-mcm76d4dcdtp          | 00:00:00:c9:41:ab | ip_address='10.11.6.200', subnet_id='22822d3c-051b-4ff8-97db-8aedaba69b75' | DOWN   |
| c57553cf-d166-4424-935f-de20b1e9ad0f | port-ew1-blm2tx2wlyak          | 00:00:00:90:52:1e | ip_address='10.11.5.141', subnet_id='2438255a-45f8-4720-8d54-93d1f15439a9' | DOWN   |
| e45b77e6-ad08-4032-8e28-ccd53ce982d9 | port-base1-63qcqb4df3pj        | 00:00:00:fb:7c:59 | ip_address='10.10.3.245', subnet_id='2d2b549c-a47f-47ed-b338-277441cd000b' | ACTIVE |
(overcloud) [stack@undercloud-0 ~]$ openstack port list | grep portA
| 1dd881f9-9f32-4a08-92b8-6a7a69c6a068 | portA-mgmt-z5ewccc2fcal         | 00:00:00:07:72:b3 | ip_address='10.12.60.197', subnet_id='fff83f5d-7883-46d5-891b-8a86e3e54f47' | ACTIVE |
| 2b9c99a3-1197-4e49-a0bf-6a34ccc2e44e | portA-ew1-xjhywcdqo5ye          | 00:00:00:a0:de:d7 | ip_address='10.11.5.147', subnet_id='2438255a-45f8-4720-8d54-93d1f15439a9' | DOWN   |
| 57393704-0300-4935-a904-01fadffb4e8c | portA-base2-7ppmpa6kgah5        | 00:00:00:84:d2:ed | ip_address='10.10.4.15', subnet_id='99c0bde1-326a-4514-9666-702391e223e5'  | ACTIVE |
| 6cf85ad7-583f-4580-91ff-61cb20d5a82f | portA-ew1-ia5kba2wg2wg          | 00:00:00:e8:81:e3 | ip_address='10.11.5.254', subnet_id='2438255a-45f8-4720-8d54-93d1f15439a9' | DOWN   |
| 6f8d8e04-9ff3-463b-8514-59ec76166c14 | portA-mgmt-jbkfkmqhwdxm         | 00:00:00:5e:8a:a5 | ip_address='10.12.60.196', subnet_id='fff83f5d-7883-46d5-891b-8a86e3e54f47' | ACTIVE |
| 7f43b8d5-f068-47a9-a8b0-97babaaa1f55 | portA-ew2-3qtiwagcqq5p          | 00:00:00:ed:09:07 | ip_address='10.11.6.230', subnet_id='22822d3c-051b-4ff8-97db-8aedaba69b75' | DOWN   |
| 81c0b093-7517-467f-9c35-56a6f42a3306 | portA-base1-dmoh267f4n3e        | 00:00:00:db:2c:b2 | ip_address='10.10.3.18', subnet_id='2d2b549c-a47f-47ed-b338-277441cd000b'  | ACTIVE |
| a31bf715-34a8-43db-be6a-853a033aebdf | portA-base1-4bwzuvruc7ls        | 00:00:00:3d:33:f5 | ip_address='10.10.3.53', subnet_id='2d2b549c-a47f-47ed-b338-277441cd000b'  | ACTIVE |
| ee5e5848-df86-412f-8a97-2d2434e95091 | portA-ew2-l4fcovqiuc3w          | 00:00:00:40:22:6f | ip_address='10.11.6.173', subnet_id='22822d3c-051b-4ff8-97db-8aedaba69b75' | DOWN   |
| f3696553-8a6c-42d9-a399-cf17a2e4cd9e | portA-base2-tulwkinons6o        | 00:00:00:de:11:32 | ip_address='10.10.4.74', subnet_id='99c0bde1-326a-4514-9666-702391e223e5'  | ACTIVE |

• In /var/log/messages we can see the following messages:

Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1140: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1140: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1124: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1124: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1123: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1123: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1122: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1122: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1046: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1046: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1045: Affinity broken due to vector space exhaustion.
Feb 21 21:00:51 overcloud-compute-0 kernel: irq 1045: Affinity broken due to vector space exhaustion.
Feb 21 21:00:56 overcloud-compute-0 kernel: irq 1232: Affinity broken due to vector space exhaustion.
...
Feb 21 21:01:07 overcloud-compute-0 kernel: i40e 0000:12:00.0: Cannot disable vlan stripping when port VLAN is set
Feb 21 21:01:07 overcloud-compute-0 kernel: i40e 0000:12:00.0: VF 59 failed opcode 28, retval: -5