Instance with SR-IOV port fail on with Cisco UCSM Mech Driver on OSP13

Issue

• Customer is configuring SR-IOV on Cisco UCS hardware using Cisco UCSM mech driver on Openstack 13.

• Same config was done on OSP10 and it was working fine.

• /var/log/containers/nova/nova-compute.log:

2019-10-10 12:19:51.527 1 ERROR nova.compute.manager [req-4e85d42b-ffe1-4519-aaee-832002c9c34f da60e34aab124eef990818c195120f97 20cee67b952e472586f7ef664bc68bf2 - default default] Instance failed network setup after 1 attempt(s): PortBindingFailed: Binding failed for port 70774caa-bc74-4088-9add-21e332279ff0, please check neutron logs for more information.
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager Traceback (most recent call last):
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1435, in _allocate_network_async
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     bind_host_id=bind_host_id)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 990, in allocate_for_instance
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     bind_host_id, available_macs, requested_ports_dict)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1123, in _update_ports_for_instance
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     vif.destroy()
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     self.force_reraise()
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1093, in _update_ports_for_instance
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     port_client, instance, port_id, port_req_body)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 471, in _update_port
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     _ensure_no_port_binding_failure(port)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 216, in _ensure_no_port_binding_failure
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     raise exception.PortBindingFailed(port_id=port['id'])
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 70774caa-bc74-4088-9add-21e332279ff0, please check neutron logs for more information.
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager

• /var/log/containers/nova/nova-compute.log:

2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [req-375e636e-36f9-410b-b95a-d65eb0be24ad 6afe4e3d4be04288bc741ec8ab782f28 8ef4b147b8a342de9b07b0f760ab1394 - default default] [instance: c4d64fca-1272-49ff-9033-46feaf74312e] Instance failed to spawn: libvirtError: internal error: missing IFLA_VF_INFO in netlink response
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e] Traceback (most recent call last):
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2273, in _build_resources
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     yield resources
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2053, in _build_and_run_instance
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     block_device_info=block_device_info)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3115, in spawn
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     destroy_disks_on_failure=True)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5645, in _create_domain_and_network
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     destroy_disks_on_failure)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     self.force_reraise()
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     six.reraise(self.type_, self.value, self.tb)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5614, in _create_domain_and_network
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     post_xml_callback=post_xml_callback)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5549, in _create_domain
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     guest.launch(pause=pause)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     self._encoded_xml, errors='ignore')
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     self.force_reraise()
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     six.reraise(self.type_, self.value, self.tb)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     return self._domain.createWithFlags(flags)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     rv = execute(f, *args, **kwargs)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     six.reraise(c, e, tb)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     rv = meth(*args, **kwargs)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e] libvirtError: internal error: missing IFLA_VF_INFO in netlink response
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e] 

• In /var/log/containers/neutron/neutron-server.log:

2019-10-10 12:19:51.097 1 DEBUG neutron.plugins.ml2.managers [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Attempting to bind port 70774caa-bc74-4088-9add-21e332279ff0 on host overcloud-compute-0 for vnic_type direct with profile {"pci_slot": "0000:13:00.4", "physical_network": "physnet-sriov-vnic9", "pci_vendor_info": "1137:0071"} bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:745
2019-10-10 12:19:51.097 1 DEBUG neutron.plugins.ml2.managers [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Attempting to bind port 70774caa-bc74-4088-9add-21e332279ff0 on host overcloud-compute-0 at level 0 using segments [{'network_id': 'a748cc76-0426-4868-8c0b-aaaa63baaf25', 'segmentation_id': 1603, 'physical_network': u'physnet-sriov-vnic9', 'id': 'bfd87337-5930-49a8-9633-fd955fc16918', 'network_type': u'vlan'}] _bind_port_level /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:766
2019-10-10 12:19:51.098 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.mech_driver.mech_openvswitch [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Refusing to bind due to unsupported vnic_type: direct with no switchdev capability bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/mech_driver/mech_openvswitch.py:90
2019-10-10 12:19:51.098 1 DEBUG networking_cisco.ml2_drivers.ucsm.mech_cisco_ucsm [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Attempting to bind port 70774caa-bc74-4088-9add-21e332279ff0 with vnic_type direct on network a748cc76-0426-4868-8c0b-aaaa63baaf25  bind_port /usr/lib/python2.7/site-packages/networking_cisco/ml2_drivers/ucsm/mech_cisco_ucsm.py:335
2019-10-10 12:19:51.098 1 ERROR neutron.plugins.ml2.managers [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Failed to bind port 70774caa-bc74-4088-9add-21e332279ff0 on host overcloud-compute-0 for vnic_type direct using segments [{'network_id': 'a748cc76-0426-4868-8c0b-aaaa63baaf25', 'segmentation_id': 1603, 'physical_network': u'physnet-sriov-vnic9', 'id': 'bfd87337-5930-49a8-9633-fd955fc16918', 'network_type': u'vlan'}]