Instance with SR-IOV port fail on with Cisco UCSM Mech Driver on OSP13

Solution In Progress - Updated -

Issue

  • Customer is configuring SR-IOV on Cisco UCS hardware using Cisco UCSM mech driver on Openstack 13.

  • Same config was done on OSP10 and it was working fine.

  • /var/log/containers/nova/nova-compute.log:

2019-10-10 12:19:51.527 1 ERROR nova.compute.manager [req-4e85d42b-ffe1-4519-aaee-832002c9c34f da60e34aab124eef990818c195120f97 20cee67b952e472586f7ef664bc68bf2 - default default] Instance failed network setup after 1 attempt(s): PortBindingFailed: Binding failed for port 70774caa-bc74-4088-9add-21e332279ff0, please check neutron logs for more information.
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager Traceback (most recent call last):
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1435, in _allocate_network_async
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     bind_host_id=bind_host_id)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 990, in allocate_for_instance
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     bind_host_id, available_macs, requested_ports_dict)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1123, in _update_ports_for_instance
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     vif.destroy()
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     self.force_reraise()
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1093, in _update_ports_for_instance
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     port_client, instance, port_id, port_req_body)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 471, in _update_port
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     _ensure_no_port_binding_failure(port)
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 216, in _ensure_no_port_binding_failure
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager     raise exception.PortBindingFailed(port_id=port['id'])
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 70774caa-bc74-4088-9add-21e332279ff0, please check neutron logs for more information.
2019-10-10 12:19:51.527 1 ERROR nova.compute.manager
  • /var/log/containers/nova/nova-compute.log:
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [req-375e636e-36f9-410b-b95a-d65eb0be24ad 6afe4e3d4be04288bc741ec8ab782f28 8ef4b147b8a342de9b07b0f760ab1394 - default default] [instance: c4d64fca-1272-49ff-9033-46feaf74312e] Instance failed to spawn: libvirtError: internal error: missing IFLA_VF_INFO in netlink response
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e] Traceback (most recent call last):
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2273, in _build_resources
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     yield resources
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2053, in _build_and_run_instance
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     block_device_info=block_device_info)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3115, in spawn
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     destroy_disks_on_failure=True)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5645, in _create_domain_and_network
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     destroy_disks_on_failure)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     self.force_reraise()
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     six.reraise(self.type_, self.value, self.tb)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5614, in _create_domain_and_network
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     post_xml_callback=post_xml_callback)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5549, in _create_domain
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     guest.launch(pause=pause)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     self._encoded_xml, errors='ignore')
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     self.force_reraise()
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     six.reraise(self.type_, self.value, self.tb)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     return self._domain.createWithFlags(flags)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     rv = execute(f, *args, **kwargs)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     six.reraise(c, e, tb)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     rv = meth(*args, **kwargs)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e] libvirtError: internal error: missing IFLA_VF_INFO in netlink response
2019-10-11 20:29:11.709 1 ERROR nova.compute.manager [instance: c4d64fca-1272-49ff-9033-46feaf74312e] 
  • In /var/log/containers/neutron/neutron-server.log:
2019-10-10 12:19:51.097 1 DEBUG neutron.plugins.ml2.managers [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Attempting to bind port 70774caa-bc74-4088-9add-21e332279ff0 on host overcloud-compute-0 for vnic_type direct with profile {"pci_slot": "0000:13:00.4", "physical_network": "physnet-sriov-vnic9", "pci_vendor_info": "1137:0071"} bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:745
2019-10-10 12:19:51.097 1 DEBUG neutron.plugins.ml2.managers [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Attempting to bind port 70774caa-bc74-4088-9add-21e332279ff0 on host overcloud-compute-0 at level 0 using segments [{'network_id': 'a748cc76-0426-4868-8c0b-aaaa63baaf25', 'segmentation_id': 1603, 'physical_network': u'physnet-sriov-vnic9', 'id': 'bfd87337-5930-49a8-9633-fd955fc16918', 'network_type': u'vlan'}] _bind_port_level /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:766
2019-10-10 12:19:51.098 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.mech_driver.mech_openvswitch [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Refusing to bind due to unsupported vnic_type: direct with no switchdev capability bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/mech_driver/mech_openvswitch.py:90
2019-10-10 12:19:51.098 1 DEBUG networking_cisco.ml2_drivers.ucsm.mech_cisco_ucsm [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Attempting to bind port 70774caa-bc74-4088-9add-21e332279ff0 with vnic_type direct on network a748cc76-0426-4868-8c0b-aaaa63baaf25  bind_port /usr/lib/python2.7/site-packages/networking_cisco/ml2_drivers/ucsm/mech_cisco_ucsm.py:335
2019-10-10 12:19:51.098 1 ERROR neutron.plugins.ml2.managers [req-c9cda82b-ec2d-49d4-a814-cc5f0a459162 8d7a02a0dc4f4f8684bbc68b86001cd7 76927bbe3b1b47f8a222b12c0bdd0fae - default default] Failed to bind port 70774caa-bc74-4088-9add-21e332279ff0 on host overcloud-compute-0 for vnic_type direct using segments [{'network_id': 'a748cc76-0426-4868-8c0b-aaaa63baaf25', 'segmentation_id': 1603, 'physical_network': u'physnet-sriov-vnic9', 'id': 'bfd87337-5930-49a8-9633-fd955fc16918', 'network_type': u'vlan'}]

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content