Hi all,

I have rackmounted Cisco Server available with SR-IOV capable Cisco VIC.

I see that "virtual functions" for SR-IOV capable VIC in output -

lspci -nn | grep Cisco | grep Ethernet

06:00.0 Ethernet controller [0200]: Cisco Systems Inc VIC Ethernet NIC [1137:0043] (rev a2)
07:00.0 Ethernet controller [0200]: Cisco Systems Inc VIC Ethernet NIC [1137:0043] (rev a2)
08:00.0 Ethernet controller [0200]: Cisco Systems Inc VIC Ethernet NIC [1137:0043] (rev a2)
08:00.1 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)
08:00.2 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)
08:00.3 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)
08:00.4 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)
08:00.5 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)
08:00.6 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)
08:00.7 Ethernet controller [0200]: Cisco Systems Inc VIC SR-IOV VF [1137:0071] (rev a2)

Correspondingly, I have done all the settings -

In /etc/nova/nova.conf of compute -

pci_passthrough_whitelist=[{"vendor_id":"1137","product_id":"0071","devname": "enp8s0f0", "physical_network":"physnet2"}]

In /etc/nova/nova.conf of controller -
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,PciPassthroughFilter

In /etc/neutron/plugins/ml2/ml2_conf.ini of controller -

type_drivers = vlan
tenant_network_types = vlan
mechanism_drivers =openvswitch, sriovnicswitch
network_vlan_ranges =physnet1:AA:AA,physnet1:XX:XX,physnet2:AA:BB,physnet2:XX:XX
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

In /etc/neutron/plugins/ml2/ml2_conf_sriov.ini of controller -

supported_pci_vendor_devs = 1137:0071
agent_required = True
physical_device_mappings = physnet2:enp8s0f0

Furthermore both /usr/lib/systemd/system/neutron-server.service and /usr/lib/systemd/system/neutron-sriov-nic-agent.service are modified to use ml2_conf_sriov.ini.
/etc/neutron/plugin.ini is symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini

Machine has been booted with "intel_iommu=pt" option and Intel-VT d is enabled as evident from following output -

rdmsr -f 2:2 0x3a

1

Created a network with " --provider:physical_network physnet2" option, created associated subnet, created a port with "--binding:vnic-type direct" option and launched VM using nova boot to use port created. Gets following error in nova-compute.log -

2015-09-19 13:51:43.491 13599 ERROR nova.compute.manager [req-97577208-37b4-4279-a6f1-cc6fab799243 - - - - -] [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] Instance failed to spawn
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] Traceback (most recent call last):
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2455, in build_resources
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] yield resources
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2327, in _build_and_run_instance
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] block_device_info=block_device_info)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2431, in spawn
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] block_device_info=block_device_info)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4531, in _create_domain_and_network
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] power_on=power_on)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4462, in _create_domain
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] LOG.error(err)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in exit
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] six.reraise(self.type, self.value, self.tb)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4452, in _create_domain
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] domain.createWithFlags(launch_flags)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] rv = execute(f, *args, **kwargs)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] six.reraise(c, e, tb)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] rv = meth(*args, **kwargs)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 996, in createWithFlags
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf] libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices
2015-09-19 13:51:43.491 13599 TRACE nova.compute.manager [instance: ac8eae44-53bf-4046-be20-3e5a96987ebf]