Adding additional vNICs to compute node breaks DPDK interface assignments with Cisco VIC-1340 and Cisco UCS in Red Hat OpenStack Platform
Issue
Adding additional vNICs to compute node breaks DPDK interface assignments with Cisco VIC-1340 and Cisco UCS in Red Hat OpenStack Platform
This seems to happen when passing from 8 vNics to 9 vNics
Baseline before adding new vNic
[root@overcloud-compute-0 ~]# /usr/sbin/dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
0000:0e:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
0000:0f:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
Network devices using kernel driver
===================================
0000:06:00.0 'VIC Ethernet NIC' if=eth0 drv=enic unused=vfio-pci *Active*
0000:07:00.0 'VIC Ethernet NIC' if=eth1 drv=enic unused=vfio-pci
0000:08:00.0 'VIC Ethernet NIC' if=eth2 drv=enic unused=vfio-pci *Active*
0000:09:00.0 'VIC Ethernet NIC' if=eth3 drv=enic unused=vfio-pci *Active*
0000:10:00.0 'VIC Ethernet NIC' if=eth6 drv=enic unused=vfio-pci
0000:11:00.0 'VIC Ethernet NIC' if=eth7 drv=enic unused=vfio-pci
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 vfio-pci
0000:0f:00.0 vfio-pci
[root@overcloud-compute-0 ~]# lspci -tv | grep VIC -C3
| \-1f.2 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
+-01.0-[01]----00.0 LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]
+-02.0-[02-09]----00.0-[03-09]----00.0-[04-09]----00.0-[05-09]--+-00.0-[06]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[07]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[08]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-03.0-[09]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-02.2-[0a-11]----00.0-[0b-11]----00.0-[0c-11]----00.0-[0d-11]--+-00.0-[0e]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[0f]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[10]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-03.0-[11]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-03.0-[12]--
+-05.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management
+-05.1 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug
From lspci:
0e:00.0 Ethernet controller [0200]: Cisco Systems Inc VIC Ethernet NIC [1137:0043] (rev a2)
Subsystem: Cisco Systems Inc VIC 1340 MLOM Ethernet NIC [1137:012c]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 34
NUMA node: 0
Region 0: Memory at c6c00000 (64-bit, non-prefetchable) [size=32K]
Region 2: Memory at c6c08000 (64-bit, non-prefetchable) [size=8K]
Capabilities: [40] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] MSI-X: Enable+ Count=20 Masked-
Vector table: BAR=2 offset=00000000
PBA: BAR=2 offset=00001000
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-25-b5-ff-ff-aa-aa-aa
Capabilities: [120 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: vfio-pci
Kernel modules: enic
After adding additional vNic
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 (none)
0000:0f:00.0 vfio-pci
[root@overcloud-compute-0 ~]# lspci -tv | grep VIC -C3
| \-1f.2 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
+-01.0-[01]----00.0 LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]
+-02.0-[02-0a]----00.0-[03-0a]----00.0-[04-0a]----00.0-[05-0a]--+-00.0-[06]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[07]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[08]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-03.0-[09]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-04.0-[0a]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-02.2-[0b-12]----00.0-[0c-12]----00.0-[0d-12]----00.0-[0e-12]--+-00.0-[0f]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[10]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[11]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-03.0-[12]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-03.0-[13]--
+-05.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management
+-05.1 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug
[root@overcloud-compute-0 ~]# /usr/sbin/dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
0000:0f:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
Network devices using kernel driver
===================================
0000:06:00.0 'VIC Ethernet NIC' if=eth0 drv=enic unused=vfio-pci *Active*
0000:07:00.0 'VIC Ethernet NIC' if=eth1 drv=enic unused=vfio-pci
0000:08:00.0 'VIC Ethernet NIC' if=eth2 drv=enic unused=vfio-pci *Active*
0000:09:00.0 'VIC Ethernet NIC' if=eth3 drv=enic unused=vfio-pci *Active*
0000:0a:00.0 'VIC Ethernet NIC' if=eth4 drv=enic unused=vfio-pci
0000:10:00.0 'VIC Ethernet NIC' if=eth6 drv=enic unused=vfio-pci
0000:11:00.0 'VIC Ethernet NIC' if=eth7 drv=enic unused=vfio-pci
0000:12:00.0 'VIC Ethernet NIC' if=eth8 drv=enic unused=vfio-pci
Other network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other crypto devices
====================
<none>
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 (none)
0000:0f:00.0 vfio-pci
Indeed, in the non-working case, 0000:0e:00.0 is not an eNic:
0e:00.0 PCI bridge [0604]: Cisco Systems Inc VIC PCIe Downstream Port [1137:0041] (rev a2) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
NUMA node: 0
Bus: primary=0e, secondary=0f, subordinate=0f, sec-latency=0
Memory behind bridge: c7200000-c72fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] Express (v2) Downstream Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [a0] Subsystem: Cisco Systems Inc Device [1137:012c]
Capabilities: [100 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
Kernel modules: shpchp
Environment
Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Close
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.