Adding additional vNICs to compute node breaks DPDK interface assignments with Cisco VIC-1340 and Cisco UCS in Red Hat OpenStack Platform
Issue
Adding additional vNICs to compute node breaks DPDK interface assignments with Cisco VIC-1340 and Cisco UCS in Red Hat OpenStack Platform
This seems to happen when passing from 8 vNics to 9 vNics
Baseline before adding new vNic
[root@overcloud-compute-0 ~]# /usr/sbin/dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
0000:0e:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
0000:0f:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
Network devices using kernel driver
===================================
0000:06:00.0 'VIC Ethernet NIC' if=eth0 drv=enic unused=vfio-pci *Active*
0000:07:00.0 'VIC Ethernet NIC' if=eth1 drv=enic unused=vfio-pci
0000:08:00.0 'VIC Ethernet NIC' if=eth2 drv=enic unused=vfio-pci *Active*
0000:09:00.0 'VIC Ethernet NIC' if=eth3 drv=enic unused=vfio-pci *Active*
0000:10:00.0 'VIC Ethernet NIC' if=eth6 drv=enic unused=vfio-pci
0000:11:00.0 'VIC Ethernet NIC' if=eth7 drv=enic unused=vfio-pci
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 vfio-pci
0000:0f:00.0 vfio-pci
[root@overcloud-compute-0 ~]# lspci -tv | grep VIC -C3
| \-1f.2 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
+-01.0-[01]----00.0 LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]
+-02.0-[02-09]----00.0-[03-09]----00.0-[04-09]----00.0-[05-09]--+-00.0-[06]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[07]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[08]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-03.0-[09]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-02.2-[0a-11]----00.0-[0b-11]----00.0-[0c-11]----00.0-[0d-11]--+-00.0-[0e]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[0f]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[10]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-03.0-[11]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-03.0-[12]--
+-05.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management
+-05.1 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug
From lspci:
0e:00.0 Ethernet controller [0200]: Cisco Systems Inc VIC Ethernet NIC [1137:0043] (rev a2)
Subsystem: Cisco Systems Inc VIC 1340 MLOM Ethernet NIC [1137:012c]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 34
NUMA node: 0
Region 0: Memory at c6c00000 (64-bit, non-prefetchable) [size=32K]
Region 2: Memory at c6c08000 (64-bit, non-prefetchable) [size=8K]
Capabilities: [40] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] MSI-X: Enable+ Count=20 Masked-
Vector table: BAR=2 offset=00000000
PBA: BAR=2 offset=00001000
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-25-b5-ff-ff-aa-aa-aa
Capabilities: [120 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: vfio-pci
Kernel modules: enic
After adding additional vNic
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 (none)
0000:0f:00.0 vfio-pci
[root@overcloud-compute-0 ~]# lspci -tv | grep VIC -C3
| \-1f.2 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
+-01.0-[01]----00.0 LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]
+-02.0-[02-0a]----00.0-[03-0a]----00.0-[04-0a]----00.0-[05-0a]--+-00.0-[06]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[07]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[08]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-03.0-[09]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-04.0-[0a]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-02.2-[0b-12]----00.0-[0c-12]----00.0-[0d-12]----00.0-[0e-12]--+-00.0-[0f]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-01.0-[10]----00.0 Cisco Systems Inc VIC Ethernet NIC
| +-02.0-[11]----00.0 Cisco Systems Inc VIC Ethernet NIC
| \-03.0-[12]----00.0 Cisco Systems Inc VIC Ethernet NIC
+-03.0-[13]--
+-05.0 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management
+-05.1 Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug
[root@overcloud-compute-0 ~]# /usr/sbin/dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
0000:0f:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
Network devices using kernel driver
===================================
0000:06:00.0 'VIC Ethernet NIC' if=eth0 drv=enic unused=vfio-pci *Active*
0000:07:00.0 'VIC Ethernet NIC' if=eth1 drv=enic unused=vfio-pci
0000:08:00.0 'VIC Ethernet NIC' if=eth2 drv=enic unused=vfio-pci *Active*
0000:09:00.0 'VIC Ethernet NIC' if=eth3 drv=enic unused=vfio-pci *Active*
0000:0a:00.0 'VIC Ethernet NIC' if=eth4 drv=enic unused=vfio-pci
0000:10:00.0 'VIC Ethernet NIC' if=eth6 drv=enic unused=vfio-pci
0000:11:00.0 'VIC Ethernet NIC' if=eth7 drv=enic unused=vfio-pci
0000:12:00.0 'VIC Ethernet NIC' if=eth8 drv=enic unused=vfio-pci
Other network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other crypto devices
====================
<none>
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 (none)
0000:0f:00.0 vfio-pci
Indeed, in the non-working case, 0000:0e:00.0 is not an eNic:
0e:00.0 PCI bridge [0604]: Cisco Systems Inc VIC PCIe Downstream Port [1137:0041] (rev a2) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
NUMA node: 0
Bus: primary=0e, secondary=0f, subordinate=0f, sec-latency=0
Memory behind bridge: c7200000-c72fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] Express (v2) Downstream Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [a0] Subsystem: Cisco Systems Inc Device [1137:012c]
Capabilities: [100 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
Kernel modules: shpchp
Environment
Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
