Adding additional vNICs to compute node breaks DPDK interface assignments with Cisco VIC-1340 and Cisco UCS in Red Hat OpenStack Platform

Solution In Progress - Updated -

Issue

Adding additional vNICs to compute node breaks DPDK interface assignments with Cisco VIC-1340 and Cisco UCS in Red Hat OpenStack Platform

This seems to happen when passing from 8 vNics to 9 vNics

Baseline before adding new vNic

[root@overcloud-compute-0 ~]# /usr/sbin/dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:0e:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic
0000:0f:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic

Network devices using kernel driver
===================================
0000:06:00.0 'VIC Ethernet NIC' if=eth0 drv=enic unused=vfio-pci *Active*
0000:07:00.0 'VIC Ethernet NIC' if=eth1 drv=enic unused=vfio-pci
0000:08:00.0 'VIC Ethernet NIC' if=eth2 drv=enic unused=vfio-pci *Active*
0000:09:00.0 'VIC Ethernet NIC' if=eth3 drv=enic unused=vfio-pci *Active*
0000:10:00.0 'VIC Ethernet NIC' if=eth6 drv=enic unused=vfio-pci
0000:11:00.0 'VIC Ethernet NIC' if=eth7 drv=enic unused=vfio-pci
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 vfio-pci
0000:0f:00.0 vfio-pci
[root@overcloud-compute-0 ~]# lspci -tv | grep VIC -C3
 |           \-1f.2  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit
 \-[0000:00]-+-00.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
             +-01.0-[01]----00.0  LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]
             +-02.0-[02-09]----00.0-[03-09]----00.0-[04-09]----00.0-[05-09]--+-00.0-[06]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-01.0-[07]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-02.0-[08]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               \-03.0-[09]----00.0  Cisco Systems Inc VIC Ethernet NIC
             +-02.2-[0a-11]----00.0-[0b-11]----00.0-[0c-11]----00.0-[0d-11]--+-00.0-[0e]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-01.0-[0f]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-02.0-[10]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               \-03.0-[11]----00.0  Cisco Systems Inc VIC Ethernet NIC
             +-03.0-[12]--
             +-05.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management
             +-05.1  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug

From lspci:

0e:00.0 Ethernet controller [0200]: Cisco Systems Inc VIC Ethernet NIC [1137:0043] (rev a2)
        Subsystem: Cisco Systems Inc VIC 1340 MLOM Ethernet NIC [1137:012c]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 34
        NUMA node: 0
        Region 0: Memory at c6c00000 (64-bit, non-prefetchable) [size=32K]
        Region 2: Memory at c6c08000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [40] MSI: Enable- Count=1/16 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] MSI-X: Enable+ Count=20 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00001000
        Capabilities: [80] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-25-b5-ff-ff-aa-aa-aa
        Capabilities: [120 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Kernel driver in use: vfio-pci
        Kernel modules: enic

After adding additional vNic

[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 (none)
0000:0f:00.0 vfio-pci
[root@overcloud-compute-0 ~]# lspci -tv | grep VIC -C3
 |           \-1f.2  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit
 \-[0000:00]-+-00.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
             +-01.0-[01]----00.0  LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]
             +-02.0-[02-0a]----00.0-[03-0a]----00.0-[04-0a]----00.0-[05-0a]--+-00.0-[06]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-01.0-[07]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-02.0-[08]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-03.0-[09]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               \-04.0-[0a]----00.0  Cisco Systems Inc VIC Ethernet NIC
             +-02.2-[0b-12]----00.0-[0c-12]----00.0-[0d-12]----00.0-[0e-12]--+-00.0-[0f]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-01.0-[10]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               +-02.0-[11]----00.0  Cisco Systems Inc VIC Ethernet NIC
             |                                                               \-03.0-[12]----00.0  Cisco Systems Inc VIC Ethernet NIC
             +-03.0-[13]--
             +-05.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management
             +-05.1  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug
[root@overcloud-compute-0 ~]# /usr/sbin/dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:0f:00.0 'VIC Ethernet NIC' drv=vfio-pci unused=enic

Network devices using kernel driver
===================================
0000:06:00.0 'VIC Ethernet NIC' if=eth0 drv=enic unused=vfio-pci *Active*
0000:07:00.0 'VIC Ethernet NIC' if=eth1 drv=enic unused=vfio-pci
0000:08:00.0 'VIC Ethernet NIC' if=eth2 drv=enic unused=vfio-pci *Active*
0000:09:00.0 'VIC Ethernet NIC' if=eth3 drv=enic unused=vfio-pci *Active*
0000:0a:00.0 'VIC Ethernet NIC' if=eth4 drv=enic unused=vfio-pci
0000:10:00.0 'VIC Ethernet NIC' if=eth6 drv=enic unused=vfio-pci
0000:11:00.0 'VIC Ethernet NIC' if=eth7 drv=enic unused=vfio-pci
0000:12:00.0 'VIC Ethernet NIC' if=eth8 drv=enic unused=vfio-pci

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
[root@overcloud-compute-0 ~]# driverctl list-overrides
0000:0e:00.0 (none)
0000:0f:00.0 vfio-pci

Indeed, in the non-working case, 0000:0e:00.0 is not an eNic:

0e:00.0 PCI bridge [0604]: Cisco Systems Inc VIC PCIe Downstream Port [1137:0041] (rev a2) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        NUMA node: 0
        Bus: primary=0e, secondary=0f, subordinate=0f, sec-latency=0
        Memory behind bridge: c7200000-c72fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [60] Express (v2) Downstream Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] Subsystem: Cisco Systems Inc Device [1137:012c]
        Capabilities: [100 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
        Kernel modules: shpchp

Environment

Red Hat OpenStack Platform 10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In