RHEL: some NIC are not recognised in Multiple NIC when X2APIC is disabled from BIOS

Solution Verified - Updated -

Issue

  • Two NICs are not recognised on CPU8 or lower in Multiple NIC with following logs when X2APIC is disabled from BIOS.
[    0.064947] IRQ remapping doesn't support X2APIC mode, disable x2apic.
[    0.065052] Switched APIC routing to physical flat.
..............
[    5.150243] (0000:b0:00.0): E-Switch: Total vports 9, per vport: max uc(1024) max mc(16384)
[    5.160658] mlx5_core 0000:b0:00.0: irq 210 for MSI/MSI-X
[    5.160670] mlx5_core 0000:b0:00.0: alloc irq vectors failed
[    5.160691] (0000:b0:00.0): E-Switch: cleanup
[    5.864591] mlx5_core 0000:b0:00.0: mlx5_load_one failed with error code -28
[    5.864716] mlx5_core: probe of 0000:b0:00.0 failed with error -28
[    5.864945] mlx5_core 0000:b0:00.1: firmware version: 14.22.1414
[    6.768804] (0000:b0:00.1): E-Switch: Total vports 9, per vport: max uc(1024) max mc(16384)
[    6.775205] mlx5_core 0000:b0:00.1: alloc irq vectors failed
[    6.775211] (0000:b0:00.1): E-Switch: cleanup
[    7.471940] mlx5_core 0000:b0:00.1: mlx5_load_one failed with error code -28
[    7.472038] mlx5_core: probe of 0000:b0:00.1 failed with error -28
  • It should be visible like this:
  Interface Status:
    eno1    0000:02:00.0  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno2    0000:02:00.1  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno3    0000:02:00.2  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno4    0000:02:00.3  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    ens1f0  0000:12:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens1f1  0000:12:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens3f0  0000:13:00.0  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens3f1  0000:13:00.1  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens4f0  0000:af:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens4f1  0000:af:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens6f0  0000:b0:00.0  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens6f1  0000:b0:00.1  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens8f0  0000:d9:00.0  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f1  0000:d9:00.1  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f2  0000:d9:00.2  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f3  0000:d9:00.3  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
  • But two NIC are missing.
  Interface Status:
    eno1    0000:02:00.0  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno2    0000:02:00.1  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno3    0000:02:00.2  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno4    0000:02:00.3  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    ens1f0  0000:12:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens1f1  0000:12:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens3f0  0000:13:00.0  link=DOWN  rx ring 1024/8192  drv mlx5_core v5.0-0 / fw 14.22.1414 (HP_2420110034)
    ens3f1  0000:13:00.1  link=DOWN  rx ring 1024/8192  drv mlx5_core v5.0-0 / fw 14.22.1414 (HP_2420110034)
    ens4f0  0000:af:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens4f1  0000:af:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens8f0  0000:d9:00.0  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f1  0000:d9:00.1  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f2  0000:d9:00.2  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f3  0000:d9:00.3  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0

Environment

  • Red Hat Enterprise Linux
  • X2APIC disabled from BIOS
  • CPU 8 or lower.
  • Multiple NIC

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content