RHEL: some NIC are not recognised in Multiple NIC when X2APIC is disabled from BIOS

Issue

• Two NICs are not recognised on CPU8 or lower in Multiple NIC with following logs when X2APIC is disabled from BIOS.

[    0.064947] IRQ remapping doesn't support X2APIC mode, disable x2apic.
[    0.065052] Switched APIC routing to physical flat.
..............
[    5.150243] (0000:b0:00.0): E-Switch: Total vports 9, per vport: max uc(1024) max mc(16384)
[    5.160658] mlx5_core 0000:b0:00.0: irq 210 for MSI/MSI-X
[    5.160670] mlx5_core 0000:b0:00.0: alloc irq vectors failed
[    5.160691] (0000:b0:00.0): E-Switch: cleanup
[    5.864591] mlx5_core 0000:b0:00.0: mlx5_load_one failed with error code -28
[    5.864716] mlx5_core: probe of 0000:b0:00.0 failed with error -28
[    5.864945] mlx5_core 0000:b0:00.1: firmware version: 14.22.1414
[    6.768804] (0000:b0:00.1): E-Switch: Total vports 9, per vport: max uc(1024) max mc(16384)
[    6.775205] mlx5_core 0000:b0:00.1: alloc irq vectors failed
[    6.775211] (0000:b0:00.1): E-Switch: cleanup
[    7.471940] mlx5_core 0000:b0:00.1: mlx5_load_one failed with error code -28
[    7.472038] mlx5_core: probe of 0000:b0:00.1 failed with error -28

• It should be visible like this:

  Interface Status:
    eno1    0000:02:00.0  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno2    0000:02:00.1  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno3    0000:02:00.2  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno4    0000:02:00.3  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    ens1f0  0000:12:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens1f1  0000:12:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens3f0  0000:13:00.0  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens3f1  0000:13:00.1  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens4f0  0000:af:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens4f1  0000:af:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens6f0  0000:b0:00.0  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens6f1  0000:b0:00.1  link=DOWN  rx ring 1024/8192  drv mlx5_core v4.7-1.0.0 / fw 14.26.1040 (HP_2420110034)
    ens8f0  0000:d9:00.0  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f1  0000:d9:00.1  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f2  0000:d9:00.2  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f3  0000:d9:00.3  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0

• But two NIC are missing.

  Interface Status:
    eno1    0000:02:00.0  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno2    0000:02:00.1  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno3    0000:02:00.2  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    eno4    0000:02:00.3  link=DOWN  rx ring 200/2047   drv tg3 v3.137 / fw 5719-v1.46 NCSI v1.5.1.0
    ens1f0  0000:12:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens1f1  0000:12:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens3f0  0000:13:00.0  link=DOWN  rx ring 1024/8192  drv mlx5_core v5.0-0 / fw 14.22.1414 (HP_2420110034)
    ens3f1  0000:13:00.1  link=DOWN  rx ring 1024/8192  drv mlx5_core v5.0-0 / fw 14.22.1414 (HP_2420110034)
    ens4f0  0000:af:00.0  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens4f1  0000:af:00.1  link=DOWN  rx ring 512/4096   drv i40e v2.3.2-k / fw 10.4.3
    ens8f0  0000:d9:00.0  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f1  0000:d9:00.1  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f2  0000:d9:00.2  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0
    ens8f3  0000:d9:00.3  link=DOWN  rx ring 256/4096   drv igb v5.4.0-k / fw 1.52, 0x80000e81, 1.2028.0