Unable to enable SR-IOV and receiving the message "not enough MMIO resources for SR-IOV" in Red Hat Enterprise Linux

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (All Versions)
  • SRIOV

Issue

  • Found following error messages in /var/log/dmesg when loading the igb driver with max_vfs=4

        kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2
    kernel: Copyright (c) 2007-2009 Intel Corporation.
    kernel: igb 0000:01:00.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
    kernel: igb 0000:01:00.0: not enough MMIO resources for SR-IOV

  • Another error with i40e

        i40e 0000:01:00.0: not enough MMIO resources for SR-IOV
    i40e 0000:01:00.0: Failed to enable SR-IOV: -12

  • Another error with ice

        ice 0000:17:00.0: not enough MMIO resources for SR-IOV
    ice 0000:17:00.0: Failed to enable SR-IOV: -12

  • Another error with bnxt_en

        kernel: bnxt_en 0000:4b:00.0: not enough MMIO resources for SR-IOV
    kernel: bnxt_en 0000:4b:00.1: not enough MMIO resources for SR-IOV
    kernel: bnxt_en 0000:4b:00.1 eno12409np1: Requested VFs 8, can enable 1
    kernel: bnxt_en 0000:4b:00.1: not enough MMIO resources for SR-IOV
    kernel: bnxt_en 0000:4b:00.1: 8 VFs requested; only 1 enabled
    kernel: bnxt_en 0000:4b:00.1 eno12409np1: Cannot enable VF's as all resources are used by PF

Resolution

  • The BIOS is not providing enough MMIO space for VFs. Contact your hardware vendor for firmware or BIOS update.

  • As a workaround solution, one can pass "pci=realloc" to kernel 2.6.32-228.el6 during booting.

  • On Dell PowerEdge R750 with RHEL 8.7 you can apply the following changes to resolve the issue :

    iDRAC -> Configuration -> BIOS Settings -> Integrated Devices -> SR-IOV Global Enable = ENABLED

Root Cause

  • PCI code can not re-allocate enough MMIO due to a limitation or a bug with the BIOS. RHEL's SR-IOV support is such that it must have enough resources to map all possible VFs or all VF MMIO space allocation fails.

  • Seen for other NIC models also. The error is trggered from kernel sriov_enable function:

    // From drivers/pci/iov.c

static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
{
...
    if (nres != iov->nres) {
       pci_err(dev, "not enough MMIO resources for SR-IOV\n");
       return -ENOMEM;
   }

Diagnostic Steps

  • Enable VT-D and Intr-Remap at the BIOS level.
  • Append the following parameters "intel_iommu=on pci_pt_e820_access=on pci=assign-busses" to the kernel line in grub,conf,
    3 , If it is a xen kernel, append iommu=1 msi=1 to xen kernel line (also include the above parameters to the kernel line),
    4,options igb max_vfs=XX in /etc/modprobe,conf,

    Reboot the system after enabling the above kernel options

    If SR-IOV is enabled successfully, you would notice the following

    lspci would list

03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
04:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
04:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
04:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

dmesg looks

igb 0000:04:00.0: not enough MMIO resources for SR-IOV
igb 0000:03:00,0: 5 vfs allocated

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments