Why we are unable to enable SR-IOV and receiving the following message "not enough MMIO resources for SR-IOV" in Red Hat Enterprise Linux

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6
  • Kernel 2.6.32-37.el6.x86_64
  • kernel-xen-2.6.18-274.3.1
  • Intel Corporation 82576 Gigabit Network Connection
  • Hardware: HP DL XXX G X

Issue

  • Found following error messages in /var/log/dmesg when loading the igb driver with max_vfs=4

        kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2
    kernel: Copyright (c) 2007-2009 Intel Corporation.
    kernel: igb 0000:01:00.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
    kernel: igb 0000:01:00.0: not enough MMIO resources for SR-IOV

Resolution

  • The BIOS is not providing enough MMIO space for VFs. Contact your hardware vendor for firmware or bios update.
  • As a workaround solution, one can pass "pci=realloc" to kernel 2.6.32-228.el6 during booting.

Root Cause

  • PCI code can not re-allocate enough MMIO due to a limitation or a bug with the BIOS. RHEL's SR-IOV support is such that it must have enough resources to map all possible VFs or all VF MMIO space allocation fails.

Diagnostic Steps

  • Enable VT-D and Intr-Remap at the BIOS level.
  • Append the following parameters "intel_iommu=on pci_pt_e820_access=on pci=assign-busses" to the kernel line in grub,conf,
    3 , If it is a xen kernel, append iommu=1 msi=1 to xen kernel line (also include the above parameters to the kernel line),
    4,options igb max_vfs=XX in /etc/modprobe,conf,

    Reboot the system after enabling the above kernel options

    If SR-IOV is enabled successfully, you would notice the following

    lspci would list

03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
04:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
04:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
04:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

dmesg looks

igb 0000:04:00.0: not enough MMIO resources for SR-IOV
igb 0000:03:00,0: 5 vfs allocated

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.