ixgbevf driver enabling in RHEL 7.1
Hi,
I am trying to enable ixgbevf in RHEL 7.1
Procedures followed
1. enabled SR-IOV support in BIOS
2.Intel-Vt enabled in BIOS
3. /etc/modprobe.d/sriov.conf updated with options ixgbe max_vfs=1
Reboot. However after reboot, ixgbevf are not enabled. Whether the kernel enables ixgbevf by defalut ? i couldn't see that
~]# lsmod | grep ixg
ixgbe 290931 0
mdio 13807 1 ixgbe
ptp 18933 2 igb,ixgbe
dca 15130 3 igb,ixgbe,ioatdma
Responses
From my test system
lspci | grep -i 82599
05:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
cat /etc/modprobe.d/sriov.conf
options ixgbe max_vfs=1
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=UUID=93dc91d9-5ebe-4641-b421-ba3cf623f82e ro intel_iommu=on,rd.lvm.lv=rhel_hp-bl460cg8-2/swap vconsole.font=latarcyrheb-sun16 crashkernel=auto vconsole.keymap=us rd.lvm.lv=rhel_hp-bl460cg8-2/root rhgb quiet
lsmod | grep ixgbe
ixgbevf 50849 0
ixgbe 290931 0
From dmesg
[ 3.454407] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k-rh7.1
[ 3.454411] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[ 3.533519] ixgbe 0000:05:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 3.634493] ixgbe 0000:05:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
Ignore the module deprecation warning. VF's should still be enabled
It would be interesting to see the dmesg from your system.
dmesg | grep ixgbe
and
cat /sys/module/ixgbe/version
With RHEL7, you can still enable the VF's via the module parameter. if you review the dmesg that Iposted closely, you will notice the following warning
3.533519] ixgbe 0000:05:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
Yes, after reboot, ixgbevf is loaded and the VF's are initialized in my test system.
Looking at the dmesg that you posted, ixgbe driver has not honoured the parameter setting "options ixgbe max_vfs=1"
We need to verify whether the system has the ixgbe driver in the initramfs image
lsinitrd /boot/initramfs-XXXXX.img | grep ixgbe
If yes then most likely when the driver is loaded ixgbe does not see the module option. Rebuild the initramfs image to include the file /etc/modprobe.d/sriov.conf
dracut -v -f /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img 3.10.0-229.7.2.el7.x86_64 [take a backup of the existing initramfs image]
verify
lsinitrd /tmp/initrd.img etc/modprobe.d/sriov.conf
Reboot the host.. otherwise contact Red Hat Technical support and create a case. Attach the sosreport of the system and let me know the case number.
If you have configured or created /etc/modprobe.d/sriov.conf after system installation, then dracut will not pickup the file.
network drivers are only included in the initramfs image when you install dracut-network.
If the drivers are not included in the initramfs image, then reboot was just enough to pick the configuration
Have you rebooted the box now ?.. does it enable VF's now ?
I will review the case shortly but the above suggestion should help
Can you confirm whether your VF's are initialized after reboot. If yes, we could close this thread
Just adding my comments: Our Technical Support will help you further
->> kernel: ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
okay to notice this message. This is just informational
->> eth6.1601 does not seem to be present, delaying initialization. network: [FAILED]
This seems to be a configuration error and needs correction
Since the initramfs had the ixgbe driver, driver was loaded at a very much earlier stage. After rebuilding the initrd image, now the initrd has the required module parameter options included in it thus enabling Virtual Functions
So now as per dmesg, virtual functions are up and enabled. This is what we are discussing here
Rest of the issues that you are noticing will be supported via the Red Hat Service Request. Our support folks will do a deep dive in to your configuration.
We will continue with the rest of the discussion regarding kickstart
In the post section of kickstart, you can place the sriov.conf in /etc/modprobe.d folder. So When installer creates the dracut image for the kernel, it will source the file /etc/modprobe.d/sriov.conf and include it in the initramfs.
PS: the network drivers are only loaded into initramfs image onlywhen dracut-network is installed
Some notes that might help here, I'm not sure why we would want to use the deprecated way of implementing this. Why not use the PCI Subsystem interface
PCI passthrough to assign a Virtual Function of an SR-IOV capable multiport network card to a virtual machine as a network device
BIOS Settings Needed
SR-IOV Global Enable set to Enabled
Enable Or Disable Through PCI Subsystem Interface
echo 8 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs #enable by 8
echo 0 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs #disable
Validate capability and settings with find and lspci commands
[root@dell-per720-2 ~]# find /sys -name sriov_numvfs
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/sriov_numvfs
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/sriov_numvfs
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
