ixgbevf driver enabling in RHEL 7.1

Latest response

Hi,

I am trying to enable ixgbevf in RHEL 7.1
Procedures followed
1. enabled SR-IOV support in BIOS
2.Intel-Vt enabled in BIOS
3. /etc/modprobe.d/sriov.conf updated with options ixgbe max_vfs=1

Reboot. However after reboot, ixgbevf are not enabled. Whether the kernel enables ixgbevf by defalut ? i couldn't see that

~]# lsmod | grep ixg
ixgbe 290931 0
mdio 13807 1 ixgbe
ptp 18933 2 igb,ixgbe
dca 15130 3 igb,ixgbe,ioatdma

Responses

Looks like you haven't activated the VT-D in kernel

Can you append the following " intel_iommu=on" to kernel parameters in /etc/sysconfig/grub for GRUB_CMDLINE_LINUX

Regenerate /boot/grub2/grub.cfg by running:
grub2-mkconfig -o /boot/grub2/grub.cfg

reboot the system

Yes it is set.

]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-229.7.2.el7.x86_64 root=UUID=e0ab1af4-dbe7-449b-a0ec-ce531b933eef ro crashkernel=auto console=ttyS0,38400 intel_iommu=on net.ifnames=0 biosdevname=0 rhgb quiet LANG=en_US.UTF-8 systemd.debug

From my test system

lspci | grep -i 82599

05:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

cat /etc/modprobe.d/sriov.conf

options ixgbe max_vfs=1

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=UUID=93dc91d9-5ebe-4641-b421-ba3cf623f82e ro intel_iommu=on,rd.lvm.lv=rhel_hp-bl460cg8-2/swap vconsole.font=latarcyrheb-sun16 crashkernel=auto vconsole.keymap=us rd.lvm.lv=rhel_hp-bl460cg8-2/root rhgb quiet

lsmod | grep ixgbe

ixgbevf 50849 0
ixgbe 290931 0

From dmesg

[ 3.454407] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k-rh7.1
[ 3.454411] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[ 3.533519] ixgbe 0000:05:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 3.634493] ixgbe 0000:05:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs

Ignore the module deprecation warning. VF's should still be enabled

It would be interesting to see the dmesg from your system.

dmesg | grep ixgbe
and
cat /sys/module/ixgbe/version

~]# dmesg | grep ixgbe
[ 8.437311] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k-rh7.1
[ 8.437313] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[ 8.525526] ixgbe 0000:04:00.0: irq 71 for MSI/MSI-X
[ 8.525537] ixgbe 0000:04:00.0: irq 72 for MSI/MSI-X
[ 8.525546] ixgbe 0000:04:00.0: irq 73 for MSI/MSI-X
[ 8.525555] ixgbe 0000:04:00.0: irq 74 for MSI/MSI-X
[ 8.525564] ixgbe 0000:04:00.0: irq 75 for MSI/MSI-X
[ 8.525574] ixgbe 0000:04:00.0: irq 76 for MSI/MSI-X
[ 8.525582] ixgbe 0000:04:00.0: irq 77 for MSI/MSI-X
[ 8.525603] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 6, Tx Queue count = 6
[ 8.525739] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:a0:a5:78:04:44
[ 8.525825] ixgbe 0000:04:00.0: MAC: 2, PHY: 1, PBA No: 000301-00B
[ 8.525829] ixgbe 0000:04:00.0: PCI Express bandwidth of 32GT/s available
[ 8.525831] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 8.622332] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[ 8.711469] ixgbe 0000:04:00.1: irq 78 for MSI/MSI-X
[ 8.711480] ixgbe 0000:04:00.1: irq 79 for MSI/MSI-X
[ 8.711489] ixgbe 0000:04:00.1: irq 80 for MSI/MSI-X
[ 8.711499] ixgbe 0000:04:00.1: irq 81 for MSI/MSI-X
[ 8.711508] ixgbe 0000:04:00.1: irq 82 for MSI/MSI-X
[ 8.711517] ixgbe 0000:04:00.1: irq 83 for MSI/MSI-X
[ 8.711529] ixgbe 0000:04:00.1: irq 84 for MSI/MSI-X
[ 8.711551] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 6, Tx Queue count = 6
[ 8.711681] ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) 00:a0:a5:78:04:45
[ 8.711769] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: 000301-00B
[ 8.711772] ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s available
[ 8.711774] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 8.745846] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[ 56.618549] ixgbe 0000:04:00.0: registered PHC device on eth0
[ 56.619626] ixgbe 0000:04:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 56.724288] ixgbe 0000:04:00.1: registered PHC device on eth1
[ 56.724777] ixgbe 0000:04:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

~]# cat /sys/module/ixgbe/version
4.0.1-k-rh7.1
]#

Dear Ranjith,

interested to know, after reboot, does the rule persist ?Also lsmod|grep ixgbe doesn't show ixgebvf

~]# lsmod |grep ixgbe
ixgbe 290931 0
mdio 13807 1 ixgbe
ptp 18933 2 igb,ixgbe
dca 15130 3 igb,ixgbe,ioatdma
~]#

Checking on this further found the below thread. Could you please share expert comments on this. I am trying this option will share the results.

http://www.spinics.net/lists/linux-pci/msg18300.html

With RHEL7, you can still enable the VF's via the module parameter. if you review the dmesg that Iposted closely, you will notice the following warning

3.533519] ixgbe 0000:05:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.

Yes, after reboot, ixgbevf is loaded and the VF's are initialized in my test system.

Looking at the dmesg that you posted, ixgbe driver has not honoured the parameter setting "options ixgbe max_vfs=1"

We need to verify whether the system has the ixgbe driver in the initramfs image

lsinitrd /boot/initramfs-XXXXX.img | grep ixgbe

If yes then most likely when the driver is loaded ixgbe does not see the module option. Rebuild the initramfs image to include the file /etc/modprobe.d/sriov.conf

dracut -v -f /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img 3.10.0-229.7.2.el7.x86_64 [take a backup of the existing initramfs image]

verify

lsinitrd /tmp/initrd.img etc/modprobe.d/sriov.conf

Reboot the host.. otherwise contact Red Hat Technical support and create a case. Attach the sosreport of the system and let me know the case number.

Thanks for your analysis and details.

case is already on Please find details.
https://access.redhat.com/support/cases/#/case/01465581

~]# lsinitrd /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img |grep ixgbe
drwxr-xr-x 2 root root 0 Jul 3 09:23 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbe
-rw-r--r-- 1 root root 449769 May 16 07:39 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

~]# dracut -v -f /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img 3.10.0-229.7.2.el7.x86_64
Executing: /usr/sbin/dracut -v -f /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img 3.10.0-229.7.2.el7.x86_64
*** Including module: bash ***
*** Including module: nss-softokn ***
*** Including module: i18n ***
*** Including module: network ***
*** Including module: ifcfg ***
*** Including module: drm ***
*** Including module: plymouth ***
*** Including module: kernel-modules ***
*** Including module: resume ***
*** Including module: rootfs-block ***
*** Including module: terminfo ***
*** Including module: udev-rules ***
Skipping udev rule: 91-permissions.rules
*** Including module: biosdevname ***
*** Including module: systemd ***
*** Including module: usrmount ***
*** Including module: base ***
*** Including module: fs-lib ***
*** Including module: shutdown ***
*** Including modules done ***
*** Installing kernel module dependencies and firmware ***
*** Installing kernel module dependencies and firmware done ***
*** Resolving executable dependencies ***
*** Resolving executable dependencies done***
*** Hardlinking files ***
*** Hardlinking files done ***
*** Stripping files ***
*** Stripping files done ***
*** Generating early-microcode cpio image ***
*** Constructing GenuineIntel.bin ****
*** Store current command line parameters ***
*** Creating image file ***
*** Creating image file done ***

~]# lsinitrd /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img /etc/modprobe.d/sriov.conf
options ixgbe max_vfs=7

~]# lsinitrd /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img |grep ixgbe
drwxr-xr-x 2 root root 0 Jul 6 11:41 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbe
-rw-r--r-- 1 root root 449769 May 16 07:39 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
drwxr-xr-x 2 root root 0 Jul 6 11:41 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbevf
-rw-r--r-- 1 root root 80329 May 16 07:39 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko

But interested to know, why is it not taken in first place ? i asked the same question in RHEL support case. is it becuase bios setting was done after installation ? appreciate your valuable inputs

If you have configured or created /etc/modprobe.d/sriov.conf after system installation, then dracut will not pickup the file.

network drivers are only included in the initramfs image when you install dracut-network.

If the drivers are not included in the initramfs image, then reboot was just enough to pick the configuration

Have you rebooted the box now ?.. does it enable VF's now ?

I will review the case shortly but the above suggestion should help

After reboot below error is thrown
eth6.1601 does not seem to be present, delaying initialization.
network: [FAILED]
kernel: ixgbevf: Unable to start - perhaps the
PF Driver isn't up yet
And after reboot unable to access Network as well

Can you confirm whether your VF's are initialized after reboot. If yes, we could close this thread

Just adding my comments: Our Technical Support will help you further

->> kernel: ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

okay to notice this message. This is just informational

->> eth6.1601 does not seem to be present, delaying initialization. network: [FAILED]

This seems to be a configuration error and needs correction

yes sriov.conf is created after system installation which is as suggested by Redhat Doc to make it permanent ! there aren't any procedures given.
After reboot VF interfaces are not enabled .

Jul 6 11:52:40 kernel: ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
Jul 6 11:52:40 network: Bringing up interface eth6.1601: RTNETLINK answers: Network is down
Jul 6 11:52:40 systemd-sysctl: Overwriting earlier assignment of kernel/shmmax in file '/etc/sysctl.d/99-sysctl.conf'.
Jul 6 11:52:40 network: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device eth6.1601 does not seem to be present, delaying initialization.
Jul 6 11:52:40 /etc/sysconfig/network-scripts/ifup-eth: Device eth6.1601 does not seem to be present, delaying initialization.
Jul 6 11:52:40 : [FAILED]
Jul 6 11:52:40 kernel: ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

cat /etc/sysconfig/network-scripts/ifcfg-eth6.1601
VLAN=yes
DEVICE=eth6.1601
ONBOOT=yes
USERCTL=yes
BOOTPROTO=none
NETMASK=255.255.255.240
IPADDR=192.150.2.19
PREFIX=28

~]# cat /etc/sysconfig/network-scripts/ifcfg-eth7.2601
VLAN=yes
DEVICE=eth7.2601
ONBOOT=yes
USERCTL=yes
BOOTPROTO=none
NETMASK=255.255.255.240
IPADDR=192.150.3.19
PREFIX=28

Let's look at the dmesg again

dmesg | grep ixgbe
lspci | grep -i [example lspci | grep 82599]

[ 8.430590] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k-rh7.1
[ 8.430592] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[ 8.494904] ixgbe 0000:04:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 8.596786] ixgbe 0000:04:00.0 (unregistered net_device): SR-IOV enabled with 7 VFs
[ 8.598264] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[ 8.598266] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 8.621309] ixgbe 0000:04:00.0: irq 71 for MSI/MSI-X
[ 8.621319] ixgbe 0000:04:00.0: irq 72 for MSI/MSI-X
[ 8.621331] ixgbe 0000:04:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 8.621461] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:a0:a5:78:04:44
[ 8.621546] ixgbe 0000:04:00.0: MAC: 2, PHY: 1, PBA No: 000301-00B
[ 8.621551] ixgbe 0000:04:00.0: PCI Express bandwidth of 32GT/s available
[ 8.621553] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 8.655530] ixgbe 0000:04:00.0 eth0: IOV is enabled with 7 VFs
[ 8.655548] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[ 8.720945] ixgbe 0000:04:00.1: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 8.821966] ixgbevf 0000:04:10.1: enabling device (0000 -> 0002)
[ 8.823146] ixgbevf 0000:04:10.1: PF still in reset state. Is the PF interface up?
[ 8.823149] ixgbevf 0000:04:10.1: Assigning random MAC address
[ 8.823180] ixgbevf 0000:04:10.1: irq 73 for MSI/MSI-X
[ 8.823190] ixgbevf 0000:04:10.1: irq 74 for MSI/MSI-X
[ 8.823477] ixgbevf 0000:04:10.3: enabling device (0000 -> 0002)
[ 8.824648] ixgbevf 0000:04:10.3: PF still in reset state. Is the PF interface up?
[ 8.824650] ixgbevf 0000:04:10.3: Assigning random MAC address
[ 8.824673] ixgbevf 0000:04:10.3: irq 75 for MSI/MSI-X
[ 8.824683] ixgbevf 0000:04:10.3: irq 76 for MSI/MSI-X
[ 8.824974] ixgbevf 0000:04:10.5: enabling device (0000 -> 0002)
[ 8.826143] ixgbevf 0000:04:10.5: PF still in reset state. Is the PF interface up?
[ 8.826145] ixgbevf 0000:04:10.5: Assigning random MAC address
[ 8.826169] ixgbevf 0000:04:10.5: irq 77 for MSI/MSI-X
[ 8.826182] ixgbevf 0000:04:10.5: irq 78 for MSI/MSI-X
[ 8.826511] ixgbevf 0000:04:10.7: enabling device (0000 -> 0002)
[ 8.827686] ixgbevf 0000:04:10.7: PF still in reset state. Is the PF interface up?
[ 8.827688] ixgbevf 0000:04:10.7: Assigning random MAC address
[ 8.827715] ixgbevf 0000:04:10.7: irq 86 for MSI/MSI-X
[ 8.827725] ixgbevf 0000:04:10.7: irq 87 for MSI/MSI-X
[ 9.013662] ixgbevf 0000:04:11.1: enabling device (0000 -> 0002)
[ 9.014843] ixgbevf 0000:04:11.1: PF still in reset state. Is the PF interface up?
[ 9.014846] ixgbevf 0000:04:11.1: Assigning random MAC address
[ 9.014871] ixgbevf 0000:04:11.1: irq 88 for MSI/MSI-X
[ 9.014881] ixgbevf 0000:04:11.1: irq 89 for MSI/MSI-X
[ 9.015169] ixgbevf 0000:04:11.3: enabling device (0000 -> 0002)
[ 9.016342] ixgbevf 0000:04:11.3: PF still in reset state. Is the PF interface up?
[ 9.016345] ixgbevf 0000:04:11.3: Assigning random MAC address
[ 9.016369] ixgbevf 0000:04:11.3: irq 90 for MSI/MSI-X
[ 9.016379] ixgbevf 0000:04:11.3: irq 91 for MSI/MSI-X
[ 9.016662] ixgbevf 0000:04:11.5: enabling device (0000 -> 0002)
[ 9.017837] ixgbevf 0000:04:11.5: PF still in reset state. Is the PF interface up?
[ 9.017839] ixgbevf 0000:04:11.5: Assigning random MAC address
[ 9.017863] ixgbevf 0000:04:11.5: irq 92 for MSI/MSI-X
[ 9.017873] ixgbevf 0000:04:11.5: irq 93 for MSI/MSI-X
[ 9.018021] ixgbe 0000:04:00.1 (unregistered net_device): SR-IOV enabled with 7 VFs
[ 9.042506] ixgbe 0000:04:00.1: irq 94 for MSI/MSI-X
[ 9.042516] ixgbe 0000:04:00.1: irq 95 for MSI/MSI-X
[ 9.042527] ixgbe 0000:04:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 9.042656] ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) 00:a0:a5:78:04:45
[ 9.042741] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: 000301-00B
[ 9.042745] ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s available
[ 9.042747] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 9.084290] ixgbe 0000:04:00.1 eth9: IOV is enabled with 7 VFs
[ 9.084308] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[ 9.084394] ixgbevf 0000:04:10.0: enabling device (0000 -> 0002)
[ 9.085575] ixgbevf 0000:04:10.0: PF still in reset state. Is the PF interface up?
[ 9.085577] ixgbevf 0000:04:10.0: Assigning random MAC address
[ 9.085608] ixgbevf 0000:04:10.0: irq 96 for MSI/MSI-X
[ 9.085618] ixgbevf 0000:04:10.0: irq 97 for MSI/MSI-X
[ 9.085775] ixgbevf 0000:04:10.2: enabling device (0000 -> 0002)
[ 9.086949] ixgbevf 0000:04:10.2: PF still in reset state. Is the PF interface up?
[ 9.086951] ixgbevf 0000:04:10.2: Assigning random MAC address
[ 9.086975] ixgbevf 0000:04:10.2: irq 98 for MSI/MSI-X
[ 9.086988] ixgbevf 0000:04:10.2: irq 99 for MSI/MSI-X
[ 9.087127] ixgbevf 0000:04:10.4: enabling device (0000 -> 0002)
[ 9.088303] ixgbevf 0000:04:10.4: PF still in reset state. Is the PF interface up?
[ 9.088305] ixgbevf 0000:04:10.4: Assigning random MAC address
[ 9.088329] ixgbevf 0000:04:10.4: irq 100 for MSI/MSI-X
[ 9.088339] ixgbevf 0000:04:10.4: irq 101 for MSI/MSI-X
[ 9.088492] ixgbevf 0000:04:10.6: enabling device (0000 -> 0002)
[ 9.089659] ixgbevf 0000:04:10.6: PF still in reset state. Is the PF interface up?
[ 9.089661] ixgbevf 0000:04:10.6: Assigning random MAC address
[ 9.089685] ixgbevf 0000:04:10.6: irq 102 for MSI/MSI-X
[ 9.089695] ixgbevf 0000:04:10.6: irq 103 for MSI/MSI-X
[ 9.089839] ixgbevf 0000:04:11.0: enabling device (0000 -> 0002)
[ 9.091011] ixgbevf 0000:04:11.0: PF still in reset state. Is the PF interface up?
[ 9.091014] ixgbevf 0000:04:11.0: Assigning random MAC address
[ 9.091038] ixgbevf 0000:04:11.0: irq 104 for MSI/MSI-X
[ 9.091048] ixgbevf 0000:04:11.0: irq 105 for MSI/MSI-X
[ 9.091196] ixgbevf 0000:04:11.2: enabling device (0000 -> 0002)
[ 9.092364] ixgbevf 0000:04:11.2: PF still in reset state. Is the PF interface up?
[ 9.092366] ixgbevf 0000:04:11.2: Assigning random MAC address
[ 9.092390] ixgbevf 0000:04:11.2: irq 106 for MSI/MSI-X
[ 9.092400] ixgbevf 0000:04:11.2: irq 107 for MSI/MSI-X
[ 9.092537] ixgbevf 0000:04:11.4: enabling device (0000 -> 0002)
[ 9.093705] ixgbevf 0000:04:11.4: PF still in reset state. Is the PF interface up?
[ 9.093708] ixgbevf 0000:04:11.4: Assigning random MAC address
[ 9.093735] ixgbevf 0000:04:11.4: irq 108 for MSI/MSI-X
[ 9.093745] ixgbevf 0000:04:11.4: irq 109 for MSI/MSI-X
[ 74.863412] ixgbe 0000:04:00.0: registered PHC device on eth0
[ 74.864036] ixgbe 0000:04:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 74.937304] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
[ 76.435995] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
[ 84.946823] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
[ 85.158684] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

~]# lspci | grep -i pci
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 22)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 22)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)

Hello,

As per dmesg, VF's are enabled. So we are good up to this point

But actual issue still exists ! interfaces are not coming.. also could you please suggest, what should be the procedure. so it could bring to state of an automated steps ?

using kickstart?

Since the initramfs had the ixgbe driver, driver was loaded at a very much earlier stage. After rebuilding the initrd image, now the initrd has the required module parameter options included in it thus enabling Virtual Functions

So now as per dmesg, virtual functions are up and enabled. This is what we are discussing here

Rest of the issues that you are noticing will be supported via the Red Hat Service Request. Our support folks will do a deep dive in to your configuration.

We will continue with the rest of the discussion regarding kickstart

In the post section of kickstart, you can place the sriov.conf in /etc/modprobe.d folder. So When installer creates the dracut image for the kernel, it will source the file /etc/modprobe.d/sriov.conf and include it in the initramfs.

PS: the network drivers are only loaded into initramfs image onlywhen dracut-network is installed


Some notes that might help here, I'm not sure why we would want to use the deprecated way of implementing this. Why not use the PCI Subsystem interface
PCI passthrough to assign a Virtual Function of an SR-IOV capable multiport network card to a virtual machine as a network device
BIOS Settings Needed
SR-IOV Global Enable set to Enabled
Enable Or Disable Through PCI Subsystem Interface
echo 8 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs #enable by 8
echo 0 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs #disable
Validate capability and settings with find and lspci commands

[root@dell-per720-2 ~]# find /sys -name sriov_numvfs
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/sriov_numvfs
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/sriov_numvfs

Just to add, In kickstart you can enable VT-D in the kernel using the following directive

bootloader --location=mbr --append="intel_iommu=on"

Dear Ranjith,
Thanks for the comments. Let me try that in Kickstart
Dear Dave,
I tried that options echo'ing 1 to sriv_numvfs but it is not persistent after reboot. could you please suggest so it can be set as permanent/persistent ?

To make it persistent, you will have to write script and name it as /sbin/ifup-local so that this script will be executed automatically whenever the PF is brought up
How do I run a script or program immediately after my network interface goes up?
https://access.redhat.com/solutions/8694

after enabling NetworkManager interfaces are coming. But it wrongly adds interfaces. instead of adding to ixgbe it added to igb interface !

eth6.1601: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.150.2.19 netmask 255.255.255.240 broadcast 192.150.2.31
ether 00:a0:a5:78:04:3c txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2 bytes 84 (84.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth7.2601: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.150.3.19 netmask 255.255.255.240 broadcast 192.150.3.31
ether 00:a0:a5:78:04:3d txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5 bytes 210 (210.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Jul 7 12:29:08 NetworkManager[4042]: [1436252348.552369] [platform/nm-linux-platform.c:1796] add_object(): Netlink error adding 192.150.5.16/28 via 192.150.6.30 dev bond0.3701 metric 400 mss 0 src user: Unspecific failure
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3701): Activation: Stage 5 of 5 (IPv4 Commit) failed
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3701): device state change: ip-config -> failed (reason 'config-failed') [70 120 4]
Jul 7 12:29:08 NetworkManager[4042]: Disabling autoconnect for connection 'Vlan bond0.3701'.
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3701): Activation: failed for connection 'Vlan bond0.3701'
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3701): Activation: Stage 5 of 5 (IPv4 Commit) complete.
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3801): Activation: Stage 5 of 5 (IPv4 Commit) started...
Jul 7 12:29:08 NetworkManager[4042]: [1436252348.554258] [platform/nm-linux-platform.c:1796] add_object(): Netlink error adding 192.150.8.16/28 via 192.150.9.30 dev bond0.3801 metric 400 mss 0 src user: Unspecific failure
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3801): Activation: Stage 5 of 5 (IPv4 Commit) failed
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3801): device state change: ip-config -> failed (reason 'config-failed') [70 120 4]
Jul 7 12:29:08 NetworkManager[4042]: Disabling autoconnect for connection 'Vlan bond0.3801'.
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3801): Activation: failed for connection 'Vlan bond0.3801'
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3801): Activation: Stage 5 of 5 (IPv4 Commit) complete.
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3701): device state change: failed -> disconnected (reason 'none') [120 30 0]
Jul 7 12:29:08 NetworkManager[4042]: (bond0.3701): deactivating device (reason 'none') [0]

I tried it in post of kickstart and this time it is better. ixgbevf is enabled. but interfaces are not coming up still. Gateway is not added checking further on this.

I am unable to proceed further on this. At present, set up udev rules. However while doing so, VLAN interfaces are assigned to igb instead of ixgbe

Cause of the problem:
in RHEL 7.1 intel drivers are assigning 00:00:00:00 to the virtual function interface created. Because of this packets are dropped stating it is a spoofed packet.
Solution to the problem:
Once the network is started pick the virtual function MAC address and assign it to virtual function MAC
This can be done using
Assign the mac address for the VF devices using the ip link command
example:

ip link set <pf> vf <vf_index> mac <mac-id>

This can be done using ifup script by following the below solution
https://access.redhat.com/solutions/8694

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.