Multiqueue limitation on Guest OS to 4

Issue

• Our environment is RHOSP13 with HW based on Cisco C220M5 server with Intel NIC Card XXV710 used for SRIOV interface.

• All the instance that we deploy on this environment is limited with 4 combined queue. But we want to increase this number and it is not possible. Even the Intel card support 16 queue by VF.

• Can you tell us how configure our RHEL to be able to increase the number of queue allowed into a guest vm please ?

• Hypervisor is using the i40e driver for its PF:

[root@overcloud-compute-0 sfr_adm]# modinfo i40e
filename:       /lib/modules/3.10.0-1062.4.1.el7.x86_64/updates/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.7.29
license:        GPL
description:    Intel(R) 40-10 Gigabit Ethernet Connection Network Driver
author:         Intel Corporation, e1000-devel@lists.sourceforge.net
retpoline:      Y
rhelversion:    7.7
srcversion:     2212C8D31172FBDBC9A1647
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:        ptp
vermagic:       3.10.0-1062.4.1.el7.x86_64 SMP mod_unload modversions 
parm:           debug:Debug level (0=none,...,16=all) (int)

• We can see each VFs are configured with 4 queues from the hypervisor point of view:

[root@overcloud-compute-0 sfr_adm]# dmesg | grep -i queue
[    0.187904] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    3.030868] DMAR: dmar6: Using Queued invalidation
[    3.030876] DMAR: dmar4: Using Queued invalidation
[    3.030883] DMAR: dmar3: Using Queued invalidation
[    3.030890] DMAR: dmar2: Using Queued invalidation
[    3.030896] DMAR: dmar0: Using Queued invalidation
[    3.030903] DMAR: dmar7: Using Queued invalidation
[    9.175138] iavf 0000:5e:0a.0: Multiqueue Enabled: Queue pair count = 4
[    9.177306] iavf 0000:d8:0a.0: Multiqueue Enabled: Queue pair count = 4
[    9.208003] iavf 0000:5e:0a.1: Multiqueue Enabled: Queue pair count = 4
[    9.208363] iavf 0000:5e:02.2: Multiqueue Enabled: Queue pair count = 4
[    9.209103] iavf 0000:5e:02.0: Multiqueue Enabled: Queue pair count = 4
[    9.212953] iavf 0000:d8:0a.1: Multiqueue Enabled: Queue pair count = 4
[    9.213372] iavf 0000:5e:02.3: Multiqueue Enabled: Queue pair count = 4
[    9.218488] iavf 0000:d8:02.1: Multiqueue Enabled: Queue pair count = 4
[    9.218870] iavf 0000:5e:0a.2: Multiqueue Enabled: Queue pair count = 4
[    9.220813] iavf 0000:d8:0a.2: Multiqueue Enabled: Queue pair count = 4
[    9.221464] iavf 0000:d8:02.2: Multiqueue Enabled: Queue pair count = 4
[    9.221764] iavf 0000:5e:0a.3: Multiqueue Enabled: Queue pair count = 4
[    9.222848] iavf 0000:d8:0a.3: Multiqueue Enabled: Queue pair count = 4
[    9.223979] iavf 0000:5e:02.1: Multiqueue Enabled: Queue pair count = 4
[    9.224278] iavf 0000:d8:02.0: Multiqueue Enabled: Queue pair count = 4
[    9.260097] iavf 0000:d8:02.3: Multiqueue Enabled: Queue pair count = 4

• Guest VM is running an uncertified OS:

[root@vm-01 centos]# cat /etc/centos-release
CentOS Linux release 8.4.2105

• We can see 4 Combined queues as Pre-set maximums:

[root@vm-01 centos]# ethtool -l eth1  
Channel parameters for eth1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          1
Combined:       4
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          1
Combined:       4

• We're using iavf driver in the guest:

[root@vm-01 centos]# ethtool -i eth1
driver: iavf
version: 4.18.0-305.3.1.el8.x86_64
firmware-version: N/A
expansion-rom-version: 
bus-info: 0000:00:05.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

• Lowering the number of queues works:

[root@vm-01 centos]# ethtool -L eth1 combined 3                     

• Listing the configured queues shows the new value of 3:

[root@vm-01 centos]# ethtool -l eth1 
Channel parameters for eth1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          1
Combined:       4
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          1
Combined:       3

• We can see it now have 3 IRQs:

[root@vm-01 centos]# cat /proc/interrupts | grep -i eth1
 30:         11          1          0          0   PCI-MSI 81921-edge      iavf-eth1-TxRx-0
 31:          0          0          0          0   PCI-MSI 81922-edge      iavf-eth1-TxRx-1
 32:          0          0          0          0   PCI-MSI 81923-edge      iavf-eth1-TxRx-2

• An attempt to increase the number of queues fails with:

[root@vm-01 centos]# ethtool -L eth1 combined 5
netlink error: requested channel count exceeds maximum (offset 36)
netlink error: Invalid argument

• Setting it to the maximum succeeds:

[root@vm-01 centos]# ethtool -L eth1 combined 4

• Display of configured queues:

[root@vm-01 centos]# ethtool -l eth1           
Channel parameters for eth1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          1
Combined:       4
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          1
Combined:       4

• List of interrupts seen in the guest:

[root@vm-01 centos]# cat /proc/interrupts | grep -i eth1
 30:          5          0          0          0   PCI-MSI 81921-edge      iavf-eth1-TxRx-0
 31:          0          0          0          0   PCI-MSI 81922-edge      iavf-eth1-TxRx-1
 32:          0          0          0          0   PCI-MSI 81923-edge      iavf-eth1-TxRx-2
 33:          0          0          0          0   PCI-MSI 81924-edge      iavf-eth1-TxRx-3