Low network throughput on virtual machines that run on NUMA nodes where also fiber HBA I/O is active

Solution Verified - Updated -

Issue

  • We see low network throughput on VMs running on all of the numa nodes where the hba's IRQs are bound to, so on 8 of the 16 numa nodes. Why do we experience this bottleneck, how to solve it?

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • NUMA system with these details:
    • We have an overall system (16 numa nodes). Each node has 20 cores, so overall 320 cores
    • We use kernel 2.6.32-358.2.1.el6.x86_64 .
    • 768 virtual machines are started
    • The VMs use VF devices of the physical network cards (functionality from the network driver). The IRQ's of the VFs are assigned to their respective numa nodes. Each VFs IRQs are assigned so they are local to the VM which uses the VF.
    • 8 fiber HBAs are in the system. The HBA IRQs are bound to core 0 of the numa sockets 1,3,5,7,9,11,13,15

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content