Chapter 34. Tuning the network performance

Procedure

1. Display the packet drop statistics of the interface:

   # ethtool -S enp1s0
       ...
       rx_queue_0_drops: 97326
       rx_queue_1_drops: 63783
       ...

   Note that the output of the command depends on the network card and the driver.

   High values in discard or drop counters indicate that the available buffer fills up faster than the kernel can process the packets. Increasing the ring buffers can help to avoid such loss.

2. Display the maximum ring buffer sizes:

   # ethtool -g enp1s0
    Ring parameters for enp1s0:
    Pre-set maximums:
    RX:             4096
    RX Mini:        0
    RX Jumbo:       16320
    TX:             4096
    Current hardware settings:
    RX:             255
    RX Mini:        0
    RX Jumbo:       0
    TX:             255

   If the values in the Pre-set maximums section are higher than in the Current hardware settings section, you can change the settings in the next steps.

3. Identify the NetworkManager connection profile that uses the interface:

   # nmcli connection show
   NAME                UUID                                  TYPE      DEVICE
   Example-Connection  a5eb6490-cc20-3668-81f8-0314a27f3f75  ethernet  enp1s0

4. Update the connection profile, and increase the ring buffers:

   • To increase the RX ring buffer, enter:

     # nmcli connection modify Example-Connection ethtool.ring-rx 4096

   • To increase the TX ring buffer, enter:

     # nmcli connection modify Example-Connection ethtool.ring-tx 4096

5. Reload the NetworkManager connection:

   # nmcli connection up Example-Connection

   Important

   Depending on the driver your NIC uses, changing in the ring buffer can shortly interrupt the network connection.

Procedure

1. Determine whether tuning the backlog queue is needed, display the counters in the /proc/net/softnet_stat file:

   # awk '{for (i=1; i<=NF; i++) printf strtonum("0x" $i) (i==NF?"\n":" ")}' /proc/net/softnet_stat | column -t
   221951548  0      0  0  0  0  0  0  0  0  0  0  0
   192058677  18862  0  0  0  0  0  0  0  0  0  0  1
   455324886  0      0  0  0  0  0  0  0  0  0  0  2
   ...

   This awk command converts the values in /proc/net/softnet_stat from hexadecimal to decimal format and displays them in table format. Each line represents a CPU core starting with core 0.

   The relevant columns are:

   • First column: The total number of received frames
   • Second column: The number of dropped frames because of a full backlog queue
   • Last column: The CPU core number

2. If the values in the second column of the /proc/net/softnet_stat file increment over time, increase the size of the backlog queue:

   1. Display the current backlog queue size:

      # sysctl net.core.netdev_max_backlog
      net.core.netdev_max_backlog = 1000

   2. Create the /etc/sysctl.d/10-netdev_max_backlog.conf file with the following content:

      net.core.netdev_max_backlog = 2000

      Set the net.core.netdev_max_backlog parameter to a double of the current value.

   3. Load the settings from the /etc/sysctl.d/10-netdev_max_backlog.conf file:

      # sysctl -p /etc/sysctl.d/10-netdev_max_backlog.conf

Procedure

1. Display the current transmit queue length:

   # ip -s link show enp1s0
   2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
   ...

   In this example, the transmit queue length (qlen) of the enp1s0 interface is 1000.

2. Monitor the dropped packets counter of a network interface’s software transmit queue:

   # tc -s qdisc show dev enp1s0
   qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
    Sent 16889923 bytes 426862765 pkt (dropped 191980, overlimits 0 requeues 2)
   ...

3. If you encounter a high or increasing transmit error count, set a higher transmit queue length:

   1. Identify the NetworkManager connection profile that uses this interface:

      # nmcli connection show
      NAME                UUID                                  TYPE      DEVICE
      Example-Connection  a5eb6490-cc20-3668-81f8-0314a27f3f75  ethernet  enp1s0

   2. Create the /etc/NetworkManager/dispatcher.d/99-set-tx-queue-length-up NetworkManager dispatcher script with the following content:

      #!/bin/bash
      # Set TX queue length on enp1s0 to 2000
      
      if [ "$1" == "enp1s0" ] && [ "$2" == "up" ] ; then
          ip link set dev enp1s0 txqueuelen 2000
      fi

   3. Set the executable bit on the /etc/NetworkManager/dispatcher.d/99-set-tx-queue-length-up file:

      # chmod +x /etc/NetworkManager/dispatcher.d/99-set-tx-queue-length-up

   4. Apply the changes:

      # nmcli connection up Example-Connection

Verification

1. Display the transmit queue length:

   # ip -s link show enp1s0
   2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 2000
   ...

2. Monitor the dropped packets counter:

   # tc -s qdisc show dev enp1s0

   If the dropped counter still increases, double the transmit queue length again. Repeat this process until the counter no longer increases.

Procedure

1. To determine whether tuning the net.core.netdev_budget parameter is needed, display the counters in the /proc/net/softnet_stat file:

   # awk '{for (i=1; i<=NF; i++) printf strtonum("0x" $i) (i==NF?"\n":" ")}' /proc/net/softnet_stat | column -t
   221951548  0  0      0  0  0  0  0  0  0  0  0  0
   192058677  0  20380  0  0  0  0  0  0  0  0  0  1
   455324886  0  0      0  0  0  0  0  0  0  0  0  2
   ...

   This awk command converts the values in /proc/net/softnet_stat from hexadecimal to decimal format and displays them in the table format. Each line represents a CPU core starting with core 0.

   The relevant columns are:

   • First column: The total number of received frames.
   • Third column: The number times softirqd processes that could not retrieve all packets from interfaces in one NAPI polling cycle.
   • Last column: The CPU core number.

2. If the counters in the third column of the /proc/net/softnet_stat file increment over time, tune the system:

   1. Display the current values of the net.core.netdev_budget_usecs and net.core.netdev_budget parameters:

      # sysctl net.core.netdev_budget_usecs net.core.netdev_budget
      net.core.netdev_budget_usecs = 2000
      net.core.netdev_budget = 300

      With these settings, softirqd processes have up to 2000 microseconds to process up to 300 messages from the NIC in one polling cycle. Polling ends based on which condition is met first.

   2. Create the /etc/sysctl.d/10-netdev_budget.conf file with the following content:

      net.core.netdev_budget = 600
      net.core.netdev_budget_usecs = 4000

      Set the parameters to a double of their current values.

   3. Load the settings from the /etc/sysctl.d/10-netdev_budget.conf file:

      # sysctl -p /etc/sysctl.d/10-netdev_budget.conf

Procedure

1. Display the active profile:

   # tuned-adm active
   Current active profile: network-latency

2. Create a directory for the custom TuneD profile:

   # mkdir /etc/tuned/network-latency-custom/

3. Create the /etc/tuned/network-latency-custom/tuned.conf file with the following content:

   [main]
   include=network-latency
   
   [cpu]
   force_latency=cstate.id:1|2

   This custom profile inherits all settings from the network-latency profile. The force_latency TuneD parameter specifies the latency in microseconds (µs). If the C-state latency is higher than the specified value, the idle driver in Red Hat Enterprise Linux prevents the CPU from moving to a higher C-state. With force_latency=cstate.id:1|2, TuneD first checks if the /sys/devices/system/cpu/cpu_<number>_/cpuidle/state_<cstate.id>_/ directory exists. In this case, TuneD reads the latency value from the latency file in this directory. If the directory does not exist, TuneD uses 2 microseconds as a fallback value.

4. Activate the network-latency-custom profile:

   # tuned-adm profile network-latency-custom

Procedure

1. Display the idle driver the system uses:

   # cat /sys/devices/system/cpu/cpuidle/current_driver
   intel_idle

2. If the host uses the intel_idle driver, set the intel_idle.max_cstate kernel parameter to define the highest C-state that CPU cores should be able to use:

   # grubby --update-kernel=ALL --args="intel_idle.max_cstate=0"

   Setting intel_idle.max_cstate=0 disables the intel_idle driver. Consequently, the kernel uses the acpi_idle driver that uses the C-state values set in the EFI firmware. For this reason, also set processor.max_cstate to override these C-state settings.

3. On every host, independent from the CPU vendor, set the highest C-state that CPU cores should be able to use:

   # grubby --update-kernel=ALL --args="processor.max_cstate=0"

   Important

   If you set processor.max_cstate=0 in addition to intel_idle.max_cstate=0, the acpi_idle driver overrides the value of processor.max_cstate and sets it to 1. As a result, with processor.max_cstate=0 intel_idle.max_cstate=0, the highest C-state the kernel will use is C1, not C0.

4. Restart the host for the changes to take effect:

   # reboot

Verification

1. Display the maximum C-state:

   # cat /sys/module/processor/parameters/max_cstate
   1

2. If the host uses the intel_idle driver, display the maximum C-state:

   # cat /sys/module/intel_idle/parameters/max_cstate
   0

Procedure

1. Optional: Display the current MTU:

   # ip link show
   ...
   3: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
       link/ether 52:54:00:74:79:56 brd ff:ff:ff:ff:ff:ff
   ...

2. Optional: Display the NetworkManager connection profiles:

   # nmcli connection show
   NAME     UUID                                  TYPE      DEVICE
   Example  f2f33f29-bb5c-3a07-9069-be72eaec3ecf  ethernet  enp1s0
   ...

3. Set the MTU in the profile that manages the connection to the network with the divergent MTU:

   # nmcli connection modify Example mtu 9000

4. Reactivate the connection:

   # nmcli connection up Example

Verification

1. Display the MTU setting:

   # ip link show
   ...
   3: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
       link/ether 52:54:00:74:79:56 brd ff:ff:ff:ff:ff:ff
   ...

2. Verify that no host on the transmission paths fragments the packets:

   • On the receiver side, display the IP reassembly statistics of the kernel:

     # nstat -az IpReasm*
     #kernel
     IpReasmTimeout 0 0.0
     IpReasmReqds 0 0.0
     IpReasmOKs 0 0.0
     IpReasmFails 0 0.0

     If the counters return 0, packets were not reassembled.

   • On the sender side, transmit an ICMP request with the prohibit-fragmentation-bit:

     # ping -c1 -Mdo -s 8972 destination_host

     If the command succeeds, the packet was not fragmented.

     Calculate the value for the -s packet size option as follows: MTU size - 8 bytes ICMP header - 20 bytes IPv4 header = packet size

Procedure

1. Optional: Display the maximum network speed of the network interface controller (NIC) on both the server and client:

   # ethtool enp1s0 | grep "Speed"
      Speed: 100000Mb/s

2. On the server:

   1. Temporarily open the default iperf3 TCP port 5201 in the firewalld service:

      # firewall-cmd --add-port=5201/tcp
      # firewall-cmd --reload

   2. Start iperf3 in server mode:

      # iperf3 --server

      The service now is waiting for incoming client connections.

3. On the client:

   1. Start measuring the throughput:

      # iperf3 --time 60 --zerocopy --client 192.0.2.1

      • --time <seconds>: Defines the time in seconds when the client stops the transmission.

        Set this parameter to a value that you expect to work and increase it in later measurements. If the server sends packets at a faster rate than the devices on the transmit path or the client can process, packets can be dropped.

      • --zerocopy: Enables a zero copy method instead of using the write() system call. You require this option only if you want to simulate a zero-copy-capable application or to reach 40 Gbps and more on a single stream.
      • --client <server>: Enables the client mode and sets the IP address or name of the server that runs the iperf3 server.

4. Wait until iperf3 completes the test. Both the server and the client display statistics every second and a summary at the end. For example, the following is a summary displayed on a client:

   [ ID] Interval         Transfer    Bitrate         Retr
   [  5] 0.00-60.00  sec  101 GBytes   14.4 Gbits/sec   0   sender
   [  5] 0.00-60.04  sec  101 GBytes   14.4 Gbits/sec       receiver

   In this example, the average bitrate was 14.4 Gbps.

5. On the server:

   1. Press Ctrl+C to stop the iperf3 server.

   2. Close the TCP port 5201 in firewalld:

      # firewall-cmd --remove-port=5201/tcp
      # firewall-cmd --reload

Procedure

1. Determine the latency of the connection. For example, ping from the client to server to measure the average Round Trip Time (RTT):

   # ping -c 10 server.example.com
   ...
   --- server.example.com ping statistics ---
   10 packets transmitted, 10 received, 0% packet loss, time 9014ms
   rtt min/avg/max/mdev = 117.208/117.056/119.333/0.616 ms

   In this example, the latency is 117 ms.

2. Use the following formula to calculate the Bandwidth Delay Product (BDP) for the traffic you want to tune:

   connection speed in bytes * latency in ms = BDP in bytes

   For example, to calculate the BDP for a 10 Gbps connection that has a 117 ms latency:

   (10 * 1000 * 1000 * 1000 / 8) * 117 = 10683760 bytes

3. Create the /etc/sysctl.d/10-tcp-socket-buffers.conf file and either set the maximum read or write buffer size, or both, based on your requirements:

   net.ipv4.tcp_rmem = 4096 262144 21367520
   net.ipv4.tcp_wmem = 4096 24576 21367520

   Specify the values in bytes. Use the following rule of thumb when you try to identify optimized values for your environment:

   • Default buffer size (second value): Increase this value only slightly or set it to 524288 (512 KiB) at most. A too high default buffer size can cause buffer collapsing and, consequently, latency spikes.
   • Maximum buffer size (third value): A value double to triple of the BDP is often sufficient.

4. Load the settings from the /etc/sysctl.d/10-tcp-socket-buffers.conf file:

   # sysctl -p /etc/sysctl.d/10-tcp-socket-buffers.conf

5. Configure your applications to use a larger socket buffer size. The third value in the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem parameters defines the maximum buffer size that the setsockopt() function in an application can request.

   For further details, see the documentation of the programming language of your application. If you are not the developer of the application, contact the developer.

6. If you have changed the second value in the net.ipv4.tcp_rmem or net.ipv4.tcp_wmem parameter, restart the applications to use the new TCP buffer sizes.

   If you have changed only the third value, you do not need to restart the application because auto-tuning applies these settings dynamically.

Verification

1. Optional: Test the TCP throughput using iperf3.

2. Monitor the packet drop statistics using the same method that you used when you encountered the packet drops.

   If packet drops still occur but at a lower rate, increase the buffer sizes further.

Procedure

1. Identify if the network interface controller (NIC) drops packets:

   1. Display the NIC and driver-specific statistics:

      # ethtool -S enp1s0
      NIC statistics:
           ...
           rx_queue_0_drops: 17657
           ...

      The naming of the statistics and if they are available depend on the NIC and the driver.

   2. Display the interface statistics:

      # ip -s link show enp1s0
      2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
          link/ether 52:54:00:74:79:56 brd ff:ff:ff:ff:ff:ff
          RX:   bytes  packets errors dropped  missed   mcast_
          84697611107 56866482      0   10904       0       0
          TX:   bytes  packets errors dropped carrier collsns_
           5540028184  3722234      0       0       0       0

      RX represents the statistics of received packets and TX of transmitted packets.

2. Identify UDP protocol-specific packet drops due to too small socket buffers or slow application processing:

   # nstat -az UdpSndbufErrors UdpRcvbufErrors
   #kernel
   UdpSndbufErrors           4    0.0
   UdpRcvbufErrors    45716659    0.0

   The second column in the output lists the counters.

Procedure

1. Optional: Display the maximum network speed of the network interface controller (NIC) on both the server and client:

   # ethtool enp1s0 | grep "Speed"
      Speed: 10000Mb/s

2. On the server:

   1. Display the maximum UDP socket read buffer size, and note the value:

      # sysctl net.core.rmem_max
      net.core.rmem_max = 16777216

      The displayed value is in bytes.

   2. Temporarily open the default iperf3 port 5201 in the firewalld service:

      # firewall-cmd --add-port=5201/tcp --add-port=5201/udp
      # firewall-cmd --reload

      Note that, iperf3 opens only a TCP socket on the server. If a clients wants to use UDP, it first connects to this TCP port, and then the server opens a UDP socket on the same port number for performing the UDP traffic throughput test. For this reason, you must open port 5201 for both the TCP and UDP protocol in the local firewall.

   3. Start iperf3 in server mode:

      # iperf3 --server

      The service now waits for incoming client connections.

3. On the client:

   1. Display the Maximum Transmission Unit (MTU) of the interface that the client will use for the connection to the server, and note the value:

      # ip link show enp1s0
      2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
      ...

   2. Display the maximum UDP socket write buffer size, and note the value:

      # sysctl net.core.wmem_max
      net.core.wmem_max = 16777216

      The displayed value is in bytes.

   3. Start measuring the throughput:

      # iperf3 --udp --time 60 --window 16777216 --length 1472 --bitrate 2G --client 192.0.2.1

      • --udp: Use the UDP protocol for the test.
      • --time <seconds>: Defines the time in seconds when the client stops the transmission.
      • --window <size>: Sets the UDP socket buffer size. Ideally, the sizes are the same on both the client and server. In case that they are different, set this parameter to the value that is smaller: net.core.wmem_max on the client or net.core.rmem_max on the server.
      • --length <size>: Sets the length of the buffer to read and write. Set this option to the largest unfragmented payload. Calculate the ideal value as follows: MTU - IP header (20 bytes for IPv4 and 40 bytes for IPv6) - 8 bytes UDP header.

      • --bitrate <rate>: Limits the bit rate to the specified value in bits per second. You can specify units, such as 2G for 2 Gbps.

        Set this parameter to a value that you expect to work and increase it in later measurements. If the server sends packets at a faster rate than the devices on the transmit path or the client can process them, packets can be dropped.

      • --client <server>: Enables the client mode and sets the IP address or name of the server that runs the iperf3 server.

4. Wait until iperf3 completes the test. Both the server and the client display statistics every second and a summary at the end. For example, the following is a summary displayed on a client:

   [ ID] Interval       Transfer      Bitrate         Jitter    Lost/Total Datagrams
   [ 5] 0.00-60.00 sec  14.0 GBytes   2.00 Gbits/sec  0.000 ms  0/10190216 (0%) sender
   [ 5] 0.00-60.04 sec  14.0 GBytes   2.00 Gbits/sec  0.002 ms  0/10190216 (0%) receiver

   In this example, the average bit rate was 2 Gbps, and no packets were lost.

5. On the server:

   1. Press Ctrl+C to stop the iperf3 server.

   2. Close port 5201 in firewalld:

      # firewall-cmd --remove-port=5201/tcp --remove-port=5201/udp
      # firewall-cmd --reload

Procedure

1. Create the /etc/sysctl.d/10-udp-socket-buffers.conf file and either set the maximum read or write buffer size, or both, based on your requirements:

   net.core.rmem_max = 16777216
   net.core.wmem_max = 16777216

   Specify the values in bytes. The values in this example set the maximum size of buffers to 16 MiB. The default values of both parameters are 212992 bytes (208 KiB).

2. Load the settings from the /etc/sysctl.d/10-udp-socket-buffers.conf file:

   # sysctl -p /etc/sysctl.d/10-udp-socket-buffers.conf

3. Configure your applications to use the larger socket buffer sizes.

   The net.core.rmem_max and net.core.wmem_max parameters define the maximum buffer size that the setsockopt() function in an application can request. Note that, if you configure your application to not use the setsockopt() function, the kernel uses the values from the rmem_default and wmem_default parameters.

   For further details, see the documentation of the programming language of your application. If you are not the developer of the application, contact the developer.

4. Restart the applications to use the new UDP buffer sizes.

Procedure

1. Use the nstat utility to query the TcpExtTCPRcvCollapsed and TcpExtRcvPruned counters:

   # nstat -az TcpExtTCPRcvCollapsed TcpExtRcvPruned
   #kernel
   TcpExtRcvPruned            0         0.0
   TcpExtTCPRcvCollapsed      612859    0.0

2. Wait some time and re-run the nstat command:

   # nstat -az TcpExtTCPRcvCollapsed TcpExtRcvPruned
   #kernel
   TcpExtRcvPruned            0         0.0
   TcpExtTCPRcvCollapsed      620358    0.0

3. If the values of the counters have increased compared to the first run, tuning is required:

   • If the application uses the setsockopt(SO_RCVBUF) call, consider removing it. With this call, the application only uses the receive buffer size specified in the call and turns off the socket’s ability to auto-tune its size.
   • If the application does not use the setsockopt(SO_RCVBUF) call, tune the default and maximum values of the TCP read socket buffer.

4. Display the receive backlog queue (Recv-Q):

   # ss -nti
   State   Recv-Q   Send-Q   Local Address:Port   Peer Address:Port   Process
   ESTAB   0        0        192.0.2.1:443        192.0.2.125:41574
         :7,7 ... lastrcv:543 ...
   ESTAB   78       0        192.0.2.1:443        192.0.2.56:42612
         :7,7 ... lastrcv:658 ...
   ESTAB   88       0        192.0.2.1:443        192.0.2.97:40313
         :7,7 ... lastrcv:5764 ...
   ...

5. Run the ss -nt command multiple times with a few seconds waiting time between each run.

   If the output lists only one case of a high value in the Recv-Q column, the application was between two receive operations. However, if the values in Recv-Q stays constant while lastrcv continually grows, or Recv-Q continually increases over time, one of the following problems can be the cause:

   • The application does not check its socket buffers often enough. Contact the application vendor for details about how you can solve this problem.

   • The application does not get enough CPU time. To further debug this problem:

     1. Display on which CPU cores the application runs:

        # ps -eo pid,tid,psr,pcpu,stat,wchan:20,comm
            PID     TID PSR %CPU STAT WCHAN                COMMAND
        ...
          44594   44594   5  0.0 Ss   do_select            httpd
          44595   44595   3  0.0 S    skb_wait_for_more_pa httpd
          44596   44596   5  0.0 Sl   pipe_read            httpd
          44597   44597   5  0.0 Sl   pipe_read            httpd
          44602   44602   5  0.0 Sl   pipe_read            httpd
        ...

        The PSR column displays the CPU cores the process is currently assigned to.

     2. Identify other processes running on the same cores and consider assigning them to other cores.

Procedure

1. To verify whether tuning is required, display the statistics for the affected port:

   # ss -ntl '( sport = :443 )'
   State    Recv-Q   Send-Q   Local Address:Port   Peer Address:Port  Process
   LISTEN   650      500      192.0.2.1:443        0.0.0.0:*

   If the current number of connections in the backlog (Recv-Q) is larger than the socket backlog (Send-Q), the listen backlog is still not large enough and tuning is required.

2. Optional: Display the current TCP listen backlog limit:

   # sysctl net.core.somaxconn
   net.core.somaxconn = 4096

3. Create the /etc/sysctl.d/10-socket-backlog-limit.conf file, and set a larger listen backlog limit:

   net.core.somaxconn = 8192

   Note that applications can request a larger listen backlog than specified in the net.core.somaxconn kernel parameter but the kernel limits the application to the number you set in this parameter.

4. Load the setting from the /etc/sysctl.d/10-socket-backlog-limit.conf file:

   # sysctl -p /etc/sysctl.d/10-socket-backlog-limit.conf

5. Reconfigure the application to use the new listen backlog limit:

   • If the application provides a config option for the limit, update it. For example, the Apache HTTP Server provides the ListenBacklog configuration option to set the listen backlog limit for this service.
   • If you cannot configure the limit, recompile the application.
6. Restart the application.

Verification

1. Monitor the Systemd journal for further occurrences of possible SYN flooding on port <port_number> error messages.

2. Monitor the current number of connections in the backlog and compare it with the socket backlog:

   # ss -ntl '( sport = :443 )'
   State    Recv-Q   Send-Q   Local Address:Port   Peer Address:Port  Process
   LISTEN   0        500      192.0.2.1:443        0.0.0.0:*

   If the current number of connections in the backlog (Recv-Q) is larger than the socket backlog (Send-Q), the listen backlog is not large enough and further tuning is required.

1. Enable the rhel-8-for-x86_64-baseos-debug-rpms repository:

   # subscription-manager repos --enable rhel-8-for-x86_64-baseos-debug-rpms

2. Install the kernel-debuginfo-common-x86_64 package:

   # yum install kernel-debuginfo-common-x86_64

3. The code example is now available in the /usr/src/debug/kernel-<version>/linux-<version>/tools/testing/selftests/net/reuseport_bpf_cpu.c file.

Procedure

1. Display the count of available queues:

   # ethtool -l enp1s0
   Channel parameters for enp1s0:
   Pre-set maximums:
   RX:		0
   TX:		0
   Other:		0
   Combined:	4
   Current hardware settings:
   RX:		0
   TX:		0
   Other:		0
   Combined:	1

   The Pre-set maximums section shows the total number of queues and Current hardware settings the number of queues that are currently assigned to the receive, transmit, other, or combined queues.

2. Optional: If you require queues on specific channels, assign them accordingly. For example, to assign the 4 queues to the Combined channel, enter:

   # ethtool -L enp1s0 combined 4

3. Display to which Non-Uniform Memory Access (NUMA) node the NIC is assigned:

   # cat /sys/class/net/enp1s0/device/numa_node
   0

   If the file is not found or the command returns -1, the host is not a NUMA system.

4. If the host is a NUMA system, display which CPUs are assigned to which NUMA node:

   # lscpu | grep NUMA
   NUMA node(s):       2
   NUMA node0 CPU(s):  0-3
   NUMA node1 CPU(s):  4-7

5. In the example above, the NIC has 4 queues and the NIC is assigned to NUMA node 0. This node uses the CPU cores 0-3. Consequently, map each transmit queue to one of the CPU cores from 0-3:

   # echo 1 > /sys/class/net/enp1s0/queues/tx-0/xps_cpus
   # echo 2 > /sys/class/net/enp1s0/queues/tx-1/xps_cpus
   # echo 4 > /sys/class/net/enp1s0/queues/tx-2/xps_cpus
   # echo 8 > /sys/class/net/enp1s0/queues/tx-3/xps_cpus

   If the number of CPU cores and transmit (TX) queues is the same, use a 1 to 1 mapping to avoid any kind of contention on the TX queue. Otherwise, if you map multiple CPUs on the same TX queue, transmit operations on different CPUs will cause TX queue lock contention and negatively impacts the transmit throughput.

   Note that you must pass the bitmap, containing the CPU’s core numbers, to the queues. Use the following command to calculate the bitmap:

   # printf %x $((1 << <core_number> ))

Verification

1. Identify the process IDs (PIDs) of services that send traffic:

   # pidof <process_name>
   12345 98765

2. Pin the PIDs to cores that use XPS:

   # numactl -C 0-3 12345 98765

3. Monitor the requeues counter while the process send traffic:

   # tc -s qdisc
   qdisc fq_codel 0: dev enp10s0u1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
    Sent 125728849 bytes 1067587 pkt (dropped 0, overlimits 0 requeues 30)
    backlog 0b 0p requeues 30
    ...

   If the requeues counter no longer increases at a significant rate, TX queue lock contention no longer happens.

Procedure

1. Optional: Display the NetworkManager connection profiles:

   # nmcli connection show
   NAME     UUID                                  TYPE      DEVICE
   example  f2f33f29-bb5c-3a07-9069-be72eaec3ecf  ethernet  enp1s0

2. Disable GRO support in the connection profile:

   # nmcli connection modify example ethtool.feature-gro off

3. Reactivate the connection profile:

   # nmcli connection up example

Verification

1. Verify that GRO is disabled:

   # ethtool -k enp1s0 | grep generic-receive-offload
   generic-receive-offload: off

2. Monitor the throughput on the server. Re-enable GRO in the NetworkManager profile if the setting has negative side effects to other applications on the host.

Procedure

1. Identify the driver:

   # ethtool -i enp0s31f6
   driver: e1000e
   version: ...
   firmware-version: ...
   ...

   Note that certain features can require a specific driver and firmware version.

2. Display the available parameters of the kernel module:

   # modinfo -p e1000e
   ...
   SmartPowerDownEnable:Enable PHY smart power down (array of int)
   parm:RxIntDelay:Receive Interrupt Delay (array of int)

   For further details on the parameters, see the kernel module’s documentation. For modules in RHEL, see the documentation in the /usr/share/doc/kernel-doc-<version>/Documentation/networking/device_drivers/ directory that is provided by the kernel-doc package.

3. Create the /etc/modprobe.d/nic-parameters.conf file and specify the parameters for the module:

   options <module_name> <parameter1>=<value> <parameter2>=<value>

   For example, to enable the port power saving mechanism and set the generation of receive interrupts to 4 units, enter:

   options e1000e SmartPowerDownEnable=1 RxIntDelay=4

4. Unload the module:

   # modprobe -r e1000e

   Warning

   Unloading a module that an active network interface uses, immediately terminates the connection and you can lock yourself out of the server.

5. Load the module:

   # modprobe e1000e

6. Reactivate the network connections:

   # nmcli connection up <profile_name>

Verification

1. Display the kernel messages:

   # dmesg
   ...
   [35309.225765] e1000e 0000:00:1f.6: Transmit Interrupt Delay set to 16
   [35309.225769] e1000e 0000:00:1f.6: PHY Smart Power Down Enabled
   ...

   Note that not all modules log parameter settings to the kernel ring buffer.

2. Certain kernel modules create files for each module parameter in the /sys/module/<driver>/parameters/ directory. Each of these files contain the current value of this parameter. You can display these files to verify a setting:

   # cat /sys/module/<driver_name>/parameters/<parameter_name>

Procedure

1. Display the interface’s available offload features and their current state:

   # ethtool -k enp1s0
   ...
   esp-hw-offload: on
   ntuple-filters: off
   rx-vlan-filter: off [fixed]
   ...

   The output depends on the capabilities of the hardware and its driver. Note that you cannot change the state of features that are flagged with [fixed].

2. Temporarily disable an offload feature:

   # ethtool -K <interface> <feature> [on|off]

   • For example, to temporarily disable IPsec Encapsulating Security Payload (ESP) offload on the enp10s0u1 interface, enter:

     # ethtool -K enp10s0u1 esp-hw-offload off

   • For example, to temporarily enable accelerated Receive Flow Steering (aRFS) filtering on the enp10s0u1 interface, enter:

     # ethtool -K enp10s0u1 ntuple-filters on

Verification

1. Display the states of the offload features:

   # ethtool -k enp1s0
   ...
   esp-hw-offload: off
   ntuple-filters: on
   ...

2. Test whether the problem you encountered before changing the offload feature still exists.

   • If the problem no longer exists after changing a specific offload feature:

     1. Contact Red Hat Support and report the problem.
     2. Consider permanently setting the offload feature until a fix is available.

   • If the problem still exists after disabling a specific offload feature:

     1. Reset the setting to its previous state by using the ethtool -K <interface> <feature> [on|off] command.
     2. Enable or disable a different offload feature to narrow down the problem.

Procedure

1. Identify the connection profile that uses the network interface on which you want to change the state of the offload feature:

   # nmcli connection show
   NAME     UUID                                  TYPE      DEVICE
   Example  a5eb6490-cc20-3668-81f8-0314a27f3f75  ethernet  enp1ss0
   ...

2. Permanently change the state of the offload feature:

   # nmcli connection modify <connection_name> <feature> [on|off]

   • For example, to permanently disable IPsec Encapsulating Security Payload (ESP) offload in the Example connection profile, enter:

     # nmcli connection modify Example ethtool.feature-esp-hw-offload off

   • For example, to permanently enable accelerated Receive Flow Steering (aRFS) filtering in the Example connection profile, enter:

     # nmcli connection modify Example ethtool.feature-ntuple on

3. Reactivate the connection profile:

   # nmcli connection up Example

Procedure

1. Identify the network interface that is experiencing the bottleneck:

   # ethtool -S enp1s0
   NIC statistics:
        rx_packets: 1234
        tx_packets: 5678
        rx_bytes: 12345678
        tx_bytes: 87654321
        rx_errors: 0
        tx_errors: 0
        rx_missed: 0
        tx_dropped: 0
        coalesced_pkts: 0
        coalesced_events: 0
        coalesced_aborts: 0

   Identify the packet counters containing "drop", "discard", or "error" in their name. These particular statistics measure the actual packet loss at the network interface card (NIC) packet buffer, which can be caused by NIC coalescence.

2. Monitor values of packet counters you identified in the previous step.

   Compare them to the expected values for your network to determine whether any particular interface experiences a bottleneck. Some common signs of a network bottleneck include, but are not limited to:

   • Many errors on a network interface
   • High packet loss

   • Heavy usage of the network interface

     Note

     Other important factors are for example CPU usage, memory usage, and disk I/O when identifying a network bottleneck.

3. View the current coalescence settings:

   # ethtool enp1s0
   Settings for enp1s0:
           Supported ports: [ TP ]
           Supported link modes:   10baseT/Half 10baseT/Full
                                   100baseT/Half 100baseT/Full
                                   1000baseT/Full
           Supported pause frame use: No
           Supports auto-negotiation: Yes
           Advertised link modes:  10baseT/Half 10baseT/Full
                                   100baseT/Half 100baseT/Full
                                   1000baseT/Full
           Advertised pause frame use: No
           Advertised auto-negotiation: Yes
           Speed: 1000Mb/s
           Duplex: Full
           Port: Twisted Pair
           PHYAD: 0
           Transceiver: internal
           Auto-negotiation: on
           MDI-X: Unknown
           Supports Wake-on: g
           Wake-on: g
           Current message level: 0x00000033 (51)
                                  drv probe link
           Link detected: yes

   In this output, monitor the Speed and Duplex fields. These fields display information about the network interface operation and whether it is running at its expected values.

4. Check the current interrupt coalescence settings:

   # ethtool -c enp1s0
   Coalesce parameters for enp1s0:
           Adaptive RX: off
           Adaptive TX: off
           RX usecs: 100
           RX frames: 8
           RX usecs irq: 100
           RX frames irq: 8
           TX usecs: 100
           TX frames: 8
           TX usecs irq: 100
           TX frames irq: 8

   • The usecs values refer to the number of microseconds that the receiver or transmitter waits before generating an interrupt.
   • The frames values refer to the number of frames that the receiver or transmitter waits before generating an interrupt.

   • The irq values are used to configure the interrupt moderation when the network interface is already handling an interrupt.

     Note

     Not all network interface cards support reporting and changing all values from the example output.

   • The Adaptive RX/TX value represents the adaptive interrupt coalescence mechanism, which adjusts the interrupt coalescence settings dynamically. Based on the packet conditions, the NIC driver auto-calculates coalesce values when Adaptive RX/TX are enabled (the algorithm differs for every NIC driver).

5. Modify the coalescence settings as needed. For example:

   • While ethtool.coalesce-adaptive-rx is disabled, configure ethtool.coalesce-rx-usecs to set the delay before generating an interrupt to 100 microseconds for the RX packets:

     # nmcli connection modify enp1s0 ethtool.coalesce-rx-usecs 100

   • Enable ethtool.coalesce-adaptive-rx while ethtool.coalesce-rx-usecs is set to its default value:

     # nmcli connection modify enp1s0 ethtool.coalesce-adaptive-rx on

     Red Hat recommends that modifying the Adaptive-RX setting as follows:

     • Users concerned with low latency (sub-50us) should not enable Adaptive-RX.
     • Users concerned with throughput can probably enable Adaptive-RX with no harm. If they do not want to use the adaptive interrupt coalescence mechanism, they can try setting large values like 100us, or 250us to ethtool.coalesce-rx-usecs.
     • Users unsure about their needs should not modify this setting until an issue occurs.

6. Re-activate the connection:

   # nmcli connection up enp1s0