Chapter 32. Increasing the ring buffers to reduce a high packet drop rate

Receive ring buffers are shared between the device driver and network interface controller (NIC). The card assigns a transmit (TX) and receive (RX) ring buffer. As the name implies, the ring buffer is a circular buffer where an overflow overwrites existing data. There are two ways to move data from the NIC to the kernel, hardware interrupts and software interrupts, also called SoftIRQs.

The kernel uses the RX ring buffer to store incoming packets until they can be processed by the device driver. The device driver drains the RX ring, typically using SoftIRQs, which puts the incoming packets into a kernel data structure called an sk_buff or skb to begin its journey through the kernel and up to the application which owns the relevant socket.

The kernel uses the TX ring buffer to hold outgoing packets which are destined for the wire. These ring buffers reside at the bottom of the stack and are a crucial point at which packet drop can occur, which in turn will adversely affect network performance.

Increase the size of an Ethernet device’s ring buffers if the packet drop rate causes applications to report a loss of data, timeouts, or other issues.

Procedure

  1. Display the packet drop statistics of the interface: 

    # ethtool -S enp1s0
    ...
    rx_queue_0_drops: 97326
    rx_queue_1_drops: 63783
    ...

    Note that the output of the command depends on the network card and the driver.

    High values in counters named discard or drop indicate that the available buffer fills up faster than the kernel can process the packets and tuning the ring buffers is required.

  2. Display the maximum ring buffer sizes: 

    # ethtool -g enp1s0
 Ring parameters for enp1s0:
 Pre-set maximums:
 RX:             4096
 RX Mini:        0
 RX Jumbo:       16320
 TX:             4096
 Current hardware settings:
 RX:             255
 RX Mini:        0
 RX Jumbo:       0
 TX:             255

    If the values in the Pre-set maximums section are higher than in the Current hardware settings section, you can change the settings in the next steps.

  3. Identify the NetworkManager connection profile that uses the interface: 

    # nmcli connection show
NAME                UUID                                  TYPE      DEVICE
Example-Connection  a5eb6490-cc20-3668-81f8-0314a27f3f75  ethernet  enp1s0

  4. Update the connection profile, and increase the ring buffers:

    • To increase the RX ring buffer, enter: 

      # nmcli connection modify Example-Connection ethtool.ring-rx 4096

    • To increase the TX ring buffer, enter: 

      # nmcli connection modify Example-Connection ethtool.ring-tx 4096

  5. Reload the NetworkManager connection: 

    # nmcli connection up Example-Connection
    Important

    Depending on the driver your NIC uses, changing in the ring buffer can shortly interrupt the network connection.

Additional resources