How to increase Ring Buffer size in Red Hat Enterprise Linux CoreOS 47 and higher?

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4
  • Red Hat Enterprise Linux CoreOS

Issue

  • How to increase the Ring Buffer Size in RHCOS due to Ring Buffer errors in ethtool -S output?

Resolution

  • Starting in OpenShift Red Hat Enterprise Linux CoreOS 47 the ethtool is now an included package. This provides the ability to check the status and modify of the NIC configuration via MachineConfig.

    The following MachineConfig will create a NetworkManager Dispatcher.d script which will run at boot. The script will check to see if the network interface card is up and if the driver is vmxnet3. If both conditions are met the script will log that the script is running and then increase the RX and TX Ring buffers to the maximum size allowed by the driver which is 4096.

    NOTE: To get the driver of your network card run the following command:
    nmcli -t -m tabular -f general.driver dev show <device_name>

  • Dispatcher.d script:

#!/bin/bash

driver=$(nmcli -t -m tabular -f general.driver dev show "${DEVICE_IFACE}")

if [[ "$2" == "up" && "${driver}" == "vmxnet3" && -f /usr/sbin/ethtool ]]; then
  logger -s "99-vsphere-increase-rx-tx-ring-buffer triggered by ${2} on device ${DEVICE_IFACE}."
  ethtool -G ${DEVICE_IFACE} rx 4096 tx 4096
fi
  • Encode the value in base64
$ vi script
$ cat script
#!/bin/bash
driver=$(nmcli -t -m tabular -f general.driver dev show "${DEVICE_IFACE}")
if [[ "$2" == "up" && "${driver}" == "vmxnet3" && -f /usr/sbin/ethtool ]]; then
  logger -s "99-vsphere-increase-rx-tx-ring-buffer triggered by ${2} on device ${DEVICE_IFACE}."
  ethtool -G ${DEVICE_IFACE} rx 4096 tx 4096
fi
$ cat script | base64
IyEvYmluL2Jhc2gKCmRyaXZlcj0kKG5tY2xpIC10IC1tIHRhYnVsYXIgLWYgZ2VuZXJhbC5kcml2
ZXIgZGV2IHNob3cgIiR7REVWSUNFX0lGQUNFfSIpCgppZiBbWyAiJDIiID09ICJ1cCIgJiYgIiR7
ZHJpdmVyfSIgPT0gInZteG5ldDMiICYmIC1mIC91c3Ivc2Jpbi9ldGh0b29sIF1dOyB0aGVuCiAg
bG9nZ2VyIC1zICI5OS12c3BoZXJlLWluY3JlYXNlLXJ4LXR4LXJpbmctYnVmZmVyIHRyaWdnZXJl
ZCBieSAkezJ9IG9uIGRldmljZSAke0RFVklDRV9JRkFDRX0uIgogIGV0aHRvb2wgLUcgJHtERVZJ
Q0VfSUZBQ0V9IHJ4IDQwOTYgdHggNDA5NgpmaQo=
  • Apply the base64 value via a MachineConfig:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-vsphere-increase-rx-tx-ring-buffer-worker
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 3.1.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKCmRyaXZlcj0kKG5tY2xpIC10IC1tIHRhYnVsYXIgLWYgZ2VuZXJhbC5kcml2ZXIgZGV2IHNob3cgIiR7REVWSUNFX0lGQUNFfSIpCgppZiBbWyAiJDIiID09ICJ1cCIgJiYgIiR7ZHJpdmVyfSIgPT0gInZteG5ldDMiICYmIC1mIC91c3Ivc2Jpbi9ldGh0b29sIF1dOyB0aGVuCiAgbG9nZ2VyIC1zICI5OS12c3BoZXJlLWluY3JlYXNlLXJ4LXR4LXJpbmctYnVmZmVyIHRyaWdnZXJlZCBieSAkezJ9IG9uIGRldmljZSAke0RFVklDRV9JRkFDRX0uIgogIGV0aHRvb2wgLUcgJHtERVZJQ0VfSUZBQ0V9IHJ4IDQwOTYgdHggNDA5NgpmaQo=
        mode: 484
        overwrite: true
        path: /etc/NetworkManager/dispatcher.d/99-vsphere-increase-rx-tx-ring-buffer
  osImageURL: ""
  • Once the MachineSet has been applied the configuration can be confirmed by running the following command from the node.
$ ethtool -g ens192
Ring parameters for ens192:
Pre-set maximums:
RX:             4096
RX Mini:        2048
RX Jumbo:       4096
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        128
RX Jumbo:       256
TX:             4096

Diagnostic Steps

  • Verify Ring Buffer issues on a Red Hat CoreOS node by running the following command:
$ sudo ethtool -S ens192
...
     Rx Queue#: 2
       LRO pkts rx: 0
       LRO byte rx: 0
       ucast pkts rx: 7903511
       ucast bytes rx: 9121802867
       mcast pkts rx: 5
       mcast bytes rx: 342
       bcast pkts rx: 2870
       bcast bytes rx: 192682
       **pkts rx OOB: 632**
       pkts rx err: 0
       drv dropped rx total: 0
          err: 0
          fcs: 0
...

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments