How to increase ring buffer size in Red Hat Enterprise Linux CoreOS 47?

Resolution

Starting with OpenShift 4.7 (RHCOS 47), the ethtool is now an included package. This provides the ability to check the status and modify of the NIC configuration via MachineConfig.

The following MachineConfig example will create a NetworkManager dispatcher.d script which will run at boot. The script will check to see if the network interface card is up and if the driver is vmxnet3. If both conditions are met the script will log that the script is running and then increase the RX and TX ring buffers to the maximum size allowed by the driver which is 4096.

NOTE: To get the driver of the network card run the following command in the nodes:

nmcli -t -m tabular -f general.driver dev show <device_name>

A dispatcher.d script example

IMPORTANT NOTE: before changing the ring buffer size, verify that there are ring buffer issues in the node(s) as explained in the Diagnostic Steps section.

NOTE: DEVICE_IFACE is an environmental variable passed automatically to the Dispatcher.d script by NetworkManager. You do not need to modify this variable.

• The example script with RX and TX configured to 4096:

  #!/bin/bash
  
  driver=$(nmcli -t -m tabular -f general.driver dev show "${DEVICE_IFACE}")
  
  if [[ "$2" == "up" && "${driver}" == "vmxnet3" && -f /usr/sbin/ethtool ]]; then
    logger -s "99-increase-rx-tx-ring-buffer triggered by ${2} on device ${DEVICE_IFACE}."
    ethtool -G ${DEVICE_IFACE} rx 4096 tx 4096
  fi
  

• Encode the script in base64:

  $ cat script | base64
  IyEvYmluL2Jhc2gKCmRyaXZlcj0kKG5tY2xpIC10IC1tIHRhYnVsYXIgLWYgZ2VuZXJhbC5kcml2
  ZXIgZGV2IHNob3cgIiR7REVWSUNFX0lGQUNFfSIpCgppZiBbWyAiJDIiID09ICJ1cCIgJiYgIiR7
  ZHJpdmVyfSIgPT0gInZteG5ldDMiICYmIC1mIC91c3Ivc2Jpbi9ldGh0b29sIF1dOyB0aGVuCiAg
  bG9nZ2VyIC1zICI5OS12c3BoZXJlLWluY3JlYXNlLXJ4LXR4LXJpbmctYnVmZmVyIHRyaWdnZXJl
  ZCBieSAkezJ9IG9uIGRldmljZSAke0RFVklDRV9JRkFDRX0uIgogIGV0aHRvb2wgLUcgJHtERVZJ
  Q0VfSUZBQ0V9IHJ4IDQwOTYgdHggNDA5NgpmaQo=
  

• Apply the base64 value via a MachineConfig:

  apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  metadata:
    labels:
      machineconfiguration.openshift.io/role: worker
    name: 99-increase-rx-tx-ring-buffer-worker
  spec:
    config:
      ignition:
        config: {}
        security:
          tls: {}
        timeouts: {}
        version: 3.1.0
      networkd: {}
      passwd: {}
      storage:
        files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKCmRyaXZlcj0kKG5tY2xpIC10IC1tIHRhYnVsYXIgLWYgZ2VuZXJhbC5kcml2ZXIgZGV2IHNob3cgIiR7REVWSUNFX0lGQUNFfSIpCgppZiBbWyAiJDIiID09ICJ1cCIgJiYgIiR7ZHJpdmVyfSIgPT0gInZteG5ldDMiICYmIC1mIC91c3Ivc2Jpbi9ldGh0b29sIF1dOyB0aGVuCiAgbG9nZ2VyIC1zICI5OS12c3BoZXJlLWluY3JlYXNlLXJ4LXR4LXJpbmctYnVmZmVyIHRyaWdnZXJlZCBieSAkezJ9IG9uIGRldmljZSAke0RFVklDRV9JRkFDRX0uIgogIGV0aHRvb2wgLUcgJHtERVZJQ0VfSUZBQ0V9IHJ4IDQwOTYgdHggNDA5NgpmaQo=
          mode: 484
          overwrite: true
          path: /etc/NetworkManager/dispatcher.d/99-vsphere-increase-rx-tx-ring-buffer
    osImageURL: ""
  

• Apply the above MachineConfig (for example, from a file called 99-increase-rx-tx-ring-buffer-worker.yaml):

  $ oc apply -f 99-increase-rx-tx-ring-buffer-worker.yaml
  

• Once the MachineConfig has been applied, the configuration can be confirmed by running the following command from the node:

  $ ethtool -g <device_name>
  Ring parameters for ens192:
  Pre-set maximums:
  RX:             4096
  RX Mini:        2048
  RX Jumbo:       4096
  TX:             4096
  Current hardware settings:
  RX:             4096
  RX Mini:        128
  RX Jumbo:       256
  TX:             4096