Performance addons operator advanced configuration

Resolution

Advanced Configuration

This is an advanced guide for changing low level performance configuration on a cluster to hotfix an issue or test impact.
Instructions for editing (add/remove/change) kernel arguments,sysfs,proc parameters and are described in this section.

Default tunings

Default tunings are applied with the openshift-performance base profile, it is the base for creating a Tuned CR that would be detected by cluster node tuning operator and finally be executed by tuned.

Additional kernel arguments

When creating a performance profile CR , a default set of kernel arguments are created from the openshift-performance base profile in addition to tuned generated argument and can include for example:

nohz=on rcu_nocbs=<isolated_cores> tuned.non_isolcpus=<not_isolated_cpumask> intel_pstate=disable nosoftlockup tsc=nowatchdog intel_iommu=on iommu=pt systemd.cpu_affinity=<not_isolated_cores> isolcpus=<isolated_cores> default_hugepagesz=<DefaultHugepagesSize> hugepagesz=<hugepages_size> hugepages=<hugepages_>

Note: isolcpus is added only when balanceIsolated is disabled.

Additional kernel arguments could be added in the performance profile CR using the additionalKernelArgs field:

apiVersion: performance.openshift.io/v1
kind: PerformanceProfile
metadata:
  name: example-performanceprofile
spec:
  additionalKernelArgs:
  - "nmi_watchdog=0"
  - "audit=0"
  - "mce=off"
  - "processor.max_cstate=1"
  - "idle=poll"
  - "intel_idle.max_cstate=0"  
...

Note: These arguments will be added on top of the default arguments mentioned above. Editing these additional arguments could be done when editing the CR.

Note: This should be used for simple additions, for more complex operations see the following custom tunings section.

Additional Kubelet Arguments

To avoid KubeletConfig custom resource limitation, we introduced the experimental kubelet config snippet annotation kubeletconfig.experimental. It gives you a way to configure an additional set of kubelet parameters on top of the KubeletConfig custom resource generated by the performance-addon-operator. But take into account that it is the experimental feature, and we do not have any validations for the kubelet config snippet that you will provide under the annotation, so please make sure to set only parameters specified under the v1beta1 specification. Also, by default, the performance-addon-operator will override:
1. CPU manager policy
2. CPU manager reconcile period
3. Topology manager policy
4. Reserved CPUs
5. Memory manager policy

Please avoid specifying them and use the relevant API to configure these parameters.

To update the KubeletConfig CR, you should pass the KubeletConfig v1beta1 snippet in the json format, for example:

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: unsafe-sysctl-performanceprofile
  annotations:
    kubeletconfig.experimental: |
      {"allowedUnsafeSysctls":["net.core.somaxconn"]}
spec:
  ...

Custom tunings

To perform hotfixes on top of the tuned openshift-performance base profile, a tuned custom profile (A child profile) will be used to apply the desired changes.
This profile will inherit the base tuned profile and override its fields where needed.

For complete details about customizing tuned see : Customizing Tuned profiles.

Getting the current deployed tuned profile

In order to apply changes we will need to get the name of the deployed tuned profile that was generated by the performance addon operator:

#oc describe performanceprofile <profile name> | grep Tuned
Tuned:  <tuned namespace>/<tuned name>

Any tuned profile created for custom tunings will need to inherit from this tuned profile:

include=<tuned name>

The custom Tuned CR should be under the same tuned namespace:

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: ...
  namespace: <tuned namespace>

for example:

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: configuration-hotfixes
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=...
      # override performance addons generated tuned profile
      include=openshift-node-performance-manual
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker-cnf"
    priority: 19
    profile: configuration-hotfixes 

Important Note: The Tuned CR created for custom configuration must take precedence over the existing one.
To do so - its priority under the recommend section must be a lower number than the existing tuned CR (lower number = higher priority).

Example use cases

Initial performance profile

performance_profile.yaml
apiVersion: performance.openshift.io/v1
kind: PerformanceProfile
metadata:
  name: manual
spec:
  additionalKernelArgs:
    - "nmi_watchdog=0"
    - "audit=0"
    - "mce=off"
    - "processor.max_cstate=1"
    - "idle=poll"
    - "intel_idle.max_cstate=0"
  cpu:
    isolated: "1-3"
    reserved: "0"
  hugepages:
    defaultHugepagesSize: "1G"
    pages:
      - size: "1G"
        count: 1
        node: 0
  realTimeKernel:
    enabled: true
  numa:
    topologyPolicy: "single-numa-node"
  nodeSelector:
    node-role.kubernetes.io/worker-cnf: ""

Example of the kernel arguments generated after initial profile deployment:

sh-4.2# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-35750ad692eb3cc24529d0bc23857ad3cc29340d39912b43e3a40d255f05f740/vmlinuz-4.18.0-147.8.1.rt24.101.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.1/rhcos/35750ad692eb3cc24529d0bc23857ad3cc29340d39912b43e3a40d255f05f740/0 ignition.platform.id=gcp skew_tick=1 nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 `**idle=poll**` intel_idle.max_cstate=0 nohz=on rcu_nocbs=1-3 tuned.non_isolcpus=00000001 intel_pstate=disable nosoftlockup default_hugepagesz=1G tsc=nowatchdog intel_iommu=on iommu=pt systemd.cpu_affinity=0

Note: check /proc/cmdline on the nodes to get the current kernel arguments list.

Removing kernel argument

oc create -f- <<_EOF_
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: configuration-hotfixes
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Configuration changes profile inherited from performance created tuned

      include=openshift-node-performance-manual
      [bootloader]
      cmdline_removeKernelArgs=-idle=poll
    name: openshift-configuration-hotfixes
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker-cnf"
    priority: 19
    profile: openshift-configuration-hotfixes
_EOF_

The kernel argument is now removed:

sh-4.2# cat /proc/cmdline | grep "idle=poll"
sh-4.2# 

Changing sysctl values

sh-4.2# sysctl -n kernel.hung_task_timeout_secs
600
sh-4.2# sysctl -n kernel.nmi_watchdog          
0
sh-4.2# sysctl -n kernel.sched_rt_runtime_us
-1

oc create -f- <<_EOF_
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: configuration-hotfixes
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Configuration changes profile inherited from performance created tuned

      include=openshift-node-performance-manual
      [sysctl]
      kernel.hung_task_timeout_secs = 700  # change value from 600 to 700
      kernel.nmi_watchdog=     #set empty value
      kernel.sched_rt_runtime_us=-   # try removal


    name: openshift-configuration-hotfixes
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker-cnf"
    priority: 19
    profile: openshift-configuration-hotfixes
_EOF_

sh-4.2# sysctl -n kernel.hung_task_timeout_secs
700
sh-4.2# sysctl -n kernel.nmi_watchdog
0
sh-4.2# sysctl -n kernel.sched_rt_runtime_us
950000

RPS settings

The default RPS settings for a performance profile are to set the RPS mask as the reserved CPUs, on the host level for all network devices excluding virtual(veth) devices and physical devices(pci) and on the container level for all virtual network devices(veth).

RPS and workload hints (Openshift 4.12 and above)

When the realtime workload hint is explicitly disabled there is no need for any RPS settings to be applied since it is relevant only for the realtime use case.
The following will result in no RPS settings applied on the cluster at all:

performance_profile.yaml
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: example-performanceprofile
spec:
  workloadHints:
    realTime: false

In special cases where there is a need to explicitly specify the realtime workload hint as false but keep the RPS settings,
an override annotation performance.openshift.io/enable-rps could be added to the performance profile that will keep the default RPS settings:

performance_profile.yaml
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: example-performanceprofile
  annotations:
     performance.openshift.io/enable-rps: "true"  
spec:
  workloadHints:
    realTime: false

Enable RPS on physical devices annotation (Openshift 4.11 and above)

In case there is a need to set RPS mask for physical(pci) devices as well on the host side an override annotation performance.openshift.io/enable-physical-dev-rps to the default RPS settings could be added to the performance profile:

performance_profile.yaml
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: example-performanceprofile
  annotations:
     performance.openshift.io/enable-physical-dev-rps: "true"

Note: performance.openshift.io/enable-physical-dev-rps annotation can be applied only when realtime workload hint is
NOT explicitly set to false unless performance.openshift.io/enable-rps is set to true.