1. Create a YAML configuration file, cluster-monitoring-config.yaml. For example:

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       prometheusK8s:
         retention: {{PROMETHEUS_RETENTION_PERIOD}} 1
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} 2
             resources:
               requests:
                 storage: {{PROMETHEUS_STORAGE_SIZE}} 3
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} 4
             resources:
               requests:
                 storage: {{ALERTMANAGER_STORAGE_SIZE}} 5
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring

   1

   The default value of Prometheus retention is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.

   2 4

   The storage class for your cluster.

   3

   A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

   5

   A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

2. Add values for the retention period, storage class, and storage sizes.
3. Save the file.

4. Apply the changes by running:

   $ oc create -f cluster-monitoring-config.yaml

Procedure

1. Attach the new disk to the cluster and verify that the disk is detected in the node by using the lsblk command in a debug shell:

   $ oc debug node/<node_name>

   # lsblk

   Note the device name of the new disk reported by the lsblk command.

2. Create a MachineConfig YAML file named etcd-mc.yml with contents such as the following, replacing instances of <new_disk_name> with the noted device name:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 98-var-lib-etcd
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - contents: |
             [Unit]
             Description=Make File System on /dev/<new_disk_name>
             DefaultDependencies=no
             BindsTo=dev-<new_disk_name>.device
             After=dev-<new_disk_name>.device var.mount
             Before=systemd-fsck@dev-<new_disk_name>.service
   
             [Service]
             Type=oneshot
             RemainAfterExit=yes
             ExecStart=/usr/lib/systemd/systemd-makefs xfs /dev/<new_disk_name>
             TimeoutSec=0
   
             [Install]
             WantedBy=var-lib-containers.mount
           enabled: true
           name: systemd-mkfs@dev-<new_disk_name>.service
         - contents: |
             [Unit]
             Description=Mount /dev/<new_disk_name> to /var/lib/etcd
             Before=local-fs.target
             Requires=systemd-mkfs@dev-<new_disk_name>.service
             After=systemd-mkfs@dev-<new_disk_name>.service var.mount
   
             [Mount]
             What=/dev/<new_disk_name>
             Where=/var/lib/etcd
             Type=xfs
             Options=defaults,prjquota
   
             [Install]
             WantedBy=local-fs.target
           enabled: true
           name: var-lib-etcd.mount
         - contents: |
             [Unit]
             Description=Sync etcd data if new mount is empty
             DefaultDependencies=no
             After=var-lib-etcd.mount var.mount
             Before=crio.service
   
             [Service]
             Type=oneshot
             RemainAfterExit=yes
             ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
             ExecStart=semanage fcontext -a -e /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
             ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
             TimeoutSec=0
   
             [Install]
             WantedBy=multi-user.target graphical.target
           enabled: true
           name: sync-var-lib-etcd-to-etcd.service
         - contents: |
             [Unit]
             Description=Restore recursive SELinux security contexts
             DefaultDependencies=no
             After=var-lib-etcd.mount
             Before=crio.service
   
             [Service]
             Type=oneshot
             RemainAfterExit=yes
             ExecStart=/sbin/restorecon -R /var/lib/etcd/
             TimeoutSec=0
   
             [Install]
             WantedBy=multi-user.target graphical.target
           enabled: true
           name: restorecon-var-lib-etcd.service

3. Log in to the cluster as a user with cluster-admin privileges and create the machine configuration:

   $ oc login -u <username> -p <password>

   $ oc create -f etcd-mc.yml

   The nodes are updated and rebooted. After the reboot completes, the following events occur:

   • An XFS file system is created on the specified disk.
   • The disk mounts to /var/lib/etcd.
   • The content from /sysroot/ostree/deploy/rhcos/var/lib/etcd syncs to /var/lib/etcd.
   • A restore of SELinux labels is forced for /var/lib/etcd.
   • The old content is not removed.

4. After the nodes are on a separate disk, update the etcd-mc.yml file with contents such as the following, replacing instances of <new_disk_name> with the noted device name:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 98-var-lib-etcd
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - contents: |
             [Unit]
             Description=Mount /dev/<new_disk_name> to /var/lib/etcd
             Before=local-fs.target
             Requires=systemd-mkfs@dev-<new_disk_name>.service
             After=systemd-mkfs@dev-<new_disk_name>.service var.mount
   
             [Mount]
             What=/dev/<new_disk_name>
             Where=/var/lib/etcd
             Type=xfs
             Options=defaults,prjquota
   
             [Install]
             WantedBy=local-fs.target
           enabled: true
           name: var-lib-etcd.mount

5. Apply the modified version that removes the logic for creating and syncing the device to prevent the nodes from rebooting:

   $ oc replace -f etcd-mc.yml

1. gp3 disks with a baseline performance of 3000 IOPS and 125 MiB per second are used for control plane/etcd nodes because etcd is latency sensitive. gp3 volumes do not use burst performance.
2. Infra nodes are used to host Monitoring, Ingress, and Registry components to ensure they have enough resources to run at large scale.
3. Workload node is dedicated to run performance and scalability workload generators.
4. Larger disk size is used so that there is enough space to store the large amounts of data that is collected during the performance and scalability test run.
5. Cluster is scaled in iterations and performance and scalability tests are executed at the specified node counts.

1. io1 disks with 120 / 10 IOPS per GiB are used for control plane/etcd nodes as etcd is I/O intensive and latency sensitive.
2. Infra nodes are used to host Monitoring, Ingress, and Registry components to ensure they have enough resources to run at large scale.
3. Workload node is dedicated to run performance and scalability workload generators.
4. Larger disk size is used so that there is enough space to store the large amounts of data that is collected during the performance and scalability test run.
5. Cluster is scaled in iterations.

1. Nodes are distributed between two logical control units (LCUs) to optimize disk I/O load of the control plane/etcd nodes as etcd is I/O intensive and latency sensitive. Etcd I/O demand should not interfere with other workloads.
2. Four compute nodes are used for the tests running several iterations with 100/250/500 pods at the same time. First, idling pods were used to evaluate if pods can be instanced. Next, a network and CPU demanding client/server workload were used to evaluate the stability of the system under stress. Client and server pods were pairwise deployed and each pair was spread over two compute nodes.
3. No separate workload node was used. The workload simulates a microservice workload between two compute nodes.
4. Physical number of processors used is six Integrated Facilities for Linux (IFLs).
5. Total physical memory used is 512 GiB.

Procedure

1. Copy the following MCO sample profile into a YAML file. For example, enable-rfs.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 50-enable-rfs
   spec:
     config:
       ignition:
         version: 2.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=US-ASCII,%23%20turn%20on%20Receive%20Flow%20Steering%20%28RFS%29%20for%20all%20network%20interfaces%0ASUBSYSTEM%3D%3D%22net%22%2C%20ACTION%3D%3D%22add%22%2C%20RUN%7Bprogram%7D%2B%3D%22/bin/bash%20-c%20%27for%20x%20in%20/sys/%24DEVPATH/queues/rx-%2A%3B%20do%20echo%208192%20%3E%20%24x/rps_flow_cnt%3B%20%20done%27%22%0A
           filesystem: root
           mode: 0644
           path: /etc/udev/rules.d/70-persistent-net.rules
         - contents:
             source: data:text/plain;charset=US-ASCII,%23%20define%20sock%20flow%20enbtried%20for%20%20Receive%20Flow%20Steering%20%28RFS%29%0Anet.core.rps_sock_flow_entries%3D8192%0A
           filesystem: root
           mode: 0644
           path: /etc/sysctl.d/95-enable-rps.conf

2. Create the MCO profile:

   $ oc create -f enable-rfs.yaml

3. Verify that an entry named 50-enable-rfs is listed:

   $ oc get mc

4. To deactivate, enter:

   $ oc delete mc 50-enable-rfs

Procedure

1. Copy the following MCO sample profile into a YAML file for the control plane node. For example, 05-master-kernelarg-hpav.yaml:

   $ cat 05-master-kernelarg-hpav.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 05-master-kernelarg-hpav
   spec:
     config:
       ignition:
         version: 3.1.0
     kernelArguments:
       - rd.dasd=800-805

2. Copy the following MCO sample profile into a YAML file for the compute node. For example, 05-worker-kernelarg-hpav.yaml:

   $ cat 05-worker-kernelarg-hpav.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 05-worker-kernelarg-hpav
   spec:
     config:
       ignition:
         version: 3.1.0
     kernelArguments:
       - rd.dasd=800-805

   Note

   You must modify the rd.dasd arguments to fit the device IDs.

3. Create the MCO profiles:

   $ oc create -f 05-master-kernelarg-hpav.yaml

   $ oc create -f 05-worker-kernelarg-hpav.yaml

4. To deactivate, enter:

   $ oc delete -f 05-master-kernelarg-hpav.yaml

   $ oc delete -f 05-worker-kernelarg-hpav.yaml

Procedure

1. Open /etc/libvirt/qemu.conf with an editor of your choice.
2. Go to the cgroup_controllers line.
3. Duplicate the entire line and remove the leading number sign (#) from the copy.

4. Remove the cpuset entry, as follows:

   cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuacct" ]

5. For the new setting to take effect, you must restart the libvirtd daemon:

   1. Stop all virtual machines.

   2. Run the following command:

      # systemctl restart libvirtd

   3. Restart the virtual machines.

Procedure

1. Create a namespace for the NUMA Resources Operator:

   1. Save the following YAML in the nro-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-numaresources

   2. Create the Namespace CR by running the following command:

      $ oc create -f nro-namespace.yaml

2. Create the Operator group for the NUMA Resources Operator:

   1. Save the following YAML in the nro-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: numaresources-operator
        namespace: openshift-numaresources
      spec:
        targetNamespaces:
        - openshift-numaresources

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f nro-operatorgroup.yaml

3. Create the subscription for the NUMA Resources Operator:

   1. Save the following YAML in the nro-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: numaresources-operator
        namespace: openshift-numaresources
      spec:
        channel: "4.12"
        name: numaresources-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription CR by running the following command:

      $ oc create -f nro-sub.yaml

Verification

1. Verify that the installation succeeded by inspecting the CSV resource in the openshift-numaresources namespace. Run the following command:

   $ oc get csv -n openshift-numaresources

   Example output

   NAME                             DISPLAY                  VERSION   REPLACES   PHASE
   numaresources-operator.v4.12.2   numaresources-operator   4.12.2               Succeeded

Procedure

1. Create a namespace for the NUMA Resources Operator:

   1. In the OpenShift Container Platform web console, click Administration → Namespaces.
   2. Click Create Namespace, enter openshift-numaresources in the Name field, and then click Create.

2. Install the NUMA Resources Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose NUMA Resources Operator from the list of available Operators, and then click Install.
   3. In the Installed Namespaces field, select the openshift-numaresources namespace, and then click Install.

3. Optional: Verify that the NUMA Resources Operator installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that NUMA Resources Operator is listed in the openshift-numaresources namespace with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the default project.

Procedure

1. Create the MachineConfigPool custom resource that enables custom kubelet configurations for worker nodes:

   1. Save the following YAML in the nro-machineconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        labels:
          cnf-worker-tuning: enabled
          machineconfiguration.openshift.io/mco-built-in: ""
          pools.operator.machineconfiguration.openshift.io/worker: ""
        name: worker
      spec:
        machineConfigSelector:
          matchLabels:
            machineconfiguration.openshift.io/role: worker
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""

   2. Create the MachineConfigPool CR by running the following command:

      $ oc create -f nro-machineconfig.yaml

2. Create the NUMAResourcesOperator custom resource:

   1. Save the following YAML in the nrop.yaml file:

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesOperator
      metadata:
        name: numaresourcesoperator
      spec:
        nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              pools.operator.machineconfiguration.openshift.io/worker: "" 1

      1

      Should match the label applied to worker nodes in the related MachineConfigPool CR.

   2. Create the NUMAResourcesOperator CR by running the following command:

      $ oc create -f nrop.yaml

Procedure

1. Create the KubeletConfig custom resource that configures the pod admittance policy for the machine profile:

   1. Save the following YAML in the nro-kubeletconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: cnf-worker-tuning
      spec:
        machineConfigPoolSelector:
          matchLabels:
            cnf-worker-tuning: enabled
        kubeletConfig:
          cpuManagerPolicy: "static" 1
          cpuManagerReconcilePeriod: "5s"
          reservedSystemCPUs: "0,1"
          memoryManagerPolicy: "Static" 2
          evictionHard:
            memory.available: "100Mi"
          kubeReserved:
            memory: "512Mi"
          reservedMemory:
            - numaNode: 0
              limits:
                memory: "1124Mi"
          systemReserved:
            memory: "512Mi"
          topologyManagerPolicy: "single-numa-node" 3
          topologyManagerScope: "pod"

      1

      For cpuManagerPolicy, static must use a lowercase s.

      2

      For memoryManagerPolicy, Static must use an uppercase S.

      3

      topologyManagerPolicy must be set to single-numa-node.

   2. Create the KubeletConfig custom resource (CR) by running the following command:

      $ oc create -f nro-kubeletconfig.yaml

2. Create the NUMAResourcesScheduler custom resource that deploys the NUMA-aware custom pod scheduler:

   1. Save the following YAML in the nro-scheduler.yaml file:

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesScheduler
      metadata:
        name: numaresourcesscheduler
      spec:
        imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.12"

   2. Create the NUMAResourcesScheduler CR by running the following command:

      $ oc create -f nro-scheduler.yaml

Procedure

1. Get the name of the NUMA-aware scheduler that is deployed in the cluster by running the following command:

   $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'

   Example output

   topo-aware-scheduler

2. Create a Deployment CR that uses scheduler named topo-aware-scheduler, for example:

   1. Save the following YAML in the nro-deployment.yaml file:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: numa-deployment-1
        namespace: openshift-numaresources
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: test
        template:
          metadata:
            labels:
              app: test
          spec:
            schedulerName: topo-aware-scheduler 1
            containers:
            - name: ctnr
              image: quay.io/openshifttest/hello-openshift:openshift
              imagePullPolicy: IfNotPresent
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "10"
                requests:
                  memory: "100Mi"
                  cpu: "10"
            - name: ctnr2
              image: registry.access.redhat.com/rhel:latest
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: [ "while true; do sleep 1h; done;" ]
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "8"
                requests:
                  memory: "100Mi"
                  cpu: "8"

      1

      schedulerName must match the name of the NUMA-aware scheduler that is deployed in your cluster, for example topo-aware-scheduler.

   2. Create the Deployment CR by running the following command:

      $ oc create -f nro-deployment.yaml

Verification

1. Verify that the deployment was successful:

   $ oc get pods -n openshift-numaresources

   Example output

   NAME                                                READY   STATUS    RESTARTS   AGE
   numa-deployment-1-56954b7b46-pfgw8                  2/2     Running   0          129m
   numaresources-controller-manager-7575848485-bns4s   1/1     Running   0          15h
   numaresourcesoperator-worker-dvj4n                  2/2     Running   0          18h
   numaresourcesoperator-worker-lcg4t                  2/2     Running   0          16h
   secondary-scheduler-56994cf6cf-7qf4q                1/1     Running   0          18h

2. Verify that the topo-aware-scheduler is scheduling the deployed pod by running the following command:

   $ oc describe pod numa-deployment-1-56954b7b46-pfgw8 -n openshift-numaresources

   Example output

   Events:
     Type    Reason          Age   From                  Message
     ----    ------          ----  ----                  -------
     Normal  Scheduled       130m  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-56954b7b46-pfgw8 to compute-0.example.com

   Note

   Deployments that request more resources than is available for scheduling will fail with a MinimumReplicasUnavailable error. The deployment succeeds when the required resources become available. Pods remain in the Pending state until the required resources are available.

3. Verify that the expected allocated resources are listed for the node. Run the following command:

   $ oc describe noderesourcetopologies.topology.node.k8s.io

   Example output

   ...
   
   Zones:
     Costs:
       Name:   node-0
       Value:  10
       Name:   node-1
       Value:  21
     Name:     node-0
     Resources:
       Allocatable:  39
       Available:    21 1
       Capacity:     40
       Name:         cpu
       Allocatable:  6442450944
       Available:    6442450944
       Capacity:     6442450944
       Name:         hugepages-1Gi
       Allocatable:  134217728
       Available:    134217728
       Capacity:     134217728
       Name:         hugepages-2Mi
       Allocatable:  262415904768
       Available:    262206189568
       Capacity:     270146007040
       Name:         memory
     Type:           Node

   1

   The Available capacity is reduced because of the resources that have been allocated to the guaranteed pod.

   Resources consumed by guaranteed pods are subtracted from the available node resources listed under noderesourcetopologies.topology.node.k8s.io.

4. Resource allocations for pods with a Best-effort or Burstable quality of service (qosClass) are not reflected in the NUMA node resources under noderesourcetopologies.topology.node.k8s.io. If a pod’s consumed resources are not reflected in the node resource calculation, verify that the pod has qosClass of Guaranteed by running the following command:

   $ oc get pod <pod_name> -n <pod_namespace> -o jsonpath="{ .status.qosClass }"

   Example output

   Guaranteed

Procedure

1. Optional: Create the MachineConfigPool custom resource that enables custom kubelet configurations for worker nodes:

   Note

   By default, OpenShift Container Platform creates a MachineConfigPool resource for worker nodes in the cluster. You can create a custom MachineConfigPool resource if required.

   1. Save the following YAML in the nro-machineconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        labels:
          cnf-worker-tuning: enabled
          machineconfiguration.openshift.io/mco-built-in: ""
          pools.operator.machineconfiguration.openshift.io/worker: ""
        name: worker
      spec:
        machineConfigSelector:
          matchLabels:
            machineconfiguration.openshift.io/role: worker
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""

   2. Create the MachineConfigPool CR by running the following command:

      $ oc create -f nro-machineconfig.yaml

2. Create the NUMAResourcesOperator custom resource:

   1. Save the following YAML in the nrop.yaml file:

      apiVersion: nodetopology.openshift.io/v1
      kind: NUMAResourcesOperator
      metadata:
        name: numaresourcesoperator
      spec:
        nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              pools.operator.machineconfiguration.openshift.io/worker: "" 1

      1

      Should match the label applied to worker nodes in the related MachineConfigPool CR.

   2. Create the NUMAResourcesOperator CR by running the following command:

      $ oc create -f nrop.yaml

Procedure

1. Create the KubeletConfig custom resource that configures the pod admittance policy for the machine profile:

   1. Save the following YAML in the nro-kubeletconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: cnf-worker-tuning
      spec:
        machineConfigPoolSelector:
          matchLabels:
            cnf-worker-tuning: enabled
        kubeletConfig:
          cpuManagerPolicy: "static" 1
          cpuManagerReconcilePeriod: "5s"
          reservedSystemCPUs: "0,1"
          memoryManagerPolicy: "Static" 2
          evictionHard:
            memory.available: "100Mi"
          reservedMemory:
            - numaNode: 0
              limits:
                memory: "1124Mi"
          systemReserved:
            memory: "512Mi"
          topologyManagerPolicy: "single-numa-node" 3
          topologyManagerScope: "pod"

      1

      For cpuManagerPolicy, static must use a lowercase s.

      2

      For memoryManagerPolicy, Static must use an uppercase S.

      3

      topologyManagerPolicy must be set to single-numa-node.

   2. Create the KubeletConfig custom resource (CR) by running the following command:

      $ oc create -f nro-kubeletconfig.yaml

2. Create the NUMAResourcesScheduler custom resource that deploys the NUMA-aware custom pod scheduler:

   1. Save the following YAML in the nro-scheduler.yaml file:

      apiVersion: nodetopology.openshift.io/v1
      kind: NUMAResourcesScheduler
      metadata:
        name: numaresourcesscheduler
      spec:
        imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.12"
        cacheResyncPeriod: "5s" 1

      1

      Enter an interval value in seconds for synchronization of the scheduler cache. A value of 5s is typical for most implementations.

      Note

      • Enable the cacheResyncPeriod specification to help the NUMA Resource Operator report more exact resource availability by monitoring pending resources on nodes and synchronizing this information in the scheduler cache at a defined interval. This also helps to minimize Topology Affinity Error errors because of sub-optimal scheduling decisions. The lower the interval the greater the network load. The cacheResyncPeriod specification is disabled by default.
      • Setting a value of Enabled for the podsFingerprinting specification in the NUMAResourcesOperator CR is a requirement for the implementation of the cacheResyncPeriod specification.

   2. Create the NUMAResourcesScheduler CR by running the following command:

      $ oc create -f nro-scheduler.yaml

Procedure

1. Get the name of the NUMA-aware scheduler that is deployed in the cluster by running the following command:

   $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'

   Example output

   topo-aware-scheduler

2. Create a Deployment CR that uses scheduler named topo-aware-scheduler, for example:

   1. Save the following YAML in the nro-deployment.yaml file:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: numa-deployment-1
        namespace: openshift-numaresources
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: test
        template:
          metadata:
            labels:
              app: test
          spec:
            schedulerName: topo-aware-scheduler 1
            containers:
            - name: ctnr
              image: quay.io/openshifttest/hello-openshift:openshift
              imagePullPolicy: IfNotPresent
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "10"
                requests:
                  memory: "100Mi"
                  cpu: "10"
            - name: ctnr2
              image: registry.access.redhat.com/rhel:latest
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: [ "while true; do sleep 1h; done;" ]
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "8"
                requests:
                  memory: "100Mi"
                  cpu: "8"

      1

      schedulerName must match the name of the NUMA-aware scheduler that is deployed in your cluster, for example topo-aware-scheduler.

   2. Create the Deployment CR by running the following command:

      $ oc create -f nro-deployment.yaml

Verification

1. Verify that the deployment was successful:

   $ oc get pods -n openshift-numaresources

   Example output

   NAME                                                READY   STATUS    RESTARTS   AGE
   numa-deployment-1-56954b7b46-pfgw8                  2/2     Running   0          129m
   numaresources-controller-manager-7575848485-bns4s   1/1     Running   0          15h
   numaresourcesoperator-worker-dvj4n                  2/2     Running   0          18h
   numaresourcesoperator-worker-lcg4t                  2/2     Running   0          16h
   secondary-scheduler-56994cf6cf-7qf4q                1/1     Running   0          18h

2. Verify that the topo-aware-scheduler is scheduling the deployed pod by running the following command:

   $ oc describe pod numa-deployment-1-56954b7b46-pfgw8 -n openshift-numaresources

   Example output

   Events:
     Type    Reason          Age   From                  Message
     ----    ------          ----  ----                  -------
     Normal  Scheduled       130m  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-56954b7b46-pfgw8 to compute-0.example.com

   Note

   Deployments that request more resources than are available for scheduling will fail with a MinimumReplicasUnavailable error. The deployment succeeds when the required resources become available. Pods remain in the Pending state until the required resources are available.

3. Verify that the expected allocated resources are listed for the node.

   1. Identify the node that is running the deployment pod by running the following command, replacing <namespace> with the namespace you specified in the Deployment CR:

      $ oc get pods -n <namespace> -o wide

      Example output

      NAME                                 READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
      numa-deployment-1-65684f8fcc-bw4bw   0/2     Running   0          82m   10.128.2.50   worker-0   <none>  <none>

   2. Run the following command, replacing <node_name> with the name of that node that is running the deployment pod:

      $ oc describe noderesourcetopologies.topology.node.k8s.io <node_name>

      Example output

      ...
      
      Zones:
        Costs:
          Name:   node-0
          Value:  10
          Name:   node-1
          Value:  21
        Name:     node-0
        Resources:
          Allocatable:  39
          Available:    21 1
          Capacity:     40
          Name:         cpu
          Allocatable:  6442450944
          Available:    6442450944
          Capacity:     6442450944
          Name:         hugepages-1Gi
          Allocatable:  134217728
          Available:    134217728
          Capacity:     134217728
          Name:         hugepages-2Mi
          Allocatable:  262415904768
          Available:    262206189568
          Capacity:     270146007040
          Name:         memory
        Type:           Node

      1

      The Available capacity is reduced because of the resources that have been allocated to the guaranteed pod.

      Resources consumed by guaranteed pods are subtracted from the available node resources listed under noderesourcetopologies.topology.node.k8s.io.

4. Resource allocations for pods with a Best-effort or Burstable quality of service (qosClass) are not reflected in the NUMA node resources under noderesourcetopologies.topology.node.k8s.io. If a pod’s consumed resources are not reflected in the node resource calculation, verify that the pod has qosClass of Guaranteed and the CPU request is an integer value, not a decimal value. You can verify the that the pod has a qosClass of Guaranteed by running the following command:

   $ oc get pod <pod_name> -n <pod_namespace> -o jsonpath="{ .status.qosClass }"

   Example output

   Guaranteed

Procedure

1. Delete the currently running NUMAResourcesScheduler resource:

   1. Get the active NUMAResourcesScheduler by running the following command:

      $ oc get NUMAResourcesScheduler

      Example output

      NAME                     AGE
      numaresourcesscheduler   90m

   2. Delete the secondary scheduler resource by running the following command:

      $ oc delete NUMAResourcesScheduler numaresourcesscheduler

      Example output

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted

2. Save the following YAML in the file nro-scheduler-debug.yaml. This example changes the log level to Debug:

   apiVersion: nodetopology.openshift.io/v1alpha1
   kind: NUMAResourcesScheduler
   metadata:
     name: numaresourcesscheduler
   spec:
     imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.12"
     logLevel: Debug

3. Create the updated Debug logging NUMAResourcesScheduler resource by running the following command:

   $ oc create -f nro-scheduler-debug.yaml

   Example output

   numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created

Verification steps

1. Check that the NUMA-aware scheduler was successfully deployed:

   1. Run the following command to check that the CRD is created succesfully:

      $ oc get crd | grep numaresourcesschedulers

      Example output

      NAME                                                              CREATED AT
      numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z

   2. Check that the new custom scheduler is available by running the following command:

      $ oc get numaresourcesschedulers.nodetopology.openshift.io

      Example output

      NAME                     AGE
      numaresourcesscheduler   3h26m

2. Check that the logs for the scheduler shows the increased log level:

   1. Get the list of pods running in the openshift-numaresources namespace by running the following command:

      $ oc get pods -n openshift-numaresources

      Example output

      NAME                                               READY   STATUS    RESTARTS   AGE
      numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
      numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
      numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
      secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m

   2. Get the logs for the secondary scheduler pod by running the following command:

      $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources

      Example output

      ...
      I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
      I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
      I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
      I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
      I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"

Procedure

1. Get the daemonsets managed by the NUMA Resources Operator. Each daemonset has a corresponding nodeGroup in the NUMAResourcesOperator CR. Run the following command:

   $ oc get numaresourcesoperators.nodetopology.openshift.io numaresourcesoperator -o jsonpath="{.status.daemonsets[0]}"

   Example output

   {"name":"numaresourcesoperator-worker","namespace":"openshift-numaresources"}

2. Get the label for the daemonset of interest using the value for name from the previous step:

   $ oc get ds -n openshift-numaresources numaresourcesoperator-worker -o jsonpath="{.spec.selector.matchLabels}"

   Example output

   {"name":"resource-topology"}

3. Get the pods using the resource-topology label by running the following command:

   $ oc get pods -n openshift-numaresources -l name=resource-topology -o wide

   Example output

   NAME                                 READY   STATUS    RESTARTS   AGE    IP            NODE
   numaresourcesoperator-worker-5wm2k   2/2     Running   0          2d1h   10.135.0.64   compute-0.example.com
   numaresourcesoperator-worker-pb75c   2/2     Running   0          2d1h   10.132.2.33   compute-1.example.com

4. Examine the logs of the resource-topology-exporter container running on the worker pod that corresponds to the node you are troubleshooting. Run the following command:

   $ oc logs -n openshift-numaresources -c resource-topology-exporter numaresourcesoperator-worker-pb75c

   Example output

   I0221 13:38:18.334140       1 main.go:206] using sysinfo:
   reservedCpus: 0,1
   reservedMemory:
     "0": 1178599424
   I0221 13:38:18.334370       1 main.go:67] === System information ===
   I0221 13:38:18.334381       1 sysinfo.go:231] cpus: reserved "0-1"
   I0221 13:38:18.334493       1 sysinfo.go:237] cpus: online "0-103"
   I0221 13:38:18.546750       1 main.go:72]
   cpus: allocatable "2-103"
   hugepages-1Gi:
     numa cell 0 -> 6
     numa cell 1 -> 1
   hugepages-2Mi:
     numa cell 0 -> 64
     numa cell 1 -> 128
   memory:
     numa cell 0 -> 45758Mi
     numa cell 1 -> 48372Mi

Procedure

1. Compare the values for spec.machineConfigPoolSelector.matchLabels in kubeletconfig and metadata.labels in the MachineConfigPool (mcp) worker CR using the following commands:

   1. Check the kubeletconfig labels by running the following command:

      $ oc get kubeletconfig -o yaml

      Example output

      machineConfigPoolSelector:
        matchLabels:
          cnf-worker-tuning: enabled

   2. Check the mcp labels by running the following command:

      $ oc get mcp worker -o yaml

      Example output

      labels:
        machineconfiguration.openshift.io/mco-built-in: ""
        pools.operator.machineconfiguration.openshift.io/worker: ""

      The cnf-worker-tuning: enabled label is not present in the MachineConfigPool object.

2. Edit the MachineConfigPool CR to include the missing label, for example:

   $ oc edit mcp worker -o yaml

   Example output

   labels:
     machineconfiguration.openshift.io/mco-built-in: ""
     pools.operator.machineconfiguration.openshift.io/worker: ""
     cnf-worker-tuning: enabled

3. Apply the label changes and wait for the cluster to apply the updated configuration. Run the following command:

1. NetApp NFS supports dynamic PV provisioning when using the Trident plugin.

Procedure

1. Create a file called mount_namespace_config.yaml with the following YAML:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 99-kubens-master
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: kubens.service
   ---
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-kubens-worker
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: kubens.service

2. Apply the mount namespace MachineConfig CR by running the following command:

   $ oc apply -f mount_namespace_config.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-kubens-master created
   machineconfig.machineconfiguration.openshift.io/99-kubens-worker created

3. The MachineConfig CR can take up to 30 minutes to finish being applied in the cluster. You can check the status of the MachineConfig CR by running the following command:

   $ oc get mcp

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-03d4bc4befb0f4ed3566a2c8f7636751   False     True       False      3              0                   0                     0                      45m
   worker   rendered-worker-10577f6ab0117ed1825f8af2ac687ddf   False     True       False      3              1                   1

4. Wait for the MachineConfig CR to be applied successfully across all control plane and worker nodes after running the following command:

   $ oc wait --for=condition=Updated mcp --all --timeout=30m

   Example output

   machineconfigpool.machineconfiguration.openshift.io/master condition met
   machineconfigpool.machineconfiguration.openshift.io/worker condition met

1. Open a debug shell to the cluster host:

   $ oc debug node/<node_name>

2. Open a chroot session:

   sh-4.4# chroot /host

3. Check the systemd mount namespace:

   sh-4.4# readlink /proc/1/ns/mnt

   Example output

   mnt:[4026531953]

4. Check kubelet mount namespace:

   sh-4.4# readlink /proc/$(pgrep kubelet)/ns/mnt

   Example output

   mnt:[4026531840]

5. Check the CRI-O mount namespace:

   sh-4.4# readlink /proc/$(pgrep crio)/ns/mnt

   Example output

   mnt:[4026531840]

Procedure

1. Open a remote SSH shell to the cluster host. For example:

   $ ssh core@<node_name>

2. Run commands using the provided kubensenter script as the root user. To run a single command inside the Kubernetes namespace, provide the command and any arguments to the kubensenter script. For example, to run the findmnt command inside the Kubernetes namespace, run the following command:

   [core@control-plane-1 ~]$ sudo kubensenter findmnt

   Example output

   kubensenter: Autodetect: kubens.service namespace found at /run/kubens/mnt
   TARGET                                SOURCE                 FSTYPE     OPTIONS
   /                                     /dev/sda4[/ostree/deploy/rhcos/deploy/32074f0e8e5ec453e56f5a8a7bc9347eaa4172349ceab9c22b709d9d71a3f4b0.0]
   |                                                            xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
                                         shm                    tmpfs
   ...

3. To start a new interactive shell inside the Kubernetes namespace, run the kubensenter script without any arguments:

   [core@control-plane-1 ~]$ sudo kubensenter

   Example output

   kubensenter: Autodetect: kubens.service namespace found at /run/kubens/mnt

Procedure

1. In the web console, navigate to Compute → Bare Metal Hosts.
2. Select Add Host → New with Dialog.
3. Specify a unique name for the new bare metal host.
4. Set the Boot MAC address.
5. Set the Baseboard Management Console (BMC) Address.
6. Enter the user credentials for the host’s baseboard management controller (BMC).
7. Select to power on the host after creation, and select Create.
8. Scale up the number of replicas to match the number of available bare metal hosts. Navigate to Compute → MachineSets, and increase the number of machine replicas in the cluster by selecting Edit Machine count from the Actions drop-down menu.

Procedure

1. In the web console, navigate to Compute → Bare Metal Hosts.
2. Select Add Host → New from YAML.

3. Copy and paste the below YAML, modifying the relevant fields with the details of your host:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: <bare_metal_host_name>
   spec:
     online: true
     bmc:
       address: <bmc_address>
       credentialsName: <secret_credentials_name>  1
       disableCertificateVerification: True 2
     bootMACAddress: <host_boot_mac_address>

   1

   credentialsName must reference a valid Secret CR. The baremetal-operator cannot manage the bare metal host without a valid Secret referenced in the credentialsName. For more information about secrets and how to create them, see Understanding secrets.

   2

   Setting disableCertificateVerification to true disables TLS host validation between the cluster and the baseboard management controller (BMC).

4. Select Create to save the YAML and create the new bare metal host.

5. Scale up the number of replicas to match the number of available bare metal hosts. Navigate to Compute → MachineSets, and increase the number of machines in the cluster by selecting Edit Machine count from the Actions drop-down menu.

   Note

   You can also manage the number of bare metal nodes using the oc scale command and the appropriate bare metal compute machine set.

Procedure

1. Annotate the compute machine set that you want to configure for automatic scaling by adding the metal3.io/autoscale-to-hosts annotation. Replace <machineset> with the name of the compute machine set.

   $ oc annotate machineset <machineset> -n openshift-machine-api 'metal3.io/autoscale-to-hosts=<any_value>'

   Wait for the new scaled machines to start.

Procedure

1. Annotate the compute machine set that you want to remove from the provisioner node by adding the baremetalhost.metal3.io/detached annotation.

   $ oc annotate machineset <machineset> -n openshift-machine-api 'baremetalhost.metal3.io/detached'

   Wait for the new machines to start.

   Note

   When you use a BareMetalHost object to create a machine in the cluster and labels or selectors are subsequently changed on the BareMetalHost, the BareMetalHost object continues be counted against the MachineSet that the Machine object was created from.

2. In the provisioning use case, remove the annotation after the reboot is complete by using the following command:

   $ oc annotate machineset <machineset> -n openshift-machine-api 'baremetalhost.metal3.io/detached-'

Procedure

1. Create a namespace for the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-bare-metal-events
        labels:
          name: openshift-bare-metal-events
          openshift.io/cluster-monitoring: "true"

   2. Create the Namespace CR:

      $ oc create -f bare-metal-events-namespace.yaml

2. Create an Operator group for the Bare Metal Event Relay Operator.

   1. Save the following YAML in the bare-metal-events-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: bare-metal-event-relay-group
        namespace: openshift-bare-metal-events
      spec:
        targetNamespaces:
        - openshift-bare-metal-events

   2. Create the OperatorGroup CR:

      $ oc create -f bare-metal-events-operatorgroup.yaml

3. Subscribe to the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: bare-metal-event-relay-subscription
        namespace: openshift-bare-metal-events
      spec:
        channel: "stable"
        name: bare-metal-event-relay
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription CR:

      $ oc create -f bare-metal-events-sub.yaml

Procedure

1. Install the Bare Metal Event Relay using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose Bare Metal Event Relay from the list of available Operators, and then click Install.
   3. On the Install Operator page, select or create a Namespace, select openshift-bare-metal-events, and then click Install.

1. Switch to the Operators → Installed Operators page.

2. Ensure that Bare Metal Event Relay is listed in the project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Procedure

1. Confirm that the node hardware has the Redfish EventService enabled by running the following curl command:

   $ curl https://<bmc_ip_address>/redfish/v1/EventService --insecure -H 'Content-Type: application/json' -u "<bmc_username>:<password>"

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   {
      "@odata.context": "/redfish/v1/$metadata#EventService.EventService",
      "@odata.id": "/redfish/v1/EventService",
      "@odata.type": "#EventService.v1_0_2.EventService",
      "Actions": {
         "#EventService.SubmitTestEvent": {
            "EventType@Redfish.AllowableValues": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
            "target": "/redfish/v1/EventService/Actions/EventService.SubmitTestEvent"
         }
      },
      "DeliveryRetryAttempts": 3,
      "DeliveryRetryIntervalSeconds": 30,
      "Description": "Event Service represents the properties for the service",
      "EventTypesForSubscription": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
      "EventTypesForSubscription@odata.count": 5,
      "Id": "EventService",
      "Name": "Event Service",
      "ServiceEnabled": true,
      "Status": {
         "Health": "OK",
         "HealthRollup": "OK",
         "State": "Enabled"
      },
      "Subscriptions": {
         "@odata.id": "/redfish/v1/EventService/Subscriptions"
      }
   }

2. Get the Bare Metal Event Relay service route for the cluster by running the following command:

   $ oc get route -n openshift-bare-metal-events

   Example output

   NAME            HOST/PORT   PATH                                                                    SERVICES                 PORT   TERMINATION   WILDCARD
   hw-event-proxy              hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com   hw-event-proxy-service   9087   edge          None

3. Create a BMCEventSubscription resource to subscribe to the Redfish events:

   1. Save the following YAML in the bmc_sub.yaml file:

      apiVersion: metal3.io/v1alpha1
      kind: BMCEventSubscription
      metadata:
        name: sub-01
        namespace: openshift-machine-api
      spec:
         hostName: <hostname> 1
         destination: <proxy_service_url> 2
         context: ''

      1

      Specifies the name or UUID of the worker node where the Redfish events are generated.

      2

      Specifies the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   2. Create the BMCEventSubscription CR:

      $ oc create -f bmc_sub.yaml

4. Optional: To delete the BMC event subscription, run the following command:

   $ oc delete -f bmc_sub.yaml

5. Optional: To manually create a Redfish event subscription without creating a BMCEventSubscription CR, run the following curl command, specifying the BMC username and password.

   $ curl -i -k -X POST -H "Content-Type: application/json"  -d '{"Destination": "https://<proxy_service_url>", "Protocol" : "Redfish", "EventTypes": ["Alert"], "Context": "root"}' -u <bmc_username>:<password> 'https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions' –v

   where:

   proxy_service_url

   is the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   HTTP/1.1 201 Created
   Server: AMI MegaRAC Redfish Service
   Location: /redfish/v1/EventService/Subscriptions/1
   Allow: GET, POST
   Access-Control-Allow-Origin: *
   Access-Control-Expose-Headers: X-Auth-Token
   Access-Control-Allow-Headers: X-Auth-Token
   Access-Control-Allow-Credentials: true
   Cache-Control: no-cache, must-revalidate
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>; rel=describedby
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>
   Link: </redfish/v1/EventService/Subscriptions>; path=
   ETag: "1651135676"
   Content-Type: application/json; charset=UTF-8
   OData-Version: 4.0
   Content-Length: 614
   Date: Thu, 28 Apr 2022 08:47:57 GMT

Procedure

1. Check the current subscriptions for the BMC by running the following curl command:

   $ curl --globoff -H "Content-Type: application/json" -k -X GET --user <bmc_username>:<password> https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 435 100 435 0 0 399 0 0:00:01 0:00:01 --:--:-- 399
   {
     "@odata.context": "/redfish/v1/$metadata#EventDestinationCollection.EventDestinationCollection",
     "@odata.etag": ""
     1651137375 "",
     "@odata.id": "/redfish/v1/EventService/Subscriptions",
     "@odata.type": "#EventDestinationCollection.EventDestinationCollection",
     "Description": "Collection for Event Subscriptions",
     "Members": [
     {
       "@odata.id": "/redfish/v1/EventService/Subscriptions/1"
     }],
     "Members@odata.count": 1,
     "Name": "Event Subscriptions Collection"
   }

   In this example, a single subscription is configured: /redfish/v1/EventService/Subscriptions/1.

2. Optional: To remove the /redfish/v1/EventService/Subscriptions/1 subscription with curl, run the following command, specifying the BMC username and password:

   $ curl --globoff -L -w "%{http_code} %{url_effective}\n" -k -u <bmc_username>:<password >-H "Content-Type: application/json" -d '{}' -X DELETE https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions/1

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

Procedure

1. Create the HardwareEvent custom resource (CR):

   Note

   Multiple HardwareEvent resources are not permitted.

   1. Save the following YAML in the hw-event.yaml file:

      apiVersion: "event.redhat-cne.org/v1alpha1"
      kind: "HardwareEvent"
      metadata:
        name: "hardware-event"
      spec:
        nodeSelector:
          node-role.kubernetes.io/hw-event: "" 1
        logLevel: "debug" 2
        msgParserTimeout: "10" 3

      1

      Required. Use the nodeSelector field to target nodes with the specified label, for example, node-role.kubernetes.io/hw-event: "".

      Note

      In OpenShift Container Platform 4.12 or later, you do not need to set the spec.transportHost field in the HardwareEvent resource when you use HTTP transport for bare-metal events. Set transportHost only when you use AMQP transport for bare-metal events.

      2

      Optional. The default value is debug. Sets the log level in hw-event-proxy logs. The following log levels are available: fatal, error, warning, info, debug, trace.

      3

      Optional. Sets the timeout value in milliseconds for the Message Parser. If a message parsing request is not responded to within the timeout duration, the original hardware event message is passed to the cloud native event framework. The default value is 10.

   2. Apply the HardwareEvent CR in the cluster:

      $ oc create -f hardware-event.yaml

2. Create a BMC username and password Secret CR that enables the hardware events proxy to access the Redfish message registry for the bare-metal host.

   1. Save the following YAML in the hw-event-bmc-secret.yaml file:

      apiVersion: v1
      kind: Secret
      metadata:
        name: redfish-basic-auth
      type: Opaque
      stringData: 1
        username: <bmc_username>
        password: <bmc_password>
        # BMC host DNS or IP address
        hostaddr: <bmc_host_ip_address>

      1

      Enter plain text values for the various items under stringData.

   2. Create the Secret CR:

      $ oc create -f hw-event-bmc-secret.yaml

1. Optional: Add a node to the OpenShift Container Platform cluster. See Setting BIOS parameters for system tuning.
2. Add the label worker-rt to the worker nodes that require the real-time capability by using the oc command.

3. Create a new machine config pool for real-time nodes:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-rt
     labels:
       machineconfiguration.openshift.io/role: worker-rt
   spec:
     machineConfigSelector:
       matchExpressions:
         - {
              key: machineconfiguration.openshift.io/role,
              operator: In,
              values: [worker, worker-rt],
           }
     paused: false
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-rt: ""

   Note that a machine config pool worker-rt is created for group of nodes that have the label worker-rt.

4. Add the node to the proper machine config pool by using node role labels.

   Note

   You must decide which nodes are configured with real-time workloads. You could configure all of the nodes in the cluster, or a subset of the nodes. The Node Tuning Operator that expects all of the nodes are part of a dedicated machine config pool. If you use all of the nodes, you must point the Node Tuning Operator to the worker node role label. If you use a subset, you must group the nodes into a new machine config pool.

5. Create the PerformanceProfile with the proper set of housekeeping cores and realTimeKernel: enabled: true.

6. You must set machineConfigPoolSelector in PerformanceProfile:

     apiVersion: performance.openshift.io/v2
     kind: PerformanceProfile
     metadata:
      name: example-performanceprofile
     spec:
     ...
       realTimeKernel:
         enabled: true
       nodeSelector:
          node-role.kubernetes.io/worker-rt: ""
       machineConfigPoolSelector:
          machineconfiguration.openshift.io/role: worker-rt

7. Verify that a matching machine config pool exists with a label:

   $ oc describe mcp/worker-rt

   Example output

   Name:         worker-rt
   Namespace:
   Labels:       machineconfiguration.openshift.io/role=worker-rt

8. OpenShift Container Platform will start configuring the nodes, which might involve multiple reboots. Wait for the nodes to settle. This can take a long time depending on the specific hardware you use, but 20 minutes per node is expected.
9. Verify everything is working as expected.

Procedure

1. Create a pod with a QoS class of Guaranteed.
2. Optional: Disable CPU load balancing for DPDK.
3. Assign a proper node selector.

1. Create the pod:

   $ oc  apply -f qos-pod.yaml --namespace=qos-example

2. View detailed information about the pod:

   $ oc get pod qos-demo --namespace=qos-example --output=yaml

   Example output

   spec:
     containers:
       ...
   status:
     qosClass: Guaranteed

   Note

   If a container specifies its own memory limit, but does not specify a memory request, OpenShift Container Platform automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own CPU limit, but does not specify a CPU request, OpenShift Container Platform automatically assigns a CPU request that matches the limit.

1. Set the offline CPUs in the performance profile and save the contents of the YAML file:

   Example performance profile with offlined CPUs

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     additionalKernelArgs:
     - nmi_watchdog=0
     - audit=0
     - mce=off
     - processor.max_cstate=1
     - intel_idle.max_cstate=0
     - idle=poll
     cpu:
       isolated: "2-23,26-47"
       reserved: "0,1,24,25"
       offlined: "48-59" 1
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     numa:
       topologyPolicy: single-numa-node
     realTimeKernel:
       enabled: true

   1

   Optional. You can list CPUs in the offlined field to take the specified CPUs offline.

2. Apply the updated profile by running the following command:

   $ oc apply -f my-performance-profile.yaml

Procedure

1. Generate a PerformanceProfile with per-pod-power-management set to true:

   $ podman run --entrypoint performance-profile-creator -v \
   /must-gather:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-operator:v4.12 \
   --mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true \
   --split-reserved-cpus-across-numa=false --topology-manager-policy=single-numa-node \
   --must-gather-dir-path /must-gather -power-consumption-mode=low-latency \ 1
   --per-pod-power-management=true > my-performance-profile.yaml

   1

   The power-consumption-mode must be default or low-latency when the per-pod-power-management is set to true.

   Example PerformanceProfile with perPodPowerManagement

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
        name: performance
   spec:
       [.....]
       workloadHints:
           realTime: true
           highPowerConsumption: false
           perPodPowerManagement: true

2. Set the default cpufreq governor as an additional kernel argument in the PerformanceProfile custom resource (CR):

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
        name: performance
   spec:
       ...
       additionalKernelArgs:
       - cpufreq.default_governor=schedutil 1

   1

   Using the schedutil governor is recommended, however, you can use other governors such as the ondemand or powersave governors.

3. Set the maximum CPU frequency in the TunedPerformancePatch CR:

   spec:
     profile:
     - data: |
         [sysfs]
         /sys/devices/system/cpu/intel_pstate/max_perf_pct = <x> 1

   1

   The max_perf_pct controls the maximum frequency the cpufreq driver is allowed to set as a percentage of the maximum supported cpu frequency. This value applies to all CPUs. You can check the maximum supported frequency in /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq. As a starting point, you can use a percentage that caps all CPUs at the All Cores Turbo frequency. The All Cores Turbo frequency is the frequency that all cores will run at when the cores are all fully occupied.

4. Add the desired annotations to your high priority workload pods. The annotations override the default settings.

   Example high priority workload annotation

   apiVersion: v1
   kind: Pod
   metadata:
     ...
     annotations:
       ...
       cpu-c-states.crio.io: "disable"
       cpu-freq-governor.crio.io: "<governor>"
       ...
     ...
   spec:
     ...
     runtimeClassName: performance-<profile_name>
     ...

5. Restart the pods.

Procedure

1. Log in to the OpenShift Container Platform cluster as a user with cluster-admin privileges.
2. Set the performance profile apiVersion to use performance.openshift.io/v2.
3. Remove the globallyDisableIrqLoadBalancing field or set it to false.

4. Set the appropriate isolated and reserved CPUs. The following snippet illustrates a profile that reserves 2 CPUs. IRQ load-balancing is enabled for pods running on the isolated CPU set:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: dynamic-irq-profile
   spec:
     cpu:
       isolated: 2-5
       reserved: 0-1
   ...

   Note

   When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.

5. Create the pod that uses exclusive CPUs, and set irq-load-balancing.crio.io and cpu-quota.crio.io annotations to disable. For example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: dynamic-irq-pod
     annotations:
        irq-load-balancing.crio.io: "disable"
        cpu-quota.crio.io: "disable"
   spec:
     containers:
     - name: dynamic-irq-pod
       image: "registry.redhat.io/openshift4/cnf-tests-rhel8:v4.12"
       command: ["sleep", "10h"]
       resources:
         requests:
           cpu: 2
           memory: "200M"
         limits:
           cpu: 2
           memory: "200M"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     runtimeClassName: performance-dynamic-irq-profile
   ...

6. Enter the pod runtimeClassName in the form performance-<profile_name>, where <profile_name> is the name from the PerformanceProfile YAML, in this example, performance-dynamic-irq-profile.
7. Set the node selector to target a cnf-worker.

8. Ensure the pod is running correctly. Status should be running, and the correct cnf-worker node should be set:

   $ oc get pod -o wide

   Expected output

   NAME              READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
   dynamic-irq-pod   1/1     Running   0          5h33m   <ip-address>   <node-name>   <none>           <none>

9. Get the CPUs that the pod configured for IRQ dynamic load balancing runs on:

   $ oc exec -it dynamic-irq-pod -- /bin/bash -c "grep Cpus_allowed_list /proc/self/status | awk '{print $2}'"

   Expected output

   Cpus_allowed_list:  2-3

10. Ensure the node configuration is applied correctly. Log in to the node to verify the configuration.

    $ oc debug node/<node-name>

    Expected output

    Starting pod/<node-name>-debug ...
    To use host binaries, run `chroot /host`
    
    Pod IP: <ip-address>
    If you don't see a command prompt, try pressing enter.
    
    sh-4.4#

11. Verify that you can use the node file system:

    sh-4.4# chroot /host

    Expected output

    sh-4.4#

12. Ensure the default system CPU affinity mask does not include the dynamic-irq-pod CPUs, for example, CPUs 2 and 3.

    $ cat /proc/irq/default_smp_affinity

    Example output

    33

13. Ensure the system IRQs are not configured to run on the dynamic-irq-pod CPUs:

    find /proc/irq/ -name smp_affinity_list -exec sh -c 'i="$1"; mask=$(cat $i); file=$(echo $i); echo $file: $mask' _ {} \;

    Example output

    /proc/irq/0/smp_affinity_list: 0-5
    /proc/irq/1/smp_affinity_list: 5
    /proc/irq/2/smp_affinity_list: 0-5
    /proc/irq/3/smp_affinity_list: 0-5
    /proc/irq/4/smp_affinity_list: 0
    /proc/irq/5/smp_affinity_list: 0-5
    /proc/irq/6/smp_affinity_list: 0-5
    /proc/irq/7/smp_affinity_list: 0-5
    /proc/irq/8/smp_affinity_list: 4
    /proc/irq/9/smp_affinity_list: 4
    /proc/irq/10/smp_affinity_list: 0-5
    /proc/irq/11/smp_affinity_list: 0
    /proc/irq/12/smp_affinity_list: 1
    /proc/irq/13/smp_affinity_list: 0-5
    /proc/irq/14/smp_affinity_list: 1
    /proc/irq/15/smp_affinity_list: 0
    /proc/irq/24/smp_affinity_list: 1
    /proc/irq/25/smp_affinity_list: 1
    /proc/irq/26/smp_affinity_list: 1
    /proc/irq/27/smp_affinity_list: 5
    /proc/irq/28/smp_affinity_list: 1
    /proc/irq/29/smp_affinity_list: 0
    /proc/irq/30/smp_affinity_list: 0-5

Procedure

1. Ascertain which threads are running on what CPUs for the host you want to configure.

   You can view which threads are running on the host CPUs by logging in to the cluster and running the following command:

   $ lscpu --all --extended

   Example output

   CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
   0   0    0      0    0:0:0:0       yes    4800.0000 400.0000
   1   0    0      1    1:1:1:0       yes    4800.0000 400.0000
   2   0    0      2    2:2:2:0       yes    4800.0000 400.0000
   3   0    0      3    3:3:3:0       yes    4800.0000 400.0000
   4   0    0      0    0:0:0:0       yes    4800.0000 400.0000
   5   0    0      1    1:1:1:0       yes    4800.0000 400.0000
   6   0    0      2    2:2:2:0       yes    4800.0000 400.0000
   7   0    0      3    3:3:3:0       yes    4800.0000 400.0000

   In this example, there are eight logical CPU cores running on four physical CPU cores. CPU0 and CPU4 are running on physical Core0, CPU1 and CPU5 are running on physical Core 1, and so on.

   Alternatively, to view the threads that are set for a particular physical CPU core (cpu0 in the example below), open a command prompt and run the following:

   $ cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list

   Example output

   0-4

2. Apply the isolated and reserved CPUs in the PerformanceProfile YAML. For example, you can set logical cores CPU0 and CPU4 as isolated, and logical cores CPU1 to CPU3 and CPU5 to CPU7 as reserved. When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.

   ...
     cpu:
       isolated: 0,4
       reserved: 1-3,5-7
   ...

   Note

   The reserved and isolated CPU pools must not overlap and together must span all available cores in the worker node.

Procedure

1. Create a PerformanceProfile appropriate for the environment’s hardware and topology as described in the table in "Understanding workload hints". Adjust the profile to match the expected workload. In this example, we tune for the lowest possible latency.

2. Add the highPowerConsumption and realTime workload hints. Both are set to true here.

       apiVersion: performance.openshift.io/v2
       kind: PerformanceProfile
       metadata:
         name: workload-hints
       spec:
         ...
         workloadHints:
           highPowerConsumption: true 1
           realTime: true 2

   1

   If highPowerConsumption is true, the node is tuned for very low latency at the cost of increased power consumption.

   2

   Disables some debugging and monitoring features that can affect system latency.

Procedure

1. Create a performance profile appropriate for the environment’s hardware and topology.

2. Add the reserved and isolated parameters with the CPUs you want reserved and isolated for the infra and application containers:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: infra-cpus
   spec:
     cpu:
       reserved: "0-4,9" 1
       isolated: "5-8" 2
     nodeSelector: 3
       node-role.kubernetes.io/worker: ""

   1

   Specify which CPUs are for infra containers to perform cluster and operating system housekeeping duties.

   2

   Specify which CPUs are for application containers to run workloads.

   3

   Optional: Specify a node selector to apply the performance profile to specific nodes.

Procedure

1. Log in to the OpenShift Container Platform cluster running the Node Tuning Operator as a user with cluster-admin privileges.
2. Create and apply a performance profile appropriate for your hardware and topology. For guidance on creating a profile, see the "Creating a performance profile" section.

3. Edit this created performance profile:

   $ oc edit -f <your_profile_name>.yaml

4. Populate the spec field with the net object. The object list can contain two fields:

   • userLevelNetworking is a required field specified as a boolean flag. If userLevelNetworking is true, the queue count is set to the reserved CPU count for all supported devices. The default is false.

   • devices is an optional field specifying a list of devices that will have the queues set to the reserved CPU count. If the device list is empty, the configuration applies to all network devices. The configuration is as follows:

     • interfaceName: This field specifies the interface name, and it supports shell-style wildcards, which can be positive or negative.

       • Example wildcard syntax is as follows: <string> .*
       • Negative rules are prefixed with an exclamation mark. To apply the net queue changes to all devices other than the excluded list, use !<device>, for example, !eno1.
     • vendorID: The network device vendor ID represented as a 16-bit hexadecimal number with a 0x prefix.

     • deviceID: The network device ID (model) represented as a 16-bit hexadecimal number with a 0x prefix.

       Note

       When a deviceID is specified, the vendorID must also be defined. A device that matches all of the device identifiers specified in a device entry interfaceName, vendorID, or a pair of vendorID plus deviceID qualifies as a network device. This network device then has its net queues count set to the reserved CPU count.

       When two or more devices are specified, the net queues count is set to any net device that matches one of them.

5. Set the queue count to the reserved CPU count for all devices by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

6. Set the queue count to the reserved CPU count for all devices matching any of the defined device identifiers by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth0"
       - interfaceName: "eth1"
       - vendorID: "0x1af4"
         deviceID: "0x1000"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

7. Set the queue count to the reserved CPU count for all devices starting with the interface name eth by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth*"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

8. Set the queue count to the reserved CPU count for all devices with an interface named anything other than eno1 by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "!eno1"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

9. Set the queue count to the reserved CPU count for all devices that have an interface name eth0, vendorID of 0x1af4, and deviceID of 0x1000 by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth0"
       - vendorID: "0x1af4"
         deviceID: "0x1000"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

10. Apply the updated performance profile:

    $ oc apply -f <your_profile_name>.yaml

1. The associated machine config pool is in a degraded state:

   # oc get mcp

   Example output

   NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master       rendered-master-2ee57a93fa6c9181b546ca46e1571d2d       True      False      False      3              3                   3                     0                      2d21h
   worker       rendered-worker-d6b2bdc07d9f5a59a6b68950acf25e5f       True      False      False      2              2                   2                     0                      2d21h
   worker-cnf   rendered-worker-cnf-6c838641b8a08fff08dbd8b02fb63f7c   False     True       True       2              1                   1                     1                      2d20h

2. The describe section of the MCP shows the reason:

   # oc describe mcp worker-cnf

   Example output

     Message:               Node node-worker-cnf is reporting: "prepping update:
     machineconfig.machineconfiguration.openshift.io \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not
     found"
       Reason:                1 nodes are reporting degraded status on sync

3. The degraded state should also appear under the performance profile status field marked as degraded = true:

   # oc describe performanceprofiles performance

   Example output

   Message: Machine config pool worker-cnf Degraded Reason: 1 nodes are reporting degraded status on sync.
   Machine config pool worker-cnf Degraded Message: Node yquinn-q8s5v-w-b-z5lqn.c.openshift-gce-devel.internal is
   reporting: "prepping update: machineconfig.machineconfiguration.openshift.io
   \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not found".    Reason:  MCPDegraded
      Status:  True
      Type:    Degraded

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to the Node Tuning Operator:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \ 1
   
    --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.12 2

   1

   The default OpenShift Container Platform must-gather image.

   2

   The must-gather image for low latency tuning diagnostics.

3. Create a compressed file from the must-gather directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

    $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.