How to use iotop command to troubleshoot the processes causing high I/O wait time on the OCP4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.14

Issue

  • How to use iotop command on the OCP4?
  • How to troubheshoot the processes causing high I/O wait time on the OCP4 ?

Resolution

Solution 1: Use oc debug node command
- Run oc debug node/node_name, It will run a debug pods for node and drop you in the shell(TTY) of this pod.

# oc debug node/<node_hostname>

sh-5.1# whereis iotop
iotop: /usr/sbin/iotop /usr/share/man/man8/iotop.8.gz

Solution 2: Running a daemon

# oc new-project metrics-debug
# oc adm policy add-scc-to-user privileged -z default/

cat << EOF > ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: metrics-daemonset
  labels:
    app: metrics-daemonset
spec:
  selector:
    matchLabels:
      app: metrics-daemonset
  template:
    metadata:
      labels:
        app: metrics-daemonset
    spec:
      nodeName: ocp-tnkfb-worker-2 ## Select nodes that you want to troubleshoot
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: metrics-daemonset
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:651732e89586140ee9fd7ca8772e0adbf9606fff461759235dbbd42af5636eae
        command:
          - bash
          - -c
          - "sleep infinity "
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          privileged: true
EOF
oc create -f ds.yaml

# oc get po
NAME                      READY   STATUS    RESTARTS   AGE
metrics-daemonset-b6zgd   1/1     Running   0          2m59s

# oc rsh metrics-daemonset-b6zgd
sh-5.1# whereis iotop
iotop: /usr/sbin/iotop /usr/share/man/man8/iotop.8.gz

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments