Understanding the OpenShift 4 must gather

Updated -

The must-gather archive is a collection of cluster resources and logs that after is ran, it creates a directory in the current working directory. Then the object is automatically deleted.

Useful links:

audit_logs/

The audit logs in this directory contain security auditing information about activity in the cluster. See documentation for how to parse these logs. The following data is gathered using the base command oc adm node-logs along with a bazillion flags. See the audit logs collection script for details.

kube-apiserver/

Contains audit logging specific to the main Kubernetes API Server running in the openshift-kube-apiserver namespace as one of the kube-apiserver Pods. The logs are located inside the Pod in /var/log/kube-apiserver.

openshift-apiserver/

Contains audit logging specific to the OpenShift API extension of the Kubernetes API running in openshift-apiserver namespace as one of the apiserver Pods. The logs are located inside the Pod in /var/log/openshift-apiserver.

*.audit_log_listing

Two of these files should exist. It just contains a listing of the logs captured.

cluster-scoped-resources/

This directory contains resources that are not restricted to a namespace. For example, StorageClasses, ClusterVersions, ClusterOperators, and Nodes are all Cluster Scoped Resources.

core/

nodes/

This contains the Node resources. The same output as running oc get nodes.

persistentvolumes/

Contains known persistent volumens. Roughly equivalent to oc get pv -o yaml

clusteroperators.yaml

Contains known clsuteroperator resources. Equivalent to oc get clusteroperator -o yaml

clusterversion.yaml

Contains cluster version resources. Equivalent to oc get clusterversion -o yaml

All other directories

All of the other directories in cluster-scoped-resources/ can be identified by the group (e.g. storage.k8s.io) and going into the directory to find the specific resource (e.g. storage.k8s.io/storageclasses).

host_service_logs/

This directory contains logs from the systemd services running on the hosts. These are not container or pod logs!

master/

This contains two log files, one for crio and other for kubelet services. These logs are gathered using the command oc adm node-logs --role=master. See here for more detail.

worker/

Same as above. Crio and kubelet logs are gathered just using --role=worker.

namespaces/

This directory contains information of all infrastructure namespaces in OpenShift. These are organized by the namespace name, resource groups (e.g. build.openshift.io, apps), and then the individual resources inside the namespace (pods, configmaps, routes, secrets, etc).

Resources Gathered

Most project scope resources are gathered. The list is defined in must-gather code here. They include:

oc get all
oc get events
oc get configmaps
oc get secrets
oc get pvc

Pod logs

Pod logs are reviewed very often and can be found in the specific project directory -> Pods directory -> Pod name -> container name -> container name (again) -> logs

There are two log files, current.log and previous.log. Current.log shows the logs from the currently running Pod. Previous.log contains the logs from the previous instance of the Pod (useful in the event of a crashing Pod). These logs are gathered using the equivalent of oc logs and oc logs -p.

  • Component
  • oc

Comments