The must-gather archive is a collection of cluster resources and logs that after is ran, it creates a directory in the current working directory. Then the object is automatically deleted.
- Must gather in product documentation
- Must gather source code
- Tool for reviewing must gathers
- Data-gathering in the Event must-gather Fails
- Creating oc adm must-gather with more details for specific use cases in Red Hat OpenShift Container Platform 4.x
The audit logs in this directory contain security auditing information about activity in the cluster. See documentation for how to parse these logs. The following data is gathered using the base command
oc adm node-logs along with a bazillion flags. See the audit logs collection script for details.
Contains audit logging specific to the main Kubernetes API Server running in the openshift-kube-apiserver namespace as one of the kube-apiserver Pods. The logs are located inside the Pod in /var/log/kube-apiserver.
Contains audit logging specific to the OpenShift API extension of the Kubernetes API running in openshift-apiserver namespace as one of the apiserver Pods. The logs are located inside the Pod in /var/log/openshift-apiserver.
Two of these files should exist. It just contains a listing of the logs captured.
This directory contains resources that are not restricted to a namespace. For example, StorageClasses, ClusterVersions, ClusterOperators, and Nodes are all Cluster Scoped Resources.
This contains the Node resources. The same output as running
oc get nodes.
Contains known persistent volumens. Roughly equivalent to
oc get pv -o yaml
Contains known clsuteroperator resources. Equivalent to
oc get clusteroperator -o yaml
Contains cluster version resources. Equivalent to
oc get clusterversion -o yaml
All other directories
All of the other directories in cluster-scoped-resources/ can be identified by the group (e.g. storage.k8s.io) and going into the directory to find the specific resource (e.g. storage.k8s.io/storageclasses).
This directory contains logs from the systemd services running on the hosts. These are not container or pod logs!
This contains two log files, one for crio and other for kubelet services. These logs are gathered using the command
oc adm node-logs --role=master. See here for more detail.
Same as above. Crio and kubelet logs are gathered just using --role=worker.
This directory contains information of all infrastructure namespaces in OpenShift. These are organized by the namespace name, resource groups (e.g. build.openshift.io, apps), and then the individual resources inside the namespace (pods, configmaps, routes, secrets, etc).
Most project scope resources are gathered. The list is defined in must-gather code here. They include:
oc get all oc get events oc get configmaps oc get secrets oc get pvc
Pod logs are reviewed very often and can be found in the specific project directory -> Pods directory -> Pod name -> container name -> container name (again) -> logs
There are two log files, current.log and previous.log. Current.log shows the logs from the currently running Pod. Previous.log contains the logs from the previous instance of the Pod (useful in the event of a crashing Pod). These logs are gathered using the equivalent of
oc logs and
oc logs -p.