Understanding the OpenShift 4 must gather
The must-gather archive is a collection of cluster resources and logs that after is ran, it creates a directory in the current working directory. Then the object is automatically deleted.
Useful links:
- Must gather in product documentation
- Must gather source code
- Tool for reviewing must gathers
- Data-gathering in the Event must-gather Fails
- Creating oc adm must-gather with more details for specific use cases in Red Hat OpenShift Container Platform 4.x
audit_logs/
The audit logs in this directory contain security auditing information about activity in the cluster. See documentation for how to parse these logs. The following data is gathered using the base command oc adm node-logs
along with a bazillion flags. See the audit logs collection script for details.
kube-apiserver/
Contains audit logging specific to the main Kubernetes API Server running in the openshift-kube-apiserver namespace as one of the kube-apiserver Pods. The logs are located inside the Pod in /var/log/kube-apiserver.
openshift-apiserver/
Contains audit logging specific to the OpenShift API extension of the Kubernetes API running in openshift-apiserver namespace as one of the apiserver Pods. The logs are located inside the Pod in /var/log/openshift-apiserver.
*.audit_log_listing
Two of these files should exist. It just contains a listing of the logs captured.
cluster-scoped-resources/
This directory contains resources that are not restricted to a namespace. For example, StorageClasses, ClusterVersions, ClusterOperators, and Nodes are all Cluster Scoped Resources.
core/
nodes/
This contains the Node resources. The same output as running oc get nodes
.
persistentvolumes/
Contains known persistent volumens. Roughly equivalent to oc get pv -o yaml
clusteroperators.yaml
Contains known clsuteroperator resources. Equivalent to oc get clusteroperator -o yaml
clusterversion.yaml
Contains cluster version resources. Equivalent to oc get clusterversion -o yaml
All other directories
All of the other directories in cluster-scoped-resources/ can be identified by the group (e.g. storage.k8s.io) and going into the directory to find the specific resource (e.g. storage.k8s.io/storageclasses).
host_service_logs/
This directory contains logs from the systemd services running on the hosts. These are not container or pod logs!
master/
This contains two log files, one for crio and other for kubelet services. These logs are gathered using the command oc adm node-logs --role=master
. See here for more detail.
worker/
Same as above. Crio and kubelet logs are gathered just using --role=worker.
namespaces/
This directory contains information of all infrastructure namespaces in OpenShift. These are organized by the namespace name, resource groups (e.g. build.openshift.io, apps), and then the individual resources inside the namespace (pods, configmaps, routes, secrets, etc).
Resources Gathered
Most project scope resources are gathered. The list is defined in must-gather code here. They include:
oc get all
oc get events
oc get configmaps
oc get secrets
oc get pvc
Pod logs
Pod logs are reviewed very often and can be found in the specific project directory -> Pods directory -> Pod name -> container name -> container name (again) -> logs
There are two log files, current.log and previous.log. Current.log shows the logs from the currently running Pod. Previous.log contains the logs from the previous instance of the Pod (useful in the event of a crashing Pod). These logs are gathered using the equivalent of oc logs
and oc logs -p
.
Comments