Chapter 2. Downloading log files and diagnostic information using must-gather

If Red Hat OpenShift Data Foundation is unable to automatically resolve a problem, use the must-gather tool to collect log files and diagnostic information so that you or Red Hat support can review the problem and determine a solution.

Important

When Red Hat OpenShift Data Foundation is deployed in external mode, must-gather only collects logs from the Red Hat OpenShift Data Foundation cluster and does not collect debug data and logs from the external Red Hat Ceph Storage cluster. To collect debug logs from the external Red Hat Ceph Storage cluster, see Red Hat Ceph Storage Troubleshooting guide and contact your Red Hat Ceph Storage Administrator.

Prerequisites

  • Optional: If OpenShift Data Foundation is deployed in a disconnected environment, ensure that you mirror the individual must-gather image to the mirror registry available from the disconnected environment. 

    $ oc image mirror registry.redhat.io/odf4/ocs-must-gather-rhel8:v4.9 <local-registry>/odf4/ocs-must-gather-rhel8:v4.9 [--registry-config=<path-to-the-registry-config>] [--insecure=true]
    <local-registry>
    Is the local image mirror registry available for a disconnected OpenShift Container Platform cluster.
    <path-to-the-registry-config>
    Is the path to your registry credentials, by default it is ~/.docker/config.json.
    --insecure
    Add this flag only if the mirror registry is insecure.

    For more information, see the Red Hat Knowledgebase solutions:

Procedure

  • Run the must-gather command from the client connected to the Red Hat OpenShift Data Foundation cluster: 

    $ oc adm must-gather --image=registry.redhat.io/odf4/ocs-must-gather-rhel8:v4.9 --dest-dir=<directory-name>
    <directory-name>

    Is the name of the directory where you want to write the data to.

    Important

    For a disconnected environment deployment, replace the image in --image parameter with the mirrored must-gather image. 

    $ oc adm must-gather --image=<local-registry>/odf4/ocs-must-gather-rhel8:v4.9 --dest-dir=<directory-name>
    <local-registry>
    Is the local image mirror registry available for a disconnected OpenShift Container Platform cluster.

    This collects the following information in the specified directory:

    • All Red Hat OpenShift Data Foundation cluster related Custom Resources (CRs) with their namespaces.
    • Pod logs of all the Red Hat OpenShift Data Foundation related pods.
    • Output of some standard Ceph commands like Status, Cluster health, and others.

Command variations

  • If one or more master nodes are not in the Ready state, use --node-name to provide a master node that is Ready so that the must-gather pod can be safely scheduled. 

    $ oc adm must-gather --image=registry.redhat.io/odf4/ocs-must-gather-rhel8:v4.9 --dest-dir=<directory-name> --node-name=<node-name>

  • If you want to gather information from a specific time:

    • To specify a relative time period for logs gathered, such as within 5 seconds or 2 days, add /usr/bin/gather since=<duration>: 

      $ oc adm must-gather --image=registry.redhat.io/odf4/ocs-must-gather-rhel8:v4.9 --dest-dir=<directory-name> /usr/bin/gather since=<duration>

    • To specify a specific time to gather logs after, add /usr/bin/gather since-time=<rfc3339-timestamp>: 

      $ oc adm must-gather --image=registry.redhat.io/odf4/ocs-must-gather-rhel8:v4.9 --dest-dir=<directory-name> /usr/bin/gather since-time=<rfc3339-timestamp>

    Replace the example values in these commands as follows:

    <node-name>
    If one or more master nodes are not in the Ready state, use this parameter to provide the name of a master node that is still in the Ready state. This avoids scheduling errors by ensuring that the must-gather pod is not scheduled on a master node that is not ready.
    <directory-name>
    The directory to store information collected by must-gather.
    <duration>
    Specify the period of time to collect information from as a relative duration, for example, 5h (starting from 5 hours ago).
    <rfc3339-timestamp>
    Specify the period of time to collect information from as an RFC 3339 timestamp, for example, 2020-11-10T04:00:00+00:00 (starting from 4am UTC on 11 Nov 2020).