[RHOCP 4.x] Cluster-logging-operator in CrashLoopBackOff State

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.9
  • IPI Microsoft Azure

Issue

  • Cluster-logging-operator in CrashLoopBackOff
$ oc get po -n openshift-logging
NAME                                            READY   STATUS             RESTARTS       AGE
cluster-logging-operator-5c8b9bb7bd-v6d69       0/1     CrashLoopBackOff   16 (84s ago)   2d18h

Resolution

  • The issue has been identified as a bug and is being tracked by the engineering team under Jira LOG-1896
  • The engineering team has released a fix for this issue in OpenShift Logging v5.4.0 under Errata RHSA-2022:1461

Diagnostic Steps

  • After the issue has been observed, to confirm the issue check the logs of the cluster-logging-operator by running the below command
$ oc logs cluster-logging-operator-5c8b9bb7bd-v6d69 -n openshift-logging

Output will be the below error message:

{"_ts":"2021-10-25T11:00:28.894749881Z","_level":"0","_component":"cluster-logging-operator","_message":"starting up...","go_arch":"amd64","go_os":"linux","go_version":"go1.15.14","operator-sdk_version":"v0.18.1","operator_version":"0.0.1"}
I1025 11:00:29.946640       1 request.go:621] Throttling request took 1.033509389s, request: GET:https://X.X.X.X:443/apis/project.openshift.io/v1?timeout=32s
I1025 11:00:40.122602       1 request.go:621] Throttling request took 1.046690596s, request: GET:https://X.X.X.X:443/apis/network.openshift.io/v1?timeout=32s
{"_ts":"2021-10-25T11:00:43.329844431Z","_level":"0","_component":"cluster-logging-operator","_message":"Registering Components."}
I1025 11:00:50.132130       1 request.go:621] Throttling request took 3.84686579s, request: GET:https://X.X.X.X:443/apis/network.openshift.io/v1?timeout=32s
{"_ts":"2021-10-25T11:00:50.534203788Z","_level":"0","_component":"cluster-logging-operator","_message":"Could not generate and serve custom resource metrics","error":{}}
I1025 11:01:00.139801       1 request.go:621] Throttling request took 2.295902909s, request: GET:https://X.X.X.X:443/apis/events.k8s.io/v1beta1?timeout=32s
panic: runtime error: slice bounds out of range [:-1] [recovered]
        panic: runtime error: slice bounds out of range [:-1]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments