logStore stanza is required to deploy fluentd stand-alone to resolve fluentd pods stuck in "Init:CrashLoopBackOff" status

Solution Verified - Updated -

Issue

  • When deploying a log receiver and creating ClusterLogging, after enabling logforwarding and only deploying fluentd (standalone), the fluentd pods are stuck in "Init:CrashLoopBackOff" and are not in a "Running" status:
$ oc get pod

NAME                                       READY   STATUS                  RESTARTS   AGE
cluster-logging-operator-74cc99dfd-drts4   1/1     Running                 0          5h33m
fluentd-6mlbv                              0/1     Init:CrashLoopBackOff   6          10m
fluentd-7bzpz                              0/1     Init:CrashLoopBackOff   6          10m
fluentd-7c77w                              0/1     Init:CrashLoopBackOff   6          10m
fluentd-8zd7l                              0/1     Init:Error              7          10m
fluentd-czq9n                              0/1     Init:Error              7          10m
fluentd-fhp9v                              0/1     Init:CrashLoopBackOff   6          10m
fluentd-g47c9                              0/1     Init:CrashLoopBackOff   6          10m
fluentd-ht252                              0/1     Init:CrashLoopBackOff   6          10m
fluentd-vv4l5                              0/1     Init:CrashLoopBackOff   6          10m
fluentdserver-578777544c-b5nwq             1/1     Running                 0          11m
$ oc get clusterlogging -o yaml

  spec:
    collection:
      logs:
        fluentd: {}
        type: fluentd
    managementState: Managed
  status:
    collection:
      logs:
        fluentdStatus:
          clusterCondition:
            fluentd-6mlbv:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-7bzpz:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-7c77w:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-8zd7l:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-czq9n:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-fhp9v:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-g47c9:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-ht252:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
            fluentd-vv4l5:
            - lastTransitionTime: "2020-06-28T08:27:28Z"
              reason: PodInitializing
              status: "True"
              type: ContainerWaiting
          daemonSet: fluentd
          nodes:
            fluentd-6mlbv: ip-10-0-137-205.us-east-2.compute.internal
            fluentd-7bzpz: ip-10-0-153-51.us-east-2.compute.internal
            fluentd-7c77w: ip-10-0-142-205.us-east-2.compute.internal
            fluentd-8zd7l: ip-10-0-201-66.us-east-2.compute.internal
            fluentd-czq9n: ip-10-0-203-83.us-east-2.compute.internal
            fluentd-fhp9v: ip-10-0-183-222.us-east-2.compute.internal
            fluentd-g47c9: ip-10-0-162-142.us-east-2.compute.internal
            fluentd-ht252: ip-10-0-161-84.us-east-2.compute.internal
            fluentd-vv4l5: ip-10-0-192-161.us-east-2.compute.internal
          pods:
            failed: []
            notReady:
            - fluentd-6mlbv
            - fluentd-7bzpz
            - fluentd-7c77w
            - fluentd-8zd7l
            - fluentd-czq9n
            - fluentd-fhp9v
            - fluentd-g47c9
            - fluentd-ht252
            - fluentd-vv4l5
            ready: []
    curation: {}
    logStore: {}
    visualization: {}
$ oc get logforwarding -o yaml

  spec:
    outputs:
    - endpoint: fluentdserver.openshift-logging.svc:24224
      insecure: true
      name: fluentd-created-by-user
      type: forward
    pipelines:
    - inputSource: logs.app
      name: app-pipeline
      outputRefs:
      - fluentd-created-by-user
    - inputSource: logs.infra
      name: infra-pipeline
      outputRefs:
      - fluentd-created-by-user
    - inputSource: logs.audit
      name: audit-pipeline
      outputRefs:
      - fluentd-created-by-user
  status:
    lastUpdated: "2020-06-28T08:15:36Z"
    reason: ResourceName
    state: Accepted
  • Version-Release number of selected component (if applicable):
$ oc get csv
NAME                                           DISPLAY                  VERSION                 REPLACES   PHASE
clusterlogging.4.5.0-202006271533.p0           Cluster Logging          4.5.0-202006271533.p0              Succeeded
elasticsearch-operator.4.5.0-202006261904.p0   Elasticsearch Operator   4.5.0-202006261904.p0              Succeeded

Environment

  • Red Hat OpenShift Container Platform (OCP)
    • 4.4
    • 4.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content