1. In the OpenShift Container Platform console, click the Application Launcher and select Logging.

2. Create your Kibana index patterns by clicking Management → Index Patterns → Create index pattern:

   • Each user must manually create index patterns when logging into Kibana the first time in order to see logs for their projects. Users must create an index pattern named app and use the @timestamp time field to view their container logs.
   • Each admin user must create index patterns when logged into Kibana the first time for the app, infra, and audit indices using the @timestamp time field.
3. Create Kibana Visualizations from the new index patterns.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources:
             limits: 1
               memory: 736Mi
             requests:
               cpu: 100m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests as needed. The values shown are the default values.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

2. Add or modify any of the following parameters:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     forwarder:
       fluentd:
         buffer:
           chunkLimitSize: 8m 1
           flushInterval: 5s 2
           flushMode: interval 3
           flushThreadCount: 3 4
           overflowAction: throw_exception 5
           retryMaxInterval: "300s" 6
           retryType: periodic 7
           retryWait: 1s 8
           totalLimitSize: 32m 9
   ...

   1

   Specify the maximum size of each chunk before it is queued for flushing.

   2

   Specify the interval between chunk flushes.

   3

   Specify the method to perform chunk flushes: lazy, interval, or immediate.

   4

   Specify the number of threads to use for chunk flushes.

   5

   Specify the chunking behavior when the queue is full: throw_exception, block, or drop_oldest_chunk.

   6

   Specify the maximum interval in seconds for the exponential_backoff chunk flushing method.

   7

   Specify the retry type when chunk flushing fails: exponential_backoff or periodic.

   8

   Specify the time in seconds before the next chunk flush.

   9

   Specify the maximum size of the chunk buffer.

3. Verify that the Fluentd pods are redeployed:

   $ oc get pods -n openshift-logging

4. Check that the new values are in the fluentd config map:

   $ oc extract configmap/fluentd --confirm

   Example fluentd.conf

   <buffer>
    @type file
    path '/var/lib/fluentd/default'
    flush_mode interval
    flush_interval 5s
    flush_thread_count 3
    retry_type periodic
    retry_wait 1s
    retry_max_interval 300s
    retry_timeout 60m
    queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
    total_limit_size 32m
    chunk_limit_size 8m
    overflow_action throw_exception
   </buffer>

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

2. If they are present, remove the logStore, visualization, curation stanzas from the ClusterLogging CR.

3. Preserve the collection stanza of the ClusterLogging CR. The result should look similar to the following example:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     collection:
       logs:
         type: "fluentd"
         fluentd: {}

4. Verify that the Fluentd pods are redeployed:

   $ oc get pods -n openshift-logging

1. Create a ClusterLogForwarder CR YAML file or edit your existing CR:

   • Create a CR to send all log types to the internal Elasticsearch instance. You can use the following example without making any changes:

     apiVersion: logging.openshift.io/v1
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       pipelines: 1
       - name: all-to-default
         inputRefs:
         - infrastructure
         - application
         - audit
         outputRefs:
         - default

     1

     A pipeline defines the type of logs to forward using the specified output. The default output forwards logs to the internal Elasticsearch instance.

     Note

     You must specify all three types of logs in the pipeline: application, infrastructure, and audit. If you do not specify a log type, those logs are not stored and will be lost.

   • If you have an existing ClusterLogForwarder CR, add a pipeline to the default output for the audit logs. You do not need to define the default output. For example:

     apiVersion: "logging.openshift.io/v1"
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       outputs:
        - name: elasticsearch-insecure
          type: "elasticsearch"
          url: http://elasticsearch-insecure.messaging.svc.cluster.local
          insecure: true
        - name: elasticsearch-secure
          type: "elasticsearch"
          url: https://elasticsearch-secure.messaging.svc.cluster.local
          secret:
            name: es-audit
        - name: secureforward-offcluster
          type: "fluentdForward"
          url: https://secureforward.offcluster.com:24224
          secret:
            name: secureforward
       pipelines:
        - name: container-logs
          inputRefs:
          - application
          outputRefs:
          - secureforward-offcluster
        - name: infra-logs
          inputRefs:
          - infrastructure
          outputRefs:
          - elasticsearch-insecure
        - name: audit-logs
          inputRefs:
          - audit
          outputRefs:
          - elasticsearch-secure
          - default 1

     1

     This pipeline sends the audit logs to the internal Elasticsearch instance in addition to an external instance.

1. Edit the ClusterLogging CR to add or modify the retentionPolicy parameter:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   ...
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       retentionPolicy: 1
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
   ...

   1

   Specify the time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 1d for one day. Logs older than the maxAge are deleted. By default, logs are retained for seven days.

2. You can verify the settings in the Elasticsearch custom resource (CR).

   For example, the Cluster Logging Operator updated the following Elasticsearch CR to configure a retention policy that includes settings to roll over active indices for the infrastructure logs every eight hours and the rolled-over indices are deleted seven days after rollover. OpenShift Container Platform checks every 15 minutes to determine if the indices need to be rolled over.

   apiVersion: "logging.openshift.io/v1"
   kind: "Elasticsearch"
   metadata:
     name: "elasticsearch"
   spec:
   ...
     indexManagement:
       policies: 1
         - name: infra-policy
           phases:
             delete:
               minAge: 7d 2
             hot:
               actions:
                 rollover:
                   maxAge: 8h 3
           pollInterval: 15m 4
   ...

   1

   For each log source, the retention policy indicates when to delete and roll over logs for that source.

   2

   When OpenShift Container Platform deletes the rolled-over indices. This setting is the maxAge you set in the ClusterLogging CR.

   3

   The index age for OpenShift Container Platform to consider when rolling over the indices. This value is determined from the maxAge you set in the ClusterLogging CR.

   4

   When OpenShift Container Platform checks if the indices should be rolled over. This setting is the default and cannot be changed.

   Note

   Modifying the Elasticsearch CR is not supported. All changes to the retention policies must be made in the ClusterLogging CR.

   The OpenShift Elasticsearch Operator deploys a cron job to roll over indices for each mapping using the defined policy, scheduled using the pollInterval.

   $ oc get cronjob

   Example output

   NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   curator                  */10 * * * *   False     0        <none>          5s
   elasticsearch-im-app     */15 * * * *   False     0        <none>          4s
   elasticsearch-im-audit   */15 * * * *   False     0        <none>          4s
   elasticsearch-im-infra   */15 * * * *   False     0        <none>          4s

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           resources: 1
             limits:
               memory: "16Gi"
             requests:
               cpu: "1"
               memory: "16Gi"
           proxy: 2
             resources:
               limits:
                 memory: 100Mi
               requests:
                 memory: 100Mi

   1

   Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

   2

   Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the ClusterLogging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   # ...
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: "gp2"
           size: "200G"

Procedure

1. Edit the ClusterLogging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Get the names of the Elasticsearch pods:

   $ oc get pods | grep elasticsearch-

3. Scale down the Fluentd pods so they stop sending new logs to Elasticsearch:

   $ oc -n openshift-logging patch daemonset/logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-fluentd": "false"}}}}}'

4. Perform a shard synced flush using the OpenShift Container Platform es_util tool to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   For example:

   $ oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6  -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   Example output

   {"_shards":{"total":4,"successful":4,"failed":0},".security":{"total":2,"successful":2,"failed":0},".kibana_1":{"total":2,"successful":2,"failed":0}}

5. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Example output

   {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":

6. After the command is complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 resumed

      A new pod is deployed. After the pod has a ready container, you can move on to the next deployment.

      $ oc get pods | grep elasticsearch-

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   2. After the deployments are complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 paused

   3. Check that the Elasticsearch cluster is in a green or yellow state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Note

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      {
        "cluster_name" : "elasticsearch",
        "status" : "yellow", 1
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      1

      Make sure this parameter value is green or yellow before proceeding.

7. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

8. After all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Example output

   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

9. Scale up the Fluentd pods so they send new logs to Elasticsearch.

   $ oc -n openshift-logging patch daemonset/logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-fluentd": "true"}}}}}'

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Extract the CA certificate from the log store and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca

   Example output

   admin-ca

3. Create the route for the log store service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 1

      1

      Add the log store CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the log store CA certificate to the route YAML you created in the previous step:

      $ cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml

      Example output

      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this service account to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}"

      The response appears similar to the following:

      Example output

      {
        "name" : "elasticsearch-cdm-i40ktba0-1",
        "cluster_name" : "elasticsearch",
        "cluster_uuid" : "0eY-tJzcR3KOdpgeMJo-MQ",
        "version" : {
        "number" : "6.8.1",
        "build_flavor" : "oss",
        "build_type" : "zip",
        "build_hash" : "Unknown",
        "build_date" : "Unknown",
        "build_snapshot" : true,
        "lucene_version" : "7.7.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
      },
        "<tagline>" : "<for search>"
      }

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance -n openshift-logging

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 2
         resources: 1
           limits:
             memory: 2Gi
           requests:
             cpu: 200m
             memory: 2Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 2
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 3
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
     curation:
       type: "curator"
       curator:
         resources: 4
           limits:
             memory: 200Mi
           requests:
             cpu: 200m
             memory: 200Mi
         schedule: "*/10 * * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd:
           resources: 5
             limits:
               memory: 736Mi
             requests:
               cpu: 200m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log curator as needed.

   5

   Specify the CPU and memory limits and requests for the log collector as needed.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas: 1 1

   1

   Specify the number of Kibana nodes.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance -n openshift-logging

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 2
         resources: 1
           limits:
             memory: 2Gi
           requests:
             cpu: 200m
             memory: 2Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 2
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 3
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
     curation:
       type: "curator"
       curator:
         resources: 4
           limits:
             memory: 200Mi
           requests:
             cpu: 200m
             memory: 200Mi
         schedule: "*/10 * * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd:
           resources: 5
             limits:
               memory: 736Mi
             requests:
               cpu: 200m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log curator as needed.

   5

   Specify the CPU and memory limits and requests for the log collector as needed.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the cluster logging pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 elasticsearch=node:NoExecute

   This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only pods that match the taint and remove existing pods that do not match.

2. Edit the logstore section of the ClusterLogging CR to configure a toleration for the Elasticsearch pods:

     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 1
         tolerations:
         - key: "elasticsearch"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the log visualizer pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and remove existing pods that do not match.

2. Edit the visualization section of the ClusterLogging CR to configure a toleration for the Kibana pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want logging collector pods to schedule logging collector pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 collector=node:NoExecute

   This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and removes existing pods that do not match.

2. Edit the collection stanza of the ClusterLogging custom resource (CR) to configure a toleration for the logging collector pods:

     collection:
       logs:
         type: "fluentd"
         fluentd:
           tolerations:
           - key: "collector"  1
             operator: "Exists"  2
             effect: "NoExecute"  3
             tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ...
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     curation:
       curator:
         nodeSelector: 1
           node-role.kubernetes.io/infra: ''
         resources: null
         schedule: 30 3 * * *
       type: curator
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 2
           node-role.kubernetes.io/infra: ''
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 3
           node-role.kubernetes.io/infra: ''
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ...

Procedure

1. Create a journald.conf file with the required settings:

   Compress=yes 1
   ForwardToConsole=no 2
   ForwardToSyslog=no
   MaxRetentionSec=1month 3
   RateLimitBurst=10000 4
   RateLimitIntervalSec=30s
   Storage=persistent 5
   SyncIntervalSec=1s 6
   SystemMaxUse=8g 7
   SystemKeepFree=20% 8
   SystemMaxFileSize=10M 9

   1

   Specify whether you want logs compressed before they are written to the file system. Specify yes to compress the message or no to not compress. The default is yes.

   2

   Configure whether to forward log messages. Defaults to no for each. Specify:

   • ForwardToConsole to forward logs to the system console.
   • ForwardToKsmg to forward logs to the kernel log buffer.
   • ForwardToSyslog to forward to a syslog daemon.
   • ForwardToWall to forward messages as wall messages to all logged-in users.

   3

   Specify the maximum time to store journal entries. Enter a number to specify seconds. Or include a unit: "year", "month", "week", "day", "h" or "m". Enter 0 to disable. The default is 1month.

   4

   Configure rate limiting. If, during the time interval defined by RateLimitIntervalSec, more logs than specified in RateLimitBurst are received, all further messages within the interval are dropped until the interval is over. It is recommended to set RateLimitIntervalSec=30s and RateLimitBurst=10000, which are the defaults.

   5

   Specify how logs are stored. The default is persistent:

   • volatile to store logs in memory in /var/log/journal/.
   • persistent to store logs to disk in /var/log/journal/. systemd creates the directory if it does not exist.
   • auto to store logs in in /var/log/journal/ if the directory exists. If it does not exist, systemd temporarily stores logs in /run/systemd/journal.
   • none to not store logs. systemd drops all logs.

   6

   Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs. systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is 1s.

   7

   Specify the maximum size the journal can use. The default is 8g.

   8

   Specify how much disk space systemd must leave free. The default is 20%.

   9

   Specify the maximum size for individual journal files stored persistently in /var/log/journal. The default is 10M.

   Note

   If you are removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.

   For more information on systemd settings, see https://www.freedesktop.org/software/systemd/man/journald.conf.html. The default settings listed on that page might not apply to OpenShift Container Platform.

2. Convert the journal.conf file to base64 and store it in a variable that is named jrnl_cnf by running the following command:

   $ export jrnl_cnf=$( cat journald.conf | base64 -w0 )

3. Create a MachineConfig object that includes the jrnl_cnf variable, which was created in the previous step. The following sample command creates a MachineConfig object for the worker:

   $ cat << EOF > ./40-worker-custom-journald.yaml 1
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker 2
     name: 40-worker-custom-journald 3
   spec:
     config:
       ignition:
         config: {}
         security:
           tls: {}
         timeouts: {}
         version: 3.1.0
       networkd: {}
       passwd: {}
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,${jrnl_cnf} 4
             verification: {}
           filesystem: root
           mode: 0644 5
           path: /etc/systemd/journald.conf.d/custom.conf
     osImageURL: ""
   EOF

   1

   Optional: For control plane (also known as master) node, you can provide the file name as 40-master-custom-journald.yaml.

   2

   Optional: For control plane (also known as master) node, provide the role as master.

   3

   Optional: For control plane (also known as master) node, you can provide the name as 40-master-custom-journald.

   4

   Optional: To include a static copy of the parameters in the journald.conf file, replace ${jrnl_cnf} with the output of the echo $jrnl_cnf command.

   5

   Set the permissions for the journal.conf file. It is recommended to set 0644 permissions.

4. Create the machine config:

   $ oc apply -f <file_name>.yaml

   The controller detects the new MachineConfig object and generates a new rendered-worker-<hash> version.

5. Monitor the status of the rollout of the new rendered configuration to each node:

   $ oc describe machineconfigpool/<node> 1

   1

   Specify the node as master or worker.

   Example output for worker

   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   
   ...
   
   Conditions:
     Message:
     Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e

1. Edit the ClusterLogging custom resource in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ...
   
     curation:
       curator:
         schedule: 30 3 * * * 1
       type: curator

   1

   Specify the schedule for Curator in cron format.

   Note

   The time zone is set based on the host node where the Curator pod runs.

1. Edit the OpenShift Container Platform custom Curator configuration file:

   $ oc edit configmap/curator

2. Set the following parameters as needed:

   config.yaml: |
     project_name:
       action
         unit:value

   The available parameters are:

   Table 3.2. Project options

   Variable Name	Description
   project_name	The actual name of a project, such as myapp-devel. For OpenShift Container Platform operations logs, use the name .operations as the project name.
   action	The action to take, currently only delete is allowed.
   unit	The period to use for deletion, days, weeks, or months.
   value	The number of units.

   Table 3.3. Filter options

   Variable Name	Description
   .defaults	Use .defaults as the project_name to set the defaults for projects that are not specified.
   .regex	The list of regular expressions that match project names.
   pattern	The valid and properly escaped regular expression pattern enclosed by single quotation marks.

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: elasticsearch-insecure 3
        type: "elasticsearch" 4
        url: http://elasticsearch.insecure.com:9200 5
      - name: elasticsearch-secure
        type: "elasticsearch"
        url: https://elasticsearch.secure.com:9200
        secret:
           name: es-secret 6
     pipelines:
      - name: application-logs 7
        inputRefs: 8
        - application
        - audit
        outputRefs:
        - elasticsearch-secure 9
        - default 10
        labels:
          myLabel: "myValue" 11
      - name: infrastructure-audit-logs 12
        inputRefs:
        - infrastructure
        outputRefs:
        - elasticsearch-insecure
        labels:
          logs: "audit-infra"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the elasticsearch type.

   5

   Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.

   6

   If using an https prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Optional: Specify a name for the pipeline.

   8

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   9

   Specify the output to use with that pipeline for forwarding the logs.

   10

   Optional: Specify the default output to send the logs to the internal Elasticsearch instance.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     pipelines:
      - name: forward-to-fluentd-secure 7
        inputRefs:  8
        - application
        - audit
        outputRefs:
        - fluentd-server-secure 9
        - default 10
        labels:
          clusterId: "C1234" 11
      - name: forward-to-fluentd-insecure 12
        inputRefs:
        - infrastructure
        outputRefs:
        - fluentd-server-insecure
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the fluentdForward type.

   5

   Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Optional. Specify a name for the pipeline.

   8

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   9

   Specify the output to use with that pipeline for forwarding the logs.

   10

   Optional. Specify the default output to forward logs to the internal Elasticsearch instance.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: rsyslog-east 3
        type: syslog 4
        syslog: 5
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514' 6
        secret: 7
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'udp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east 8
        inputRefs: 9
        - audit
        - application
        outputRefs: 10
        - rsyslog-east
        - default 11
        labels:
          secure: "true" 12
          syslog: "east"
      - name: syslog-west 13
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: "west"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the syslog type.

   5

   Optional. Specify the syslog parameters, listed below.

   6

   Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   7

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   10

   Specify the output to use with that pipeline for forwarding the logs.

   11

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   12

   Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: app-logs 3
        type: kafka 4
        url: tls://kafka.example.devlab.com:9093/app-topic 5
        secret:
          name: kafka-secret 6
      - name: infra-logs
        type: kafka
        url: tcp://kafka.devlab2.example.com:9093/infra-topic 7
      - name: audit-logs
        type: kafka
        url: tls://kafka.qelab.example.com:9093/audit-topic
        secret:
           name: kafka-secret-qe
     pipelines:
      - name: app-topic 8
        inputRefs: 9
        - application
        outputRefs: 10
        - app-logs
        labels:
          logType: "application" 11
      - name: infra-topic 12
        inputRefs:
        - infrastructure
        outputRefs:
        - infra-logs
        labels:
          logType: "infra"
      - name: audit-topic
        inputRefs:
        - audit
        outputRefs:
        - audit-logs
        - default 13
        labels:
          logType: "audit"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the kafka type.

   5

   Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   10

   Specify the output to use with that pipeline for forwarding the logs.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

   13

   Optional: Specify default to forward logs to the internal Elasticsearch instance.

2. Optional: To forward a single output to multiple kafka brokers, specify an array of kafka brokers as shown in this example:

   ...
   spec:
     outputs:
     - name: app-logs
       type: kafka
       secret:
         name: kafka-secret-dev
       kafka:  1
         brokers: 2
           - tls://kafka-broker1.example.com:9093/
           - tls://kafka-broker2.example.com:9093/
         topic: app-topic 3
   ...

   1

   Specify a kafka key that has a brokers and topic key.

   2

   Use the brokers key to specify an array of one or more brokers.

   3

   Use the topic key to specify the target topic that will receive the logs.

3. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     inputs: 7
      - name: my-app-logs
        application:
           namespaces:
           - my-project
     pipelines:
      - name: forward-to-fluentd-insecure 8
        inputRefs: 9
        - my-app-logs
        outputRefs: 10
        - fluentd-server-insecure
        labels: 11
          project: "my-project"
      - name: forward-to-fluentd-secure 12
        inputRefs:
        - application
        - audit
        - infrastructure
        outputRefs:
        - fluentd-server-secure
        - default
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the output type: elasticsearch, fluentdForward, syslog, or kafka.

   5

   Specify the URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Configuration for an input to filter application logs from the specified projects.

   8

   Configuration for a pipeline to use the input to send project application logs to an external Fluentd instance.

   9

   The my-app-logs input.

   10

   The name of the output to use.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Configuration for a pipeline to send logs to other log aggregators.

   • Optional: Specify a name for the pipeline.
   • Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.
   • Specify the output to use with that pipeline for forwarding the logs.
   • Optional: Specify the default output to forward logs to the internal Elasticsearch instance.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

1. Create a configuration file named secure-forward and specify parameters similar to the following within the <store> stanza:

   <store>
     @type forward
     <security>
       self_hostname ${hostname}
       shared_key <key> 1
     </security>
     transport tls 2
     tls_verify_hostname <value> 3
     tls_cert_path <path_to_file> 4
     <buffer> 5
       @type file
       path '/var/lib/fluentd/secureforwardlegacy'
       queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }"
       chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }"
       flush_interval "#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}"
       flush_at_shutdown "#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}"
       flush_thread_count "#{ENV['FLUSH_THREAD_COUNT'] || 2}"
       retry_max_interval "#{ENV['FORWARD_RETRY_WAIT'] || '300'}"
       retry_forever true
     </buffer>
     <server>
       name 6
       host 7
       hostlabel 8
       port 9
     </server>
     <server> 10
       name
       host
     </server>

   1

   Enter the shared key between nodes.

   2

   Specify tls to enable TLS validation.

   3

   Set to true to verify the server cert hostname. Set to false to ignore server cert hostname.

   4

   Specify the path to the private CA certificate file as /etc/ocp-forward/ca_cert.pem.

   5

   Specify the Fluentd buffer parameters as needed.

   6

   Optionally, enter a name for this server.

   7

   Specify the hostname or IP of the server.

   8

   Specify the host label of the server.

   9

   Specify the port of the server.

   10

   Optionally, add additional servers. If you specify two or more servers, forward uses these server nodes in a round-robin order.

   To use Mutual TLS (mTLS) authentication, see the Fluentd documentation for information about client certificate, key parameters, and other settings.

2. Create a config map named secure-forward in the openshift-logging project from the configuration file:

   $ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging

1. Create a configuration file named syslog.conf and specify parameters similar to the following within the <store> stanza:

   <store>
   @type <type> 1
   remote_syslog <syslog-server> 2
   port 514 3
   hostname ${hostname}
   remove_tag_prefix <prefix> 4
   facility <value>
   severity <value>
   use_record <value>
   payload_key message
   rfc 3164 5
   </store>

   1

   Specify the protocol to use, either: syslog or syslog_buffered.

   2

   Specify the FQDN or IP address of the syslog server.

   3

   Specify the port of the syslog server.

   4

   Optional: Specify the appropriate syslog parameters, for example:

   • Parameter to remove the specified tag field from the syslog prefix.
   • Parameter to set the specified field as the syslog key.
   • Parameter to specify the syslog log facility or source.
   • Parameter to specify the syslog log severity.
   • Parameter to use the severity and facility from the record if available. If true, the container_name, namespace_name, and pod_name are included in the output content.
   • Parameter to specify the key to set the payload of the syslog message. Defaults to message.

   5

   With the legacy syslog method, you must specify 3164 for the rfc value.

2. Create a config map named syslog in the openshift-logging project from the configuration file:

   $ oc create configmap syslog --from-file=syslog.conf -n openshift-logging

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. To view the cluster logging status:

   1. Get the cluster logging status:

      $ oc get clusterlogging instance -o yaml

      Example output

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      
      ....
      
      status:  1
        collection:
          logs:
            fluentdStatus:
              daemonSet: fluentd  2
              nodes:
                fluentd-2rhqp: ip-10-0-169-13.ec2.internal
                fluentd-6fgjh: ip-10-0-165-244.ec2.internal
                fluentd-6l2ff: ip-10-0-128-218.ec2.internal
                fluentd-54nx5: ip-10-0-139-30.ec2.internal
                fluentd-flpnn: ip-10-0-147-228.ec2.internal
                fluentd-n2frh: ip-10-0-157-45.ec2.internal
              pods:
                failed: []
                notReady: []
                ready:
                - fluentd-2rhqp
                - fluentd-54nx5
                - fluentd-6fgjh
                - fluentd-6l2ff
                - fluentd-flpnn
                - fluentd-n2frh
        logstore: 3
          elasticsearchStatus:
          - ShardAllocationEnabled:  all
            cluster:
              activePrimaryShards:    5
              activeShards:           5
              initializingShards:     0
              numDataNodes:           1
              numNodes:               1
              pendingTasks:           0
              relocatingShards:       0
              status:                 green
              unassignedShards:       0
            clusterName:             elasticsearch
            nodeConditions:
              elasticsearch-cdm-mkkdys93-1:
            nodeCount:  1
            pods:
              client:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
              data:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
              master:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
      visualization:  4
          kibanaStatus:
          - deployment: kibana
            pods:
              failed: []
              notReady: []
              ready:
              - kibana-7fb4fd4cc9-f2nls
            replicaSets:
            - kibana-7fb4fd4cc9
            replicas: 1

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      Information on the Fluentd pods.

      3

      Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.

      4

      Information on the Kibana pods.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. View the status of the cluster logging environment:

   $ oc describe deployment cluster-logging-operator

   Example output

   Name:                   cluster-logging-operator
   
   ....
   
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   
   ....
   
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----

3. View the status of the cluster logging replica set:

   1. Get the name of a replica set:

      Example output

      $ oc get replicaset

      Example output

      NAME                                      DESIRED   CURRENT   READY   AGE
      cluster-logging-operator-574b8987df       1         1         1       159m
      elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      kibana-5bd5544f87                         1         1         1       157m

   2. Get the status of the replica set:

      $ oc describe replicaset cluster-logging-operator-574b8987df

      Example output

      Name:           cluster-logging-operator-574b8987df
      
      ....
      
      Replicas:       1 current / 1 desired
      Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      
      ....
      
      Events:
        Type    Reason            Age   From                   Message
        ----    ------            ----  ----                   -------
        Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. To view the status:

   1. Get the name of the log store instance:

      $ oc get Elasticsearch

      Example output

      NAME            AGE
      elasticsearch   5h9m

   2. Get the log store status:

      $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      Example output

      status: 1
        cluster: 2
          activePrimaryShards: 30
          activeShards: 60
          initializingShards: 0
          numDataNodes: 3
          numNodes: 3
          pendingTasks: 0
          relocatingShards: 0
          status: green
          unassignedShards: 0
        clusterHealth: ""
        conditions: [] 3
        nodes: 4
        - deploymentName: elasticsearch-cdm-zjf34ved-1
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-2
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-3
          upgradeStatus: {}
        pods: 5
          client:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          data:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          master:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        shardAllocationEnabled: all

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      The status of the log store:

      • The number of active primary shards.
      • The number of active shards.
      • The number of shards that are initializing.
      • The number of log store data nodes.
      • The total number of log store nodes.
      • The number of pending tasks.
      • The log store status: green, red, yellow.
      • The number of unassigned shards.

      3

      Any status conditions, if present. The log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the log store and proxy containers.
      • Container Terminated for both the log store and proxy containers.
      • Pod unschedulable. Also, a condition is shown for a number of issues, see Example condition messages.

      4

      The log store nodes in the cluster, with upgradeStatus.

      5

      The log store client, data, and master pods in the cluster, listed under 'failed`, notReady or ready state.

1. In the OpenShift Container Platform console, click Monitoring → Alerting.
2. Click the Alerts tab. The alerts are listed, based on the filters selected.

1. Enable debug log of Curator:

   $ oc set env cronjob/curator CURATOR_LOG_LEVEL=DEBUG CURATOR_SCRIPT_LOG_LEVEL=DEBUG

   Specify the log level:

   • CRITICAL. Curator displays only critical messages.
   • ERROR. Curator displays only error and critical messages.
   • WARNING. Curator displays only error, warning, and critical messages.
   • INFO. Curator displays only informational, error, warning, and critical messages.

   • DEBUG. Curator displays only debug messages, in addition to all of the above.

     The default value is INFO.

     Note

     Cluster logging uses the OpenShift Container Platform custom environment variable CURATOR_SCRIPT_LOG_LEVEL in OpenShift Container Platform wrapper scripts (run.sh and convert.py). The environment variable takes the same values as CURATOR_LOG_LEVEL for script debugging, as needed.

2. Trigger next curator iteration:

   $ oc create job --from=cronjob/curator <job_name>

3. Use the following commands to control the cron job:

   • Suspend a cron job:

     $ oc patch cronjob curator -p '{"spec":{"suspend":true}}'

   • Resume a cron job:

     $ oc patch cronjob curator -p '{"spec":{"suspend":false}}'

   • Change a cron job schedule:

     $ oc patch cronjob curator -p '{"spec":{"schedule":"0 0 * * *"}}' 1

     1

     The schedule option accepts schedules in cron format.

1. Navigate to the directory where you want to store the must-gather information.

2. Run the oc adm must-gather command against the cluster logging image:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   The must-gather tool creates a new directory that starts with must-gather.local within the current directory. For example: must-gather.local.4157245944708210408.

3. Create a compressed file from the must-gather directory that was just created. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar -cvaf must-gather.tar.gz must-gather.local.4157245944708210408

4. Attach the compressed file to your support case on the Red Hat Customer Portal.