Chapter 3. Configuring your cluster logging deployment

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources:
             limits: 1
               memory: 736Mi
             requests:
               cpu: 100m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests as needed. The values shown are the default values.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

2. If they are present, remove the logStore, visualization, curation stanzas from the ClusterLogging CR.

3. Preserve the collection stanza of the ClusterLogging CR. The result should look similar to the following example:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     collection:
       logs:
         type: "fluentd"
         fluentd: {}

4. Verify that the Fluentd pods are redeployed:

   $ oc get pods -n openshift-logging

1. If the Log Forwarding API is not enabled:

   1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

      $ oc edit ClusterLogging instance

   2. Add the clusterlogging.openshift.io/logforwardingtechpreview annotation and set to enabled:

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        annotations:
          clusterlogging.openshift.io/logforwardingtechpreview: enabled 1
        name: "instance"
        namespace: "openshift-logging"
      spec:
      
      ...
      
        collection: 2
          logs:
            type: "fluentd"
            fluentd: {}

      1

      Enables and disables the Log Forwarding API. Set to enabled to use log forwarding.

      2

      The spec.collection block must be defined to use Fluentd in the ClusterLogging CR.

2. Create a LogForwarding CR YAML file or edit your existing CR:

   • Create a CR to send all log types to the internal Elasticsearch instance. You can use the following example without making any changes:

     apiVersion: logging.openshift.io/v1alpha1
     kind: LogForwarding
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       disableDefaultForwarding: true
       outputs:
         - name: clo-es
           type: elasticsearch
           endpoint: 'elasticsearch.openshift-logging.svc:9200' 1
           secret:
             name: fluentd
       pipelines:
         - name: audit-pipeline 2
           inputSource: logs.audit
           outputRefs:
             - clo-es
         - name: app-pipeline 3
           inputSource: logs.app
           outputRefs:
             - clo-es
         - name: infra-pipeline 4
           inputSource: logs.infra
           outputRefs:
             - clo-es

     1

     The endpoint parameter points to the internal Elasticsearch instance.

     2

     This parameter sends the audit logs to the specified endpoint.

     3

     This parameter sends the application logs to the specified endpoint.

     4

     This parameter sends the infrastructure logs to the specified endpoint.

     Note

     You must configure a pipeline and output for all three types of logs: application, infrastructure, and audit. If you do not specify a pipeline and output for a log type, those logs are not stored and will be lost.

   • If you have an existing LogForwarding CR, add an output for the internal Elasticsearch instance and a pipeline to that output for the audit logs. For example:

     apiVersion: "logging.openshift.io/v1alpha1"
     kind: "LogForwarding"
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       disableDefaultForwarding: true
       outputs:
        - name: elasticsearch 1
          type: "elasticsearch"
          endpoint: elasticsearch.openshift-logging.svc:9200
          secret:
             name: fluentd
        - name: elasticsearch-insecure
          type: "elasticsearch"
          endpoint: elasticsearch-insecure.messaging.svc.cluster.local
          insecure: true
        - name: secureforward-offcluster
          type: "forward"
          endpoint: https://secureforward.offcluster.com:24224
          secret:
             name: secureforward
       pipelines:
        - name: container-logs
          inputSource: logs.app
          outputRefs:
          - secureforward-offcluster
        - name: infra-logs
          inputSource: logs.infra
          outputRefs:
          - elasticsearch-insecure
        - name: audit-logs 2
          inputSource: logs.audit
          outputRefs:
          - elasticsearch

     1

     An output for the internal Elasticsearch instance.

     2

     A pipeline for sending the audit logs to the internal Elasticsearch instance.

1. Edit the ClusterLogging CR to add or modify the retentionPolicy parameter:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   ...
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       retentionPolicy: 1
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
   ...

   1

   Specify the time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 1d for one day. Logs older than the maxAge are deleted. By default, logs are retained for seven days.

2. You can verify the settings in the Elasticsearch custom resource (CR).

   For example, the Cluster Logging Operator updated the following Elasticsearch CR to configure a retention policy that includes settings to roll over active indices for the infrastructure logs every eight hours and the rolled-ver indices are deleted seven days after rollover. OpenShift Container Platform checks every 15 minutes to determine if the indices need to be rolled over.

   apiVersion: "logging.openshift.io/v1"
   kind: "Elasticsearch"
   metadata:
     name: "elasticsearch"
   spec:
   ...
     indexManagement:
       policies: 1
         - name: infra-policy
           phases:
             delete:
               minAge: 7d 2
             hot:
               actions:
                 rollover:
                   maxAge: 8h 3
           pollInterval: 15m 4
   ...

   1

   For each log source, the retention policy indicates when to delete and rollover logs for that source.

   2

   When OpenShift Container Platform deletes the rolled-over indices. This setting is the maxAge you set in the ClusterLogging CR.

   3

   The index age for OpenShift Container Platform to consider when rolling over the indices. This value is determined from the maxAge you set in the ClusterLogging CR.

   4

   When OpenShift Container Platform checks if the indices should be rolled over. This setting is the default and cannot be changed.

   Note

   Modifying the Elasticsearch CR is not supported. All changes to the retention policies must be made in the ClusterLogging CR.

   The Elasticsearch Operator deploys a cron job to roll over indices for each mapping using the defined policy, scheduled using the pollInterval.

   $ oc get cronjobs

   Example output

   NAME                           SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   elasticsearch-delete-app       */15 * * * *   False     0        <none>          27s
   elasticsearch-delete-audit     */15 * * * *   False     0        <none>          27s
   elasticsearch-delete-infra     */15 * * * *   False     0        <none>          27s
   elasticsearch-rollover-app     */15 * * * *   False     0        <none>          27s
   elasticsearch-rollover-audit   */15 * * * *   False     0        <none>          27s
   elasticsearch-rollover-infra   */15 * * * *   False     0        <none>          27s

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           resources: 1
             limits:
               memory: 16Gi
             requests:
               cpu: "1"
               memory: "64Gi"
           proxy: 2
             resources:
               limits:
                 memory: 100Mi
               requests:
                 memory: 100Mi

   1

   Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

   2

   Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the ClusterLogging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage:
             storageClassName: "gp2"
             size: "200G"

Procedure

1. Edit the ClusterLogging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Get the names of the Elasticsearch pods:

   $ oc get pods | grep elasticsearch-

3. Perform a shard synced flush using the OpenShift Container Platform es_util tool to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   For example:

   $ oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6  -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   Example output

   {"_shards":{"total":4,"successful":4,"failed":0},".security":{"total":2,"successful":2,"failed":0},".kibana_1":{"total":2,"successful":2,"failed":0}}

4. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Example output

   {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":

5. After the command is complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 resumed

      A new pod is deployed. After the pod has a ready container, you can move on to the next deployment.

      $ oc get pods | grep elasticsearch-

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   2. After the deployments are complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 paused

   3. Check that the Elasticsearch cluster is in a green or yellow state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Note

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      {
        "cluster_name" : "elasticsearch",
        "status" : "yellow", 1
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      1

      Make sure this parameter value is green or yellow before proceeding.

6. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

7. After all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Example output

   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Extract the CA certificate from the log store and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca

   Example output

   admin-ca

3. Create the route for the log store service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 1

      1

      Add the log store CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the log store CA certificate to the route YAML you created:

      $ cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml

      Example output

      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this service account to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}"

      The response appears similar to the following:

      Example output

      {
        "name" : "elasticsearch-cdm-i40ktba0-1",
        "cluster_name" : "elasticsearch",
        "cluster_uuid" : "0eY-tJzcR3KOdpgeMJo-MQ",
        "version" : {
        "number" : "6.8.1",
        "build_flavor" : "oss",
        "build_type" : "zip",
        "build_hash" : "Unknown",
        "build_date" : "Unknown",
        "build_snapshot" : true,
        "lucene_version" : "7.7.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
      },
        "<tagline>" : "<for search>"
      }

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance -n openshift-logging

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 2
         resources: 1
           limits:
             memory: 2Gi
           requests:
             cpu: 200m
             memory: 2Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 2
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 3
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
     curation:
       type: "curator"
       curator:
         resources: 4
           limits:
             memory: 200Mi
           requests:
             cpu: 200m
             memory: 200Mi
         schedule: "*/10 * * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd:
           resources: 5
             limits:
               memory: 736Mi
             requests:
               cpu: 200m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log curator as needed.

   5

   Specify the CPU and memory limits and requests for the log collector as needed.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas: 1 1

   1

   Specify the number of Kibana nodes.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the log visualizer pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and remove existing pods that do not match.

2. Edit the visualization section of the ClusterLogging CR to configure a toleration for the Kibana pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance -n openshift-logging

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 2
         resources: 1
           limits:
             memory: 2Gi
           requests:
             cpu: 200m
             memory: 2Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 2
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 3
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
     curation:
       type: "curator"
       curator:
         resources: 4
           limits:
             memory: 200Mi
           requests:
             cpu: 200m
             memory: 200Mi
         schedule: "*/10 * * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd:
           resources: 5
             limits:
               memory: 736Mi
             requests:
               cpu: 200m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log curator as needed.

   5

   Specify the CPU and memory limits and requests for the log collector as needed.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the cluster logging pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 elasticsearch=node:NoExecute

   This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only pods that match the taint and remove existing pods that do not match.

2. Edit the logstore section of the ClusterLogging CR to configure a toleration for the Elasticsearch pods:

     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 1
         tolerations:
         - key: "elasticsearch"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the log visualizer pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and remove existing pods that do not match.

2. Edit the visualization section of the ClusterLogging CR to configure a toleration for the Kibana pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want logging collector pods to schedule logging collector pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 collector=node:NoExecute

   This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and removes existing pods that do not match.

2. Edit the collection stanza of the ClusterLogging custom resource (CR) to configure a toleration for the logging collector pods:

     collection:
       logs:
         type: "fluentd"
         fluentd:
           tolerations:
           - key: "collector"  1
             operator: "Exists"  2
             effect: "NoExecute"  3
             tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ...
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     curation:
       curator:
         nodeSelector: 1
           node-role.kubernetes.io/infra: ''
         resources: null
         schedule: 30 3 * * *
       type: curator
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 2
           node-role.kubernetes.io/infra: ''
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 3
           node-role.kubernetes.io/infra: ''
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ...

Procedure

1. Create a journald.conf file with the required settings:

   Compress=yes 1
   ForwardToConsole=no 2
   ForwardToSyslog=no
   MaxRetentionSec=1month 3
   RateLimitBurst=10000 4
   RateLimitInterval=1s
   Storage=persistent 5
   SyncIntervalSec=1s 6
   SystemMaxUse=8g 7
   SystemKeepFree=20% 8
   SystemMaxFileSize=10M 9

   1

   Specify whether you want logs compressed before they are written to the file system. Specify yes to compress the message or no to not compress. The default is yes.

   2

   Configure whether to forward log messages. Defaults to no for each. Specify:

   • ForwardToConsole to forward logs to the system console.
   • ForwardToKsmg to forward logs to the kernel log buffer.
   • ForwardToSyslog to forward to a syslog daemon.
   • ForwardToWall to forward messages as wall messages to all logged-in users.

   3

   Specify the maximum time to store journal entries. Enter a number to specify seconds. Or include a unit: "year", "month", "week", "day", "h" or "m". Enter 0 to disable. The default is 1month.

   4

   Configure rate limiting. If, during the time interval defined by RateLimitIntervalSec, more logs than specified in RateLimitBurst are received, all further messages within the interval are dropped until the interval is over. It is recommended to set RateLimitInterval=1s and RateLimitBurst=10000, which are the defaults.

   5

   Specify how logs are stored. The default is persistent:

   • volatile to store logs in memory in /var/log/journal/.
   • persistent to store logs to disk in /var/log/journal/. systemd creates the directory if it does not exist.
   • auto to store logs in in /var/log/journal/ if the directory exists. If it does not exist, systemd temporarily stores logs in /run/systemd/journal.
   • none to not store logs. systemd drops all logs.

   6

   Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs. systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is 1s.

   7

   Specify the maximum size the journal can use. The default is 8g.

   8

   Specify how much disk space systemd must leave free. The default is 20%.

   9

   Specify the maximum size for individual journal files stored persistently in /var/log/journal. The default is 10M.

   Note

   If you are removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.

   For more information on systemd settings, see https://www.freedesktop.org/software/systemd/man/journald.conf.html. The default settings listed on that page might not apply to OpenShift Container Platform.

2. Convert the journal.conf file to base64:

   $ export jrnl_cnf=$( cat /journald.conf | base64 -w0 )

3. Create a new MachineConfig object for master or worker and add the journal.conf parameters:

   For example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 50-corp-journald
   spec:
     config:
       ignition:
         version: 2.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,${jrnl_cnf}
           mode: 0644 1
           overwrite: true
           path: /etc/systemd/journald.conf 2

   1

   Set the permissions for the journal.conf file. It is recommended to set 0644 permissions.

   2

   Specify the path to the base64-encoded journal.conf file.

4. Create the machine config:

   $ oc apply -f <filename>.yaml

   The controller detects the new MachineConfig object and generates a new rendered-worker-<hash> version.

5. Monitor the status of the rollout of the new rendered configuration to each node:

   $ oc describe machineconfigpool/worker

   Example output

   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   
   ...
   
   Conditions:
     Message:
     Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e

1. Edit the ClusterLogging custom resource in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ...
   
     curation:
       curator:
         schedule: 30 3 * * * 1
       type: curator

   1

   Specify the schedule for Curator in cron format.

   Note

   The time zone is set based on the host node where the Curator pod runs.

1. Edit the OpenShift Container Platform custom Curator configuration file:

   $ oc edit configmap/curator

2. Set the following parameters as needed:

   config.yaml: |
     project_name:
       action
         unit:value

   The available parameters are:

   Table 3.2. Project options

   Variable Name	Description
   project_name	The actual name of a project, such as myapp-devel. For OpenShift Container Platform operations logs, use the name .operations as the project name.
   action	The action to take, currently only delete is allowed.
   unit	The period to use for deletion, days, weeks, or months.
   value	The number of units.

   Table 3.3. Filter options

   Variable Name	Description
   .defaults	Use .defaults as the project_name to set the defaults for projects that are not specified.
   .regex	The list of regular expressions that match project names.
   pattern	The valid and properly escaped regular expression pattern enclosed by single quotation marks.