Chapter 7. Configuring your cluster logging deployment

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed" 1

   1

   Specify the management state as Managed or Unmanaged.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           resources: 1
             limits:
               memory: 16Gi
             requests:
               cpu: 500m
               memory: 16Gi

   1

   Specify the CPU and memory limits as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments.

   If you adjust the amount of Elasticsearch CPU and memory, you must change both the request value and the limit value.

   For example:

         resources:
           limits:
             cpu: "8"
             memory: "32Gi"
           requests:
             cpu: "8"
             memory: "32Gi"

   Kubernetes generally adheres the node CPU configuration and DOES not allow Elasticsearch to use the specified limits. Setting the same value for the requests and limits ensures that Elasticseach can use the CPU and memory you want, assuming the node has the CPU and memory available.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the Cluster Logging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage:
             storageClassName: "gp2"
             size: "200G"

Procedure

1. Edit the Cluster Logging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Extract the CA certificate from Elasticsearch and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca
   
   admin-ca

3. Create the route for the Elasticsearch service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 1

      1

      Add the Elasticsearch CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the Elasticsearch CA certificate to the route YAML you created:

      cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml
      
      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this ServiceAccount to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}/.operations.*/_search?size=1" | jq

      The response appears similar to the following:

        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   944  100   944    0     0     62      0  0:00:15  0:00:15 --:--:--   204
      {
        "took": 441,
        "timed_out": false,
        "_shards": {
          "total": 3,
          "successful": 3,
          "skipped": 0,
          "failed": 0
        },
        "hits": {
          "total": 89157,
          "max_score": 1,
          "hits": [
            {
              "_index": ".operations.2019.03.15",
              "_type": "com.example.viaq.common",
              "_id": "ODdiNWIyYzAtMjg5Ni0TAtNWE3MDY1MjMzNTc3",
              "_score": 1,
              "_source": {
                "_SOURCE_MONOTONIC_TIMESTAMP": "673396",
                "systemd": {
                  "t": {
                    "BOOT_ID": "246c34ee9cdeecb41a608e94",
                    "MACHINE_ID": "e904a0bb5efd3e36badee0c",
                    "TRANSPORT": "kernel"
                  },
                  "u": {
                    "SYSLOG_FACILITY": "0",
                    "SYSLOG_IDENTIFIER": "kernel"
                  }
                },
                "level": "info",
                "message": "acpiphp: Slot [30] registered",
                "hostname": "localhost.localdomain",
                "pipeline_metadata": {
                  "collector": {
                    "ipaddr4": "10.128.2.12",
                    "ipaddr6": "fe80::xx:xxxx:fe4c:5b09",
                    "inputname": "fluent-plugin-systemd",
                    "name": "fluentd",
                    "received_at": "2019-03-15T20:25:06.273017+00:00",
                    "version": "1.3.2 1.6.0"
                  }
                },
                "@timestamp": "2019-03-15T20:00:13.808226+00:00",
                "viaq_msg_id": "ODdiNWIyYzAtMYTAtNWE3MDY1MjMzNTc3"
              }
            }
          ]
        }
      }

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas:
         resources:  1
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:  2
           resources:
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi

   1

   Specify the CPU and memory limits to allocate for each node.

   2

   Specify the CPU and memory limits to allocate to the Kibana proxy.

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas: 1 1

   1

   Specify the number of Kibana nodes.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Kibana Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the visualization section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Kibana Pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

1. If necessary, get the Kibana route, which is created by default upon installation of the Cluster Logging Operator:

   $ oc get routes -n openshift-logging
   
   NAMESPACE                  NAME                       HOST/PORT                                                            PATH     SERVICES                   PORT    TERMINATION          WILDCARD
   openshift-logging          kibana                     kibana-openshift-logging.apps.openshift.com                                   kibana                     <all>   reencrypt/Redirect   None

2. Get the name of your Elasticsearch pods.

   $ oc get pods -l component=elasticsearch
   
   NAME                                            READY   STATUS    RESTARTS   AGE
   elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
   elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
   elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

3. Create the necessary per-user configuration that this procedure requires:

   1. Log in to the Kibana dashboard as the user you want to add the dashboards to.

      https://kibana-openshift-logging.apps.openshift.com 1

      1

      Where the URL is Kibana route.

   2. If the Authorize Access page appears, select all permissions and click Allow selected permissions.
   3. Log out of the Kibana dashboard.

4. Run the following command from the project where the pod is located using the name of any of your Elastiscearch pods:

   $ oc exec <es-pod> -- es_load_kibana_ui_objects <user-name>

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k -- es_load_kibana_ui_objects <user-name>

1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ...
   
     curation:
       curator:
         schedule: 30 3 * * * 1
       type: curator

   1

   Specify the schedule for Curator in cron format.

   Note

   The time zone is set based on the host node where the Curator pod runs.

1. Edit the OpenShift Container Platform custom Curator configuration file:

   $ oc edit configmap/curator

2. Set the following parameters as needed:

   config.yaml: |
     project_name:
       action
         unit:value

   The available parameters are:

   Table 7.1. Project options

   Variable Name	Description
   project_name	The actual name of a project, such as myapp-devel. For OpenShift Container Platform operations logs, use the name .operations as the project name.
   action	The action to take, currently only delete is allowed.
   unit	The period to use for deletion, days, weeks, or months.
   value	The number of units.

   Table 7.2. Filter options

   Variable Name	Description
   .defaults	Use .defaults as the project_name to set the defaults for projects that are not specified.
   .regex	The list of regular expressions that match project names.
   pattern	The valid and properly escaped regular expression pattern enclosed by single quotation marks.

1. Enable debug log of Curator:

   $ oc set env cronjob/curator CURATOR_LOG_LEVEL=DEBUG CURATOR_SCRIPT_LOG_LEVEL=DEBUG

   Specify the log level:

   • CRITICAL. Curator displays only critical messages.
   • ERROR. Curator displays only error and critical messages.
   • WARNING. Curator displays only error, warning, and critical messages.
   • INFO. Curator displays only informational, error, warning, and critical messages.

   • DEBUG. Curator displays only debug messages, in addition to all of the above.

     The default value is INFO.

     Note

     Cluster logging uses the OpenShift Container Platform custom environment variable CURATOR_SCRIPT_LOG_LEVEL in OpenShift Container Platform wrapper scripts (run.sh and convert.py). The environment variable takes the same values as CURATOR_LOG_LEVEL for script debugging, as needed.

2. Trigger next curator iteration:

   $ oc create job --from=cronjob/curator <job_name>

3. Use the following commands to control the CronJob:

   • Suspend a CronJob:

     $ oc patch cronjob curator -p '{"spec":{"suspend":true}}'

   • Resume a CronJob:

     $ oc patch cronjob curator -p '{"spec":{"suspend":false}}'

   • Change a CronJob schedule:

     $ oc patch cronjob curator -p '{"spec":{"schedule":"0 0 * * *"}}' 1

     1

     The schedule option accepts schedules in cron format.

1. Edit the Curator ConfigMap:

   oc edit cm/curator -n openshift-logging

2. Make the following changes to the action file:

   actions:
   1:
         action: delete_indices 1
         description: >-
           Delete .operations indices older than 30 days.
           Ignore the error if the filter does not
           result in an actionable list of indices (ignore_empty_list).
           See https://www.elastic.co/guide/en/elasticsearch/client/curator/5.2/ex_delete_indices.html
         options:
           # Swallow curator.exception.NoIndices exception
           ignore_empty_list: True
           # In seconds, default is 300
           timeout_override: ${CURATOR_TIMEOUT}
           # Don't swallow any other exceptions
           continue_if_exception: False
           # Optionally disable action, useful for debugging
           disable_action: False
         # All filters are bound by logical AND
         filters:            2
         - filtertype: pattern
           kind: regex
           value: '^\.operations\..*$'
           exclude: False    3
         - filtertype: age
           # Parse timestamp from index name
           source: name
           direction: older
           timestring: '%Y.%m.%d'
           unit: days
           unit_count: 30
           exclude: False

   1

   Specify delete_indices to delete the specified index.

   2

   Use the filers parameters to specify the index to be deleted. See the Elastic Search curator documentation for information on these parameters.

   3

   Specify false to allow the index to be deleted.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources:
             limits: 1
               memory: 736Mi
             requests:
               cpu: 100m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests as needed. The values shown are the default values.

1. Edit either of the following parameters in the fluentd Daemonset.

   spec:
     template:
       spec:
         containers:
             env:
             - name: FILE_BUFFER_LIMIT 1
               value: "256"
             - name: BUFFER_SIZE_LIMIT 2
               value: 8Mi

   1

   Specify the Fluentd file buffer size per output.

   2

   Specify the maximum size of each Fluentd buffer chunk.

Procedure

1. Create a template for the Event Router:

   kind: Template
   apiVersion: v1
   metadata:
     name: eventrouter-template
     annotations:
       description: "A pod forwarding kubernetes events to cluster logging stack."
       tags: "events,EFK,logging,cluster-logging"
   objects:
     - kind: ServiceAccount 1
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
     - kind: ClusterRole 2
       apiVersion: v1
       metadata:
         name: event-reader
       rules:
       - apiGroups: [""]
         resources: ["events"]
         verbs: ["get", "watch", "list"]
     - kind: ClusterRoleBinding  3
       apiVersion: v1
       metadata:
         name: event-reader-binding
       subjects:
       - kind: ServiceAccount
         name: eventrouter
         namespace: ${NAMESPACE}
       roleRef:
         kind: ClusterRole
         name: event-reader
     - kind: ConfigMap 4
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
       data:
         config.json: |-
           {
             "sink": "stdout"
           }
     - kind: Deployment 5
       apiVersion: apps/v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
         labels:
           component: eventrouter
           logging-infra: eventrouter
           provider: openshift
       spec:
         selector:
           matchLabels:
             component: eventrouter
             logging-infra: eventrouter
             provider: openshift
         replicas: 1
         template:
           metadata:
             labels:
               component: eventrouter
               logging-infra: eventrouter
               provider: openshift
             name: eventrouter
           spec:
             serviceAccount: eventrouter
             containers:
               - name: kube-eventrouter
                 image: ${IMAGE}
                 imagePullPolicy: IfNotPresent
                 resources:
                   requests:
                     cpu: ${CPU}
                     memory: ${MEMORY}
                 volumeMounts:
                 - name: config-volume
                   mountPath: /etc/eventrouter
             volumes:
               - name: config-volume
                 configMap:
                   name: eventrouter
   parameters:
     - name: IMAGE
       displayName: Image
       value: "registry.redhat.io/openshift4/ose-logging-eventrouter:latest"
     - name: CPU  6
       displayName: CPU
       value: "100m"
     - name: MEMORY 7
       displayName: Memory
       value: "128Mi"
     - name: NAMESPACE
       displayName: Namespace
       value: "openshift-logging" 8

   1

   Creates a Service Account in the openshift-logging project for the Event Router.

   2

   Creates a ClusterRole to monitor for events in the cluster.

   3

   Creates a ClusterRoleBinding to bind the ClusterRole to the ServiceAccount.

   4

   Creates a ConfigMap in the openshift-logging project to generate the required config.json file.

   5

   Creates a Deployment in the openshift-logging project to generate and configure the Event Router Pod.

   6

   Specifies the minimum amount of memory to allocate to the Event Router Pod. Defaults to 128Mi.

   7

   Specifies the minimum amount of CPU to allocate to the Event Router Pod. Defaults to 100m.

   8

   Specifies the openshift-logging project to install objects in.

2. Use the following command to process and apply the template:

   $ oc process -f <templatefile> | oc apply -f -

   For example:

   $ oc process -f eventrouter.yaml | oc apply -f -

   Example output

   serviceaccount/logging-eventrouter created
   clusterrole.authorization.openshift.io/event-reader created
   clusterrolebinding.authorization.openshift.io/event-reader-binding created
   configmap/logging-eventrouter created
   deployment.apps/logging-eventrouter created

3. Validate that the Event Router installed in the openshift-logging project:

   1. View the new Event Router Pod:

      $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Example output

      pod/cluster-logging-eventrouter-d649f97c8-qvv8r

   2. View the events collected by the Event Router:

      $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      For example:

      $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Example output

      {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch infra index.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the cluster logging Pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 elasticsearch=node:NoExecute

   This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only Pods that match the taint and remove existing Pods that do not match.

2. Edit the logstore section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Elasticsearch Pods:

     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 1
         tolerations:
         - key: "elasticsearch"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Kibana Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the visualization section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Kibana Pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Curator Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 curator=node:NoExecute

   This example places a taint on node1 that has key curator, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the curation section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Curator Pod:

     curation:
       type: "curator"
       curator:
         tolerations:
         - key: "curator"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want logging collector Pods to schedule logging collector Pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 collector=node:NoExecute

   This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and removes existing Pods that do not match.

2. Edit the collection section of the Cluster Logging Custom Resource (CR) to configure a toleration for the logging collector Pods:

     collection:
       logs:
         type: "fluentd"
         rsyslog:
           tolerations:
           - key: "collector"  1
             operator: "Exists"  2
             effect: "NoExecute"  3
             tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

1. Create a configuration file named secure-forward.conf for the forward parameters:

   1. Configure the secrets and TLS information:

       <store>
        @type forward
      
        self_hostname ${hostname} 1
        shared_key <SECRET_STRING> 2
      
        transport tls 3
      
        tls_verify_hostname true 4
        tls_cert_path <path_to_file> 5

      1

      Specify the default value of the auto-generated certificate common name (CN).

      2

      Enter the Shared key between nodes

      3

      Specify tls to enable TLS validation.

      4

      Set to true to verify the server cert hostname. Set to false to ignore server cert hostname.

      5

      Specify the path to private CA certificate file as /etc/ocp-forward/ca_cert.pem.

      To use mTLS, see the Fluentd documentation for information about client certificate, key parameters, and other settings.

   2. Configure the name, host, and port for your external Fluentd server:

        <server>
          name 1
          host 2
          hostlabel 3
          port 4
        </server>
        <server> 5
          name
          host
        </server>

      1

      Optionally, enter a name for this server.

      2

      Specify the host name or IP of the server.

      3

      Specify the host label of the server.

      4

      Specify the port of the server.

      5

      Optionally, add additional servers. If you specify two or more servers, forward uses these server nodes in a round-robin order.

      For example:

        <server>
          name externalserver1
          host 192.168.1.1
          hostlabel externalserver1.example.com
          port 24224
        </server>
        <server>
          name externalserver2
          host externalserver2.example.com
          port 24224
        </server>
        </store>

2. Create a ConfigMap named secure-forward in the openshift-logging namespace from the configuration file:

   $ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging

3. Optional: Import any secrets required for the receiver:

   $ oc create secret generic secure-forward --from-file=<arbitrary-name-of-key1>=cert_file_from_fluentd_receiver --from-literal=shared_key=value_from_fluentd_receiver

   For example:

   $ oc create secret generic secure-forward --from-file=ca-bundle.crt=ca-for-fluentd-receiver/ca.crt --from-literal=shared_key=fluentd-receiver

4. Refresh the fluentd Pods to apply the secure-forward secret and secure-forward ConfigMap:

   $ oc delete pod --selector logging-infra=fluentd

5. Configure the external log aggregator to accept messages securely from OpenShift Container Platform.

1. Create a configuration file named syslog.conf that contains the following parameters within the <store> stanza:

   1. Specify the syslog protocol type:

      @type syslog_buffered 1

      1

      Specify the protocol to use, either: syslog or syslog_buffered.

   2. Configure the name, host, and port for your external syslog server:

      remote_syslog <remote> 1
      port <number> 2
      hostname <name> 3

      1

      Specify the FQDN or IP address of the syslog server.

      2

      Specify the port of the syslog server.

      3

      Specify a name for this syslog server.

      For example:

      remote_syslog syslogserver.openshift-logging.svc.cluster.local
      port 514
      hostname fluentd-server

   3. Configure the other syslog variables as needed:

      remove_tag_prefix 1
      tag_key <key> 2
      facility <value>  3
      severity <value>  4
      use_record <value> 5
      payload_key message 6

      1

      Add this parameter to remove the tag field from the syslog prefix.

      2

      Specify the field to set the syslog key.

      3

      Specify the syslog log facility or source. For values, see RTF 3164.

      4

      Specify the syslog log severity. For values, see link:RTF 3164.

      5

      Specify true to use the severity and facility from the record if available. If true, the container_name, namespace_name, and pod_name are included in the output content.

      6

      Specify the key to set the payload of the syslog message. Defaults to message.

      For example:

      facility local0
      severity info

      The configuration file appears similar to the following:

      <store>
      @type syslog_buffered
      remote_syslog syslogserver.openshift-logging.svc.cluster.local
      port 514
      hostname fluentd-4nzfz
      tag_key ident,systemd.u.SYSLOG_IDENTIFIER
      facility local0
      severity info
      use_record false
      </store>

2. Create a ConfigMap named syslog in the openshift-logging namespace from the configuration file:

   $ oc create configmap syslog --from-file=syslog.conf -n openshift-logging

   The Cluster Logging Operator redeploys the Fluentd Pods. If the Pods do not redeploy, you can delete the Fluentd Pods to force them to redeploy.

   $ oc delete pod --selector logging-infra=fluentd

Procedure

1. Create a journald.conf file with the required settings:

   Compress=yes 1
   ForwardToConsole=no 2
   ForwardToSyslog=no
   MaxRetentionSec=1month 3
   RateLimitBurst=10000 4
   RateLimitInterval=1s
   Storage=persistent 5
   SyncIntervalSec=1s 6
   SystemMaxUse=8g 7
   SystemKeepFree=20% 8
   SystemMaxFileSize=10M 9

   1

   Specify whether you want logs compressed before they are written to the file system. Specify yes to compress the message or no to not compress. The default is yes.

   2

   Configure whether to forward log messages. Defaults to no for each. Specify:

   • ForwardToConsole to forward logs to the system console.
   • ForwardToKsmg to forward logs to the kernel log buffer.
   • ForwardToSyslog to forward to a syslog daemon.
   • ForwardToWall to forward messages as wall messages to all logged-in users.

   3

   Specify the maximum time to store journal entries. Enter a number to specify seconds. Or include a unit: "year", "month", "week", "day", "h" or "m". Enter 0 to disable. The default is 1month.

   4

   Configure rate limiting. If, during the time interval defined by RateLimitIntervalSec, more logs than specified in RateLimitBurst are received, all further messages within the interval are dropped until the interval is over. It is recommended to set RateLimitInterval=1s and RateLimitBurst=10000, which are the defaults.

   5

   Specify how logs are stored. The default is persistent:

   • volatile to store logs in memory in /var/log/journal/.
   • persistent to store logs to disk in /var/log/journal/. systemd creates the directory if it does not exist.
   • auto to store logs in in /var/log/journal/ if the directory exists. If it does not exist, systemd temporarily stores logs in /run/systemd/journal.
   • none to not store logs. systemd drops all logs.

   6

   Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs. systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is 1s.

   7

   Specify the maximum size the journal can use. The default is 8g.

   8

   Specify how much disk space systemd must leave free. The default is 20%.

   9

   Specify the maximum size for individual journal files stored persistently in /var/log/journal. The default is 10M.

   Note

   If you are removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.

   For more information on systemd settings, see https://www.freedesktop.org/software/systemd/man/journald.conf.html. The default settings listed on that page might not apply to OpenShift Container Platform.

2. Convert the journal.conf file to base64:

   $ export jrnl_cnf=$( cat /journald.conf | base64 -w0 )

3. Create a new MachineConfig for master or worker and add the journal.conf parameters:

   For example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 50-corp-journald
   spec:
     config:
       ignition:
         version: 2.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,${jrnl_cnf}
           mode: 0644 1
           overwrite: true
           path: /etc/systemd/journald.conf 2

   1

   Set the permissions for the journal.conf file. It is recommended to set 0644 permissions.

   2

   Specify the path to the base64-encoded journal.conf file.

4. Create the MachineConfig:

   $ oc apply -f <filename>.yaml

   The controller detects the new MachineConfig and generates a new rendered-worker-<hash> version.

5. Monitor the status of the rollout of the new rendered configuration to each node:

   $ oc describe machineconfigpool/worker
   
   
   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   
   ...
   
   Conditions:
     Message:
     Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e