Chapter 10. Log storage

Procedure

1. Modify the ClusterLogging CR logStore spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     logStore:
       type: <log_store_type> 1
       elasticsearch: 2
         nodeCount: <integer>
         resources: {}
         storage: {}
         redundancyPolicy: <redundancy_type> 3
       lokistack: 4
         name: {}
   # ...

   1

   Specify the log store type. This can be either lokistack or elasticsearch.

   2

   Optional configuration options for the Elasticsearch log store.

   3

   Specify the redundancy type. This value can be ZeroRedundancy, SingleRedundancy, MultipleRedundancy, or FullRedundancy.

   4

   Optional configuration options for LokiStack.

   Example ClusterLogging CR to specify LokiStack as the log store

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     managementState: Managed
     logStore:
       type: lokistack
       lokistack:
         name: logging-loki
   # ...

2. Apply the ClusterLogging CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Enter the following command to create a new group:

   $ oc adm groups new cluster-admin

2. Enter the following command to add the desired user to the cluster-admin group:

   $ oc adm groups add-users cluster-admin <username>

3. Enter the following command to add cluster-admin user role to the group:

   $ oc adm policy add-cluster-role-to-group cluster-admin cluster-admin

1. To enable stream-based retention, create a LokiStack custom resource (CR):

   Example global stream-based retention

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global: 1
         retention: 2
           days: 20
           streams:
           - days: 4
             priority: 1
             selector: '{kubernetes_namespace_name=~"test.+"}' 3
           - days: 1
             priority: 1
             selector: '{log_type="infrastructure"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v11
       secret:
         name: logging-loki-s3
         type: aws
     storageClassName: standard
     tenants:
       mode: openshift-logging

   1

   Sets retention policy for all log streams. Note: This field does not impact the retention period for stored logs in object storage.

   2

   Retention is enabled in the cluster when this block is added to the CR.

   3

   Contains the LogQL query used to define the log stream.

   Example per-tenant stream-based retention

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global:
         retention:
           days: 20
       tenants: 1
         application:
           retention:
             days: 1
             streams:
               - days: 4
                 selector: '{kubernetes_namespace_name=~"test.+"}' 2
         infrastructure:
           retention:
             days: 5
             streams:
               - days: 1
                 selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v11
       secret:
         name: logging-loki-s3
         type: aws
     storageClassName: standard
     tenants:
       mode: openshift-logging

   1

   Sets retention policy by tenant. Valid tenant types are application, audit, and infrastructure.

   2

   Contains the LogQL query used to define the log stream.

2. Apply the LokiStack CR:

   $ oc apply -f <filename>.yaml

Procedure

1. Modify the ClusterLogging CR logStore spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     logStore:
       type: <log_store_type> 1
       elasticsearch: 2
         nodeCount: <integer>
         resources: {}
         storage: {}
         redundancyPolicy: <redundancy_type> 3
       lokistack: 4
         name: {}
   # ...

   1

   Specify the log store type. This can be either lokistack or elasticsearch.

   2

   Optional configuration options for the Elasticsearch log store.

   3

   Specify the redundancy type. This value can be ZeroRedundancy, SingleRedundancy, MultipleRedundancy, or FullRedundancy.

   4

   Optional configuration options for LokiStack.

   Example ClusterLogging CR to specify LokiStack as the log store

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     managementState: Managed
     logStore:
       type: lokistack
       lokistack:
         name: logging-loki
   # ...

2. Apply the ClusterLogging CR by running the following command:

   $ oc apply -f <filename>.yaml

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   • Create a CR to send all log types to the internal Elasticsearch instance. You can use the following example without making any changes:

     apiVersion: logging.openshift.io/v1
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       pipelines: 1
       - name: all-to-default
         inputRefs:
         - infrastructure
         - application
         - audit
         outputRefs:
         - default

     1

     A pipeline defines the type of logs to forward using the specified output. The default output forwards logs to the internal Elasticsearch instance.

     Note

     You must specify all three types of logs in the pipeline: application, infrastructure, and audit. If you do not specify a log type, those logs are not stored and will be lost.

   • If you have an existing ClusterLogForwarder CR, add a pipeline to the default output for the audit logs. You do not need to define the default output. For example:

     apiVersion: "logging.openshift.io/v1"
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       outputs:
        - name: elasticsearch-insecure
          type: "elasticsearch"
          url: http://elasticsearch-insecure.messaging.svc.cluster.local
          insecure: true
        - name: elasticsearch-secure
          type: "elasticsearch"
          url: https://elasticsearch-secure.messaging.svc.cluster.local
          secret:
            name: es-audit
        - name: secureforward-offcluster
          type: "fluentdForward"
          url: https://secureforward.offcluster.com:24224
          secret:
            name: secureforward
       pipelines:
        - name: container-logs
          inputRefs:
          - application
          outputRefs:
          - secureforward-offcluster
        - name: infra-logs
          inputRefs:
          - infrastructure
          outputRefs:
          - elasticsearch-insecure
        - name: audit-logs
          inputRefs:
          - audit
          outputRefs:
          - elasticsearch-secure
          - default 1

     1

     This pipeline sends the audit logs to the internal Elasticsearch instance in addition to an external instance.

1. Edit the ClusterLogging CR to add or modify the retentionPolicy parameter:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   ...
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       retentionPolicy: 1
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
   ...

   1

   Specify the time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 1d for one day. Logs older than the maxAge are deleted. By default, logs are retained for seven days.

2. You can verify the settings in the Elasticsearch custom resource (CR).

   For example, the Red Hat OpenShift Logging Operator updated the following Elasticsearch CR to configure a retention policy that includes settings to roll over active indices for the infrastructure logs every eight hours and the rolled-over indices are deleted seven days after rollover. OpenShift Container Platform checks every 15 minutes to determine if the indices need to be rolled over.

   apiVersion: "logging.openshift.io/v1"
   kind: "Elasticsearch"
   metadata:
     name: "elasticsearch"
   spec:
   ...
     indexManagement:
       policies: 1
         - name: infra-policy
           phases:
             delete:
               minAge: 7d 2
             hot:
               actions:
                 rollover:
                   maxAge: 8h 3
           pollInterval: 15m 4
   ...

   1

   For each log source, the retention policy indicates when to delete and roll over logs for that source.

   2

   When OpenShift Container Platform deletes the rolled-over indices. This setting is the maxAge you set in the ClusterLogging CR.

   3

   The index age for OpenShift Container Platform to consider when rolling over the indices. This value is determined from the maxAge you set in the ClusterLogging CR.

   4

   When OpenShift Container Platform checks if the indices should be rolled over. This setting is the default and cannot be changed.

   Note

   Modifying the Elasticsearch CR is not supported. All changes to the retention policies must be made in the ClusterLogging CR.

   The OpenShift Elasticsearch Operator deploys a cron job to roll over indices for each mapping using the defined policy, scheduled using the pollInterval.

   $ oc get cronjob

   Example output

   NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   elasticsearch-im-app     */15 * * * *   False     0        <none>          4s
   elasticsearch-im-audit   */15 * * * *   False     0        <none>          4s
   elasticsearch-im-infra   */15 * * * *   False     0        <none>          4s

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:1
           resources:
             limits: 2
               memory: "32Gi"
             requests: 3
               cpu: "1"
               memory: "16Gi"
           proxy: 4
             resources:
               limits:
                 memory: 100Mi
               requests:
                 memory: 100Mi

   1

   Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

   2

   The maximum amount of resources a pod can use.

   3

   The minimum resources required to schedule a pod.

   4

   Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that are sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the ClusterLogging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   # ...
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: "gp2"
           size: "200G"

Procedure

1. Edit the ClusterLogging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Get the names of the Elasticsearch pods:

   $ oc get pods -l component=elasticsearch

3. Scale down the collector pods so they stop sending new logs to Elasticsearch:

   $ oc -n openshift-logging patch daemonset/collector -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-collector": "false"}}}}}'

4. Perform a shard synced flush using the OpenShift Container Platform es_util tool to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   For example:

   $ oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6  -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   Example output

   {"_shards":{"total":4,"successful":4,"failed":0},".security":{"total":2,"successful":2,"failed":0},".kibana_1":{"total":2,"successful":2,"failed":0}}

5. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Example output

   {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":

6. After the command is complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 resumed

      A new pod is deployed. After the pod has a ready container, you can move on to the next deployment.

      $ oc get pods -l component=elasticsearch-

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   2. After the deployments are complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 paused

   3. Check that the Elasticsearch cluster is in a green or yellow state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Note

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      {
        "cluster_name" : "elasticsearch",
        "status" : "yellow", 1
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      1

      Make sure this parameter value is green or yellow before proceeding.

7. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

8. After all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Example output

   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

9. Scale up the collector pods so they send new logs to Elasticsearch.

   $ oc -n openshift-logging patch daemonset/collector -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-collector": "true"}}}}}'

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Extract the CA certificate from the log store and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca

   Example output

   admin-ca

3. Create the route for the log store service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 1

      1

      Add the log store CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the log store CA certificate to the route YAML you created in the previous step:

      $ cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml

      Example output

      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this service account to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}"

      The response appears similar to the following:

      Example output

      {
        "name" : "elasticsearch-cdm-i40ktba0-1",
        "cluster_name" : "elasticsearch",
        "cluster_uuid" : "0eY-tJzcR3KOdpgeMJo-MQ",
        "version" : {
        "number" : "6.8.1",
        "build_flavor" : "oss",
        "build_type" : "zip",
        "build_hash" : "Unknown",
        "build_date" : "Unknown",
        "build_snapshot" : true,
        "lucene_version" : "7.7.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
      },
        "<tagline>" : "<for search>"
      }

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

2. If they are present, remove the logStore and visualization stanzas from the ClusterLogging CR.

3. Preserve the collection stanza of the ClusterLogging CR. The result should look similar to the following example:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     collection:
       type: "fluentd"
       fluentd: {}

4. Verify that the collector pods are redeployed:

   $ oc get pods -l component=collector -n openshift-logging