Chapter 2. Logging 5.7

1. To enable stream-based retention, create or edit the LokiStack custom resource (CR):

1. You can refer to the examples below to configure your LokiStack CR.

Procedure

1. Create an AlertingRule CR:

   oc create -f <file-name>.yaml

2. Populate your AlertingRule CR using the appropriate example below:

   Example infrastructure AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat 1
       labels: 2
         openshift.io/cluster-monitoring: "true"
     spec:
       tenantID: "infrastructure" 3
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: | 4
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical 5
               annotations:
                 summary: High Loki Operator Reconciliation Errors 6
                 description: High Loki Operator Reconciliation Errors 7

   1

   The namespace where this AlertingRule is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   AlertingRules for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   4

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   5

   Mandatory field. Must be critical, warning, or info.

   6

   Mandatory field.

   7

   Mandatory field.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns 1
       labels: 2
         openshift.io/cluster-monitoring: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: | 3
               sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical 4
               annotations:
                 summary:  5
                 description:  6

   1

   The namespace where this AlertingRule is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   4

   Mandatory field. Must be critical, warning, or info.

   5

   Mandatory field. Summary of the rule.

   6

   Mandatory field. Detailed description of the rule.

3. Apply the CR.

   oc apply -f <file-name>.yaml

1. Install the Red Hat OpenShift Logging Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Type Logging in the Filter by keyword field.
   3. Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.

   4. Select stable or stable-5.y as the Update Channel.

      Note

      The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-X where X is the version of logging you have installed.

   5. Ensure that A specific namespace on the cluster is selected under Installation Mode.
   6. Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.
   7. Select Enable Operator recommended cluster monitoring on this Namespace.

   8. Select an option for Update approval.

      • The Automatic option allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual option requires a user with appropriate credentials to approve the Operator update.
   9. Select Enable or Disable for the Console plugin.
   10. Click Install.

2. Verify that the Red Hat OpenShift Logging Operator is installed by switching to the Operators → Installed Operators page.

   1. Ensure that Red Hat OpenShift Logging is listed in the openshift-logging project with a Status of Succeeded.

3. Create a ClusterLogging instance.

   Note

   The form view of the web console does not include all available options. The YAML view is recommended for completing your setup.

   1. In the collection section, select a Collector Implementation.

      Note

      As of logging version 5.6 Fluentd is deprecated and is planned to be removed in a future release. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed. As an alternative to Fluentd, you can use Vector instead.

   2. In the logStore section, select a type.

      Note

      As of logging version 5.4.3 the OpenShift Elasticsearch Operator is deprecated and is planned to be removed in a future release. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator.

   3. Click Create.

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.

2. Type Loki in the Filter by keyword field.

   1. Choose Loki Operator from the list of available Operators, and click Install.

3. Select stable or stable-5.y as the Update Channel.

   Note

   The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-X where X is the version of logging you have installed.

4. Ensure that All namespaces on the cluster is selected under Installation Mode.
5. Ensure that openshift-operators-redhat is selected under Installed Namespace.

6. Select Enable Operator recommended cluster monitoring on this Namespace.

   This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

7. Select an option for Update approval.

   • The Automatic option allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
   • The Manual option requires a user with appropriate credentials to approve the Operator update.
8. Click Install.

9. Verify that the LokiOperator installed by switching to the Operators → Installed Operators page.

   1. Ensure that LokiOperator is listed with Status as Succeeded in all the projects.

10. Create a Secret YAML file that uses the access_key_id and access_key_secret fields to specify your credentials and bucketnames, endpoint, and region to define the object storage location. AWS is used in the following example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: logging-loki-s3
      namespace: openshift-logging
    stringData:
      access_key_id: AKIAIOSFODNN7EXAMPLE
      access_key_secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      bucketnames: s3-bucket-name
      endpoint: https://s3.eu-central-1.amazonaws.com
      region: eu-central-1

11. Select Create instance under LokiStack on the Details tab. Then select YAML view. Paste in the following template, subsituting values where appropriate.

      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: logging-loki 1
        namespace: openshift-logging
      spec:
        size: 1x.small 2
        storage:
          schemas:
          - version: v12
            effectiveDate: '2022-06-01'
          secret:
            name: logging-loki-s3 3
            type: s3 4
        storageClassName: <storage_class_name> 5
        tenants:
          mode: openshift-logging

    1

    Name should be logging-loki.

    2

    Select your Loki deployment size.

    3

    Define the secret used for your log storage.

    4

    Define corresponding storage type.

    5

    Enter the name of an existing storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed using oc get storageclasses.

    1. Apply the configuration:

       oc apply -f logging-loki.yaml

12. Create or edit a ClusterLogging CR:

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        managementState: Managed
        logStore:
          type: lokistack
          lokistack:
            name: logging-loki
          collection:
            type: vector

    1. Apply the configuration:

       oc apply -f cr-lokistack.yaml

Procedure

1. View the list of Operators available to the cluster from OperatorHub:

   $ oc get packagemanifests -n openshift-marketplace

   Example output

   NAME                               CATALOG               AGE
   3scale-operator                    Red Hat Operators     91m
   advanced-cluster-management        Red Hat Operators     91m
   amq7-cert-manager                  Red Hat Operators     91m
   ...
   couchbase-enterprise-certified     Certified Operators   91m
   crunchy-postgres-operator          Certified Operators   91m
   mongodb-enterprise                 Certified Operators   91m
   ...
   etcd                               Community Operators   91m
   jaeger                             Community Operators   91m
   kubefed                            Community Operators   91m
   ...

   Note the catalog for your desired Operator.

2. Inspect your desired Operator to verify its supported install modes and available channels:

   $ oc describe packagemanifests <operator_name> -n openshift-marketplace

3. An Operator group, defined by an OperatorGroup object, selects target namespaces in which to generate required RBAC access for all Operators in the same namespace as the Operator group.

   The namespace to which you subscribe the Operator must have an Operator group that matches the install mode of the Operator, either the AllNamespaces or SingleNamespace mode. If the Operator you intend to install uses the AllNamespaces, then the openshift-operators namespace already has an appropriate Operator group in place.

   However, if the Operator uses the SingleNamespace mode and you do not already have an appropriate Operator group in place, you must create one.

   Note

   The web console version of this procedure handles the creation of the OperatorGroup and Subscription objects automatically behind the scenes for you when choosing SingleNamespace mode.

   1. Create an OperatorGroup object YAML file, for example operatorgroup.yaml:

      Example OperatorGroup object

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: <operatorgroup_name>
        namespace: <namespace>
      spec:
        targetNamespaces:
        - <namespace>

   2. Create the OperatorGroup object:

      $ oc apply -f operatorgroup.yaml

4. Create a Subscription object YAML file to subscribe a namespace to an Operator, for example sub.yaml:

   Example Subscription object

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: <subscription_name>
     namespace: openshift-operators 1
   spec:
     channel: <channel_name> 2
     name: <operator_name> 3
     source: redhat-operators 4
     sourceNamespace: openshift-marketplace 5
     config:
       env: 6
       - name: ARGS
         value: "-v=10"
       envFrom: 7
       - secretRef:
           name: license-secret
       volumes: 8
       - name: <volume_name>
         configMap:
           name: <configmap_name>
       volumeMounts: 9
       - mountPath: <directory_name>
         name: <volume_name>
       tolerations: 10
       - operator: "Exists"
       resources: 11
         requests:
           memory: "64Mi"
           cpu: "250m"
         limits:
           memory: "128Mi"
           cpu: "500m"
       nodeSelector: 12
         foo: bar

   1

   For default AllNamespaces install mode usage, specify the openshift-operators namespace. Alternatively, you can specify a custom global namespace, if you have created one. Otherwise, specify the relevant single namespace for SingleNamespace install mode usage.

   2

   Name of the channel to subscribe to.

   3

   Name of the Operator to subscribe to.

   4

   Name of the catalog source that provides the Operator.

   5

   Namespace of the catalog source. Use openshift-marketplace for the default OperatorHub catalog sources.

   6

   The env parameter defines a list of Environment Variables that must exist in all containers in the pod created by OLM.

   7

   The envFrom parameter defines a list of sources to populate Environment Variables in the container.

   8

   The volumes parameter defines a list of Volumes that must exist on the pod created by OLM.

   9

   The volumeMounts parameter defines a list of VolumeMounts that must exist in all containers in the pod created by OLM. If a volumeMount references a volume that does not exist, OLM fails to deploy the Operator.

   10

   The tolerations parameter defines a list of Tolerations for the pod created by OLM.

   11

   The resources parameter defines resource constraints for all the containers in the pod created by OLM.

   12

   The nodeSelector parameter defines a NodeSelector for the pod created by OLM.

5. Create the Subscription object:

   $ oc apply -f sub.yaml

   At this point, OLM is now aware of the selected Operator. A cluster service version (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

Procedure

1. Navigate to the Operators → Installed Operators page.
2. Scroll or enter a keyword into the Filter by name field to find the Operator that you want to remove. Then, click on it.

3. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

   An Uninstall Operator? dialog box is displayed.

4. Select Uninstall to remove the Operator, Operator deployments, and pods. Following this action, the Operator stops running and no longer receives updates.

   Note

   This action does not remove resources managed by the Operator, including custom resource definitions (CRDs) and custom resources (CRs). Dashboards and navigation items enabled by the web console and off-cluster resources that continue to run might need manual clean up. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

Procedure

1. Check the current version of the subscribed Operator (for example, jaeger) in the currentCSV field:

   $ oc get subscription jaeger -n openshift-operators -o yaml | grep currentCSV

   Example output

     currentCSV: jaeger-operator.v1.8.2

2. Delete the subscription (for example, jaeger):

   $ oc delete subscription jaeger -n openshift-operators

   Example output

   subscription.operators.coreos.com "jaeger" deleted

3. Delete the CSV for the Operator in the target namespace using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion jaeger-operator.v1.8.2 -n openshift-operators

   Example output

   clusterserviceversion.operators.coreos.com "jaeger-operator.v1.8.2" deleted