1.3. Enable observability service

Monitor the health of your managed clusters with the observability service (multicluster-observability-operator).

Required access: Cluster administrator or the open-cluster-management:cluster-manager-admin role.

Prerequisites:

  • You must install Red Hat Advanced Cluster Management for Kubernetes. See Installing while connected online for more information.
  • You must configure an object store to create a storage solution. Red Hat Advanced Cluster Management only supports cloud providers with stable object stores, such as Amazon S3 (or other S3 compatible object stores like Ceph), Google Cloud Storage, and Azure storage.

    Important: When you configure your object store, ensure that you meet the encryption requirements necessary when sensitive data is persisted. For a complete list of the supported object stores, see Thanos documentation.

1.3.1. Enabling observability

Enable the observability service by creating a MultiClusterObservability CustomResource (CR) instance. Complete the following steps to enable the observability service:

  1. Log in to your Red Hat Advanced Cluster Management hub cluster.
  2. Create a namespace for the observability service with the following command:

    oc create namespace open-cluster-management-observability
  3. Generate your pull-secret. If Red Hat Advanced Cluster Management is installed in the open-cluster-management namespace, run the following command:

    DOCKER_CONFIG_JSON=`oc extract secret/multiclusterhub-operator-pull-secret -n open-cluster-management --to=-`
    oc create secret generic multiclusterhub-operator-pull-secret \
        -n open-cluster-management-observability \
        --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
        --type=kubernetes.io/dockerconfigjson

    If the multiclusterhub-operator-pull-secret is not defined in the namespace, copy the pull-secret from the openshift-config namespace into the open-cluster-management-observability namespace. Run the following command:

    DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`
    oc create secret generic multiclusterhub-operator-pull-secret \
        -n open-cluster-management-observability \
        --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
        --type=kubernetes.io/dockerconfigjson
  4. Create a secret for your object storage. Your secret must contain the credentials to your storage solution. For example, run the following command:

    oc create -f thanos-object-storage.yaml -n open-cluster-management-observability
    1. View the following examples of secrets for the supported object stores:

      1. For Amazon S3 or S3 compatible, your secret might resemble the following file:

        apiVersion: v1
        kind: Secret
        metadata:
          name: thanos-object-storage
        type: Opaque
        stringData:
          thanos.yaml: |
            type: s3
            config:
              bucket: YOUR_S3_BUCKET
              endpoint: YOUR_S3_ENDPOINT
              insecure: false
              access_key: YOUR_ACCESS_KEY
              secret_key: YOUR_SECRET_KEY
      2. For Google, your secret might resemble the following file:

        type: GCS
        config:
          bucket: YOUR_GCS_BUCKET
          service_account: YOUR_SERVICE_ACCOUNT
      3. For Azure your secret might resemble the following file:

        type: AZURE
        config:
          storage_account: YOUR_STORAGE_ACCT
          storage_account_key: YOUR_STORAGE_KEY
          container: YOUR_CONTAINER
          endpoint: YOUR_ENDPOINT
          max_retries: 0
  5. Create the MultiClusterObservability custom resource (mco CR) for your managed cluster by completing the following steps:

    1. Create the MultiClusterObservability custom resource YAML file named multiclusterobservability_cr.yaml.

      View the following default YAML file for observability:

      apiVersion: observability.open-cluster-management.io/v1beta1
      kind: MultiClusterObservability
      metadata:
        name: observability #Your customized name of MulticlusterObservability CR
      spec:
        availabilityConfig: High # Available values are High or Basic
        imagePullPolicy: Always
        imagePullSecret: multiclusterhub-operator-pull-secret
        observabilityAddonSpec: # The ObservabilityAddonSpec defines the global settings for all managed clusters which have observability add-on enabled
          enableMetrics: true # EnableMetrics indicates the observability addon push metrics to hub server
          interval: 60 # Interval for the observability addon push metrics to hub server
        retentionResolution1h: 30d # How long to retain samples of 1 hour in bucket
        retentionResolution5m: 14d
        retentionResolutionRaw: 5d
        storageConfigObject: # Specifies the storage to be used by Observability
          metricObjectStorage:
            name: thanos-object-storage
            key: thanos.yaml
          statefulSetSize: 10Gi # The amount of storage applied to the Observability StatefulSets, i.e. Amazon S3 store, Rule, compact and receiver.
          statefulSetStorageClass: gp2

      You might want to modify the value for the retentionResolution parameter. For more information, see Thanos Downsampling resolution and retention. Depending on the number of managed clusters, you might want to update statefulSetSize, see Observability API for more information.

    2. To deploy on infrastructure machine sets, you must set a label for your set by updating the nodeSelector in the MultiClusterObservability YAML. Your YAML might resemble the following content:

      nodeSelector:
          node-role.kubernetes.io/infra:

      For more information, see Creating infrastructure machine sets.

    3. Apply the observability YAML to your cluster by running the following command:

      oc apply -f multiclusterobservability_cr.yaml

      All the pods in open-cluster-management-observability namespace for Thanos, Grafana and AlertManager are created. All the managed clusters connected to the Red Hat Advanced Cluster Management hub cluster are enabled to send metrics back to the Red Hat Advanced Cluster Management Observability service.

  6. To validate that the observability service is enabled, launch the Grafana dashboards to make sure the data is populated. Complete the following steps:

    1. Log in to the Red Hat Advanced Cluster Management console.
    2. From the navigation menu, select Observe environments > Overview.
    3. Click the Grafana link that is near the console header to view the metrics from your managed clusters.

      Note: If you want to exclude specific managed clusters from collecting the observability data, add the following cluster label to your clusters: vendor: OpenShift.