Create a namespace for the observability service with the following command:

oc create namespace open-cluster-management-observability

Generate your pull-secret. If Red Hat Advanced Cluster Management is installed in the open-cluster-management namespace, run the following command:

DOCKER_CONFIG_JSON=`oc extract secret/multiclusterhub-operator-pull-secret -n open-cluster-management --to=-`

If the multiclusterhub-operator-pull-secret is not defined in the namespace, copy the pull-secret from the openshift-config namespace into the open-cluster-management-observability namespace. Run the following command:

DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`

Then, create the pull-secret in the open-cluster-management-observability namespace, run the following command:

oc create secret generic multiclusterhub-operator-pull-secret \
    -n open-cluster-management-observability \
    --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
    --type=kubernetes.io/dockerconfigjson

Create a secret for your object storage for your cloud provider. Your secret must contain the credentials to your storage solution. For example, run the following command:

oc create -f thanos-object-storage.yaml -n open-cluster-management-observability

View the following examples of secrets for the supported object stores:

You can retrieve the S3 access key and secret key for your cloud providers with the following commands:

YOUR_CLOUD_PROVIDER_ACCESS_KEY=$(oc -n open-cluster-management-observability get secret <object-storage-secret> -o jsonpath="{.data.thanos\.yaml}" | base64 --decode | grep access_key | awk '{print $2}')

echo $ACCESS_KEY

YOUR_CLOUD_PROVIDER_SECRET_KEY=$(oc -n open-cluster-management-observability get secret <object-storage-secret> -o jsonpath="{.data.thanos\.yaml}" | base64 --decode | grep secret_key | awk '{print $2}')

echo $SECRET_KEY

Create your image pull-secret:

Create your object storage secret named, thanos-object-storage in the open-cluster-management-observability project. In this example, create an Amazon S3 object storage secret for the observability service:

Create the MultiClusterObservability CR:

Create a ConfigMap named thanos-ruler-custom-rules in the open-cluster-management-observability namespace. The key must be named, custom_rules.yaml, as shown in the following example. You can create multiple rules in the configuration:

If you want to verify that the alert rules is functioning appropriately, complete the following steps:

Extract the data from the alertmanager-config secret. Run the following command:

oc -n open-cluster-management-observability get secret alertmanager-config --template='{{ index .data "alertmanager.yaml" }}' |base64 -d > alertmanager.yaml

Edit and save the alertmanager.yaml file configuration by running the following command:

oc -n open-cluster-management-observability create secret generic alertmanager-config --from-file=alertmanager.yaml --dry-run -o=yaml |  oc -n open-cluster-management-observability replace secret --filename=-

Your updated secret might resemble the following content:

global
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'
templates:
- '/etc/alertmanager/template/*.tmpl'
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: team-X-mails
  routes:
  - match_re:
      service: ^(foo1|foo2|baz)$
    receiver: team-X-mails

Verify that mco observability is enabled. Check for the following message in the status.conditions.message reads: Observability components are deployed and running. Run the following command:

oc get mco observability -o yaml

Create a file named observability-metrics-custom-allowlist.yaml with the following content. Add the name and recording rule of the custom metric to the metrics_list.yaml parameter. For example, collect node_memory_MemTotal_bytes and apiserver_request_duration_seconds:histogram_quantile_90 from your managed cluster. Your YAML for the ConfigMap might resemble the following content:

kind: ConfigMap
apiVersion: v1
metadata:
  name: observability-metrics-custom-allowlist
data:
  metrics_list.yaml: |
    names:
      - node_memory_MemTotal_bytes
    rules:
    - record: apiserver_request_duration_seconds:histogram_quantile_90
      expr: histogram_quantile(0.90,sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",
        verb!=\"WATCH\"}[5m])) by (verb,le))

Create the observability-metrics-custom-allowlist ConfigMap in the open-cluster-management-observability namespace by running the following command:

oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml

Verify that mco observability is enabled. Check for the following message in the status.conditions.message reads: Observability components are deployed and running. Run the following command:

oc get mco observability -o yaml

In the observability-metrics-custom-allowlist.yaml file, add the name of the default metric to the metrics_list.yaml parameter with a hyphen - at the start of the metric name. For example, add -cluster_infrastructure_provider to the metric list. Your YAML for the ConfigMap might resemble the following content:

kind: ConfigMap
apiVersion: v1
metadata:
  name: observability-metrics-custom-allowlist
data:
  metrics_list.yaml: |
    names:
      - node_memory_MemTotal_bytes
      - -cluster_infrastructure_provider

Create the observability-metrics-custom-allowlist ConfigMap in the open-cluster-management-observability namespace by running the following command:

oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml

Edit the mco observability. Run the following command:

oc edit mco observability -o yaml

Add the advanced configuration into the mco observability YAML. Your YAML file might resemble the following contents:

spec:
  advanced:
    retentionConfig:
      blockDuration: 2h
      deleteDelay: 48h
      retentionInLocal: 24h
      retentionResolutionRaw: 30d
      retentionResolution5m: 180d
      retentionResolution1h: 0d
    receive:
      resources:
        limits:
          memory: 4096Gi
      replicas: 3

Edit the YAML file in the YAML tab. Your updated YAML might resemble the following content:

spec:
   advanced:
      receive:
         replicas: 6

This means that there are six receivers in the environment.

From the OpenShift Container Platform menu navigation, select Secrets. Select the alertmanager-config secret to view the YAML file.

Note: If you make changes to the alertmanager-config secret, the evaluation interval is about one minute.

View the following example of the alertmanager-config YAML file:

global:
  slack_api_url: '<slack_webhook_url>'

route:
  receiver: 'slack-notifications'
  group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#alerts'
    text: 'https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}'

If you want to configure a proxy for alert forwarding, add the following global entry to the alertmanager-config YAML file:

global:
  slack_api_url: '<slack_webhook_url>'
  http_config:
    proxy_url: http://****

To query metrics that come from a single node cluster, add the following label in the query expression: {clusterType="SNO"}. For example, to cluster_infrastructure_provider from a single node cluster, use the following query expression: cluster_infrastructure_provider{clusterType="SNO"}

Note: Do not set the ObservabilitySpec.resources.CPU.limits parameter if observability is enabled on single node managed clusters. When you set the CPU limits, it causes the observability pod to be counted against the capacity for your managed cluster. See Management Workload Partioning for more information.

Run the setup-grafana-dev.sh to setup your Grafana instance. When you run the script the following resources are created: secret/grafana-dev-config, deployment.apps/grafana-dev, service/grafana-dev, ingress.extensions/grafana-dev, persistentvolumeclaim/grafana-dev:

./setup-grafana-dev.sh --deploy
secret/grafana-dev-config created
deployment.apps/grafana-dev created
service/grafana-dev created
ingress.extensions/grafana-dev created
persistentvolumeclaim/grafana-dev created

Switch the user role to Grafana administrator with the switch-to-grafana-admin.sh script.

After you press enter, a table list of insight policy reports appear in the Policyreport section.

Note: The PolicyReport name matches the name of the cluster.

If the search service is disabled and you want to search for a insight, run the following command from your hub cluster:

oc get policyreport --all-namespaces

If there are more vulnerabilities that you want to view, click Back to view the list of potential issues.

Note: After the issue is resolved, the Red Hat Insights are received by Red Hat Advanced Cluster Management every 30 minutues and Red Hat Insights is updated every two hours.