Chapter 4. Remote health monitoring with connected clusters

Procedure

1. Find the URL for the Prometheus service that runs in the OpenShift Container Platform cluster:

   $ oc get route prometheus-k8s -n openshift-monitoring -o jsonpath="{.spec.host}"

2. Navigate to the URL.

3. Enter this query in the Expression input box and press Execute:

   {__name__=~"cluster:usage:.*|count:up0|count:up1|cluster_version|cluster_version_available_updates|cluster_operator_up|cluster_operator_conditions|cluster_version_payload|cluster_installer|cluster_infrastructure_provider|cluster_feature_set|instance:etcd_object_counts:sum|ALERTS|code:apiserver_request_total:rate:sum|cluster:capacity_cpu_cores:sum|cluster:capacity_memory_bytes:sum|cluster:cpu_usage_cores:sum|cluster:memory_usage_bytes:sum|openshift:cpu_usage_cores:sum|openshift:memory_usage_bytes:sum|workload:cpu_usage_cores:sum|workload:memory_usage_bytes:sum|cluster:virt_platform_nodes:sum|cluster:node_instance_type_count:sum|cnv:vmi_status_running:count|cluster:vmi_request_cpu_cores:sum|node_role_os_version_machine:cpu_capacity_cores:sum|node_role_os_version_machine:cpu_capacity_sockets:sum|subscription_sync_total|olm_resolution_duration_seconds|csv_succeeded|csv_abnormal|cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum|cluster:kubelet_volume_stats_used_bytes:provisioner:sum|ceph_cluster_total_bytes|ceph_cluster_total_used_raw_bytes|ceph_health_status|job:ceph_osd_metadata:count|job:kube_pv:count|job:ceph_pools_iops:total|job:ceph_pools_iops_bytes:total|job:ceph_versions_running:count|job:noobaa_total_unhealthy_buckets:sum|job:noobaa_bucket_count:sum|job:noobaa_total_object_count:sum|noobaa_accounts_num|noobaa_total_usage|console_url|cluster:network_attachment_definition_instances:max|cluster:network_attachment_definition_enabled_instance_up:max|cluster:ingress_controller_aws_nlb_active:sum|insightsclient_request_send_total|cam_app_workload_migrations|cluster:apiserver_current_inflight_requests:sum:max_over_time:2m|cluster:alertmanager_integrations:max|cluster:telemetry_selected_series:count|openshift:prometheus_tsdb_head_series:sum|openshift:prometheus_tsdb_head_samples_appended_total:sum|monitoring:container_memory_working_set_bytes:sum|namespace_job:scrape_series_added:topk3_sum1h|namespace_job:scrape_samples_post_metric_relabeling:topk3|monitoring:haproxy_server_http_responses_total:sum|rhmi_status|cluster_legacy_scheduler_policy|cluster_master_schedulable|che_workspace_status|che_workspace_started_total|che_workspace_failure_total|che_workspace_start_time_seconds_sum|che_workspace_start_time_seconds_count|cco_credentials_mode|cluster:kube_persistentvolume_plugin_type_counts:sum|visual_web_terminal_sessions_total|acm_managed_cluster_info|cluster:vsphere_vcenter_info:sum|cluster:vsphere_esxi_version_total:sum|cluster:vsphere_node_hw_version_total:sum|openshift:build_by_strategy:sum|rhods_aggregate_availability|rhods_total_users|instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile|instance:etcd_mvcc_db_total_size_in_bytes:sum|instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile|instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum|instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile|jaeger_operator_instances_storage_types|jaeger_operator_instances_strategies|jaeger_operator_instances_agent_strategies|appsvcs:cores_by_product:sum|nto_custom_profiles:count|openshift_csi_share_configmap|openshift_csi_share_secret|openshift_csi_share_mount_failures_total|openshift_csi_share_mount_requests_total",alertstate=~"firing|"}

   This query replicates the request that Telemetry makes against a running OpenShift Container Platform cluster’s Prometheus service and returns the full set of time series captured by Telemetry.

Procedure

1. Find the name of the currently running pod for the Insights Operator:

   $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

Procedure

1. Download the global cluster pull secret to your local file system.

   $ oc extract secret/pull-secret -n openshift-config --to=.

2. In a text editor, edit the .dockerconfigjson file that was downloaded.

3. Remove the cloud.openshift.com JSON entry, for example:

   "cloud.openshift.com":{"auth":"<hash>","email":"<email_address>"}

4. Save the file.

Procedure

1. Go to the Register disconnected cluster web page on the Red Hat Hybrid Cloud Console.
2. Optional: To access the Register disconnected cluster web page from the home page of the Red Hat Hybrid Cloud Console, go to the Clusters navigation menu item and then select the Register cluster button.
3. Enter your cluster’s details in the provided fields on the Register disconnected cluster page.
4. From the Subscription settings section of the page, select the subcription settings that apply to your Red Hat subscription offering.
5. To register your disconnected cluster, select the Register cluster button.

Procedure

1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

   1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> 1

      1

      Provide the path to the pull secret file.

   2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ 1
      --auth-basic="<username>:<password>" \ 2
      --to=<pull_secret_location> 3

      1

      Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>".

      2

      Provide the credentials of the new registry.

      3

      Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

2. Enter the following command to update the global pull secret for your cluster:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> 1

   1

   Provide the path to the new pull secret file.

   This update is rolled out to all nodes, which can take some time depending on the size of your cluster.

   Note

   As of OpenShift Container Platform 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot.

Procedure

1. Navigate to https://console.redhat.com/openshift/downloads.

2. From Tokens → Pull Secret, click Download.

   The file pull-secret.txt containing your cloud.openshift.com access token in JSON format downloads:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "<your_token>",
         "email": "<email_address>"
       }
   }

3. Download the global cluster pull secret to your local file system.

   $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' > pull-secret

4. Make a backup copy of your pull secret.

   $ cp pull-secret pull-secret-backup

5. Open the pull-secret file in a text editor.
6. Append the cloud.openshift.com JSON entry from pull-secret.txt into auths.
7. Save the file.

8. Update the secret in your cluster.

   oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=pull-secret

Verification

1. Navigate to the OpenShift Container Platform Web Console Overview page.
2. Insights in the Status tile reports the number of issues found.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

   Depending on the result, Insights Advisor displays one of the following:

   • No matching recommendations found, if Insights did not identify any issues.
   • A list of issues Insights has detected, grouped by risk (low, moderate, important, and critical).
   • No clusters yet, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.

2. If any issues are displayed, click the > icon in front of the entry for more details.

   Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Click the X icons next to the Clusters Impacted and Status filters.

   You can now browse through all of the potential recommendations for your cluster.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.
2. Click the name of the recommendation to disable. You are directed to the single recommendation page.

3. To disable the recommendation for a single cluster:

   1. Click the Options menu for that cluster, and then click Disable recommendation for cluster.
   2. Enter a justification note and click Save.

4. To disable the recommendation for all of your clusters:

   1. Click Actions → Disable recommendation.
   2. Enter a justification note and click Save.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.
2. Filter the recommendations by Status → Disabled.
3. Locate the recommendation to enable.
4. Click the Options menu , and then click Enable recommendation.

Procedure

1. Navigate to Home → Overview in the OpenShift Container Platform web console.

2. Click Insights on the Status card.

   The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Insights Advisor to display more details.

Procedure

1. Find the name of the running pod for the Insights Operator:

   $ oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/<insights_operator_pod_name>:/var/lib/insights-operator ./insights-data 1

   1

   Replace <insights_operator_pod_name> with the pod name output from the preceding command.

Procedure

1. From your archive, open /insights-operator/gathers.json.

   The file contains a list of Insights Operator gather operations:

       {
         "name": "clusterconfig/authentication",
         "duration_in_ms": 730, 1
         "records_count": 1,
         "errors": null,
         "panic": null
       }

   1

   duration_in_ms is the amount of time in milliseconds for each gather operation.

2. Inspect each gather operation for abnormalities.

Procedure

1. Create a file named gather-job.yaml using this template:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: insights-operator-job
     annotations:
       config.openshift.io/inject-proxy: insights-operator
   spec:
     backoffLimit: 6
     ttlSecondsAfterFinished: 600
     template:
       spec:
         restartPolicy: OnFailure
         serviceAccountName: operator
         nodeSelector:
           beta.kubernetes.io/os: linux
           node-role.kubernetes.io/master: ""
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/master
           operator: Exists
         - effect: NoExecute
           key: node.kubernetes.io/unreachable
           operator: Exists
           tolerationSeconds: 900
         - effect: NoExecute
           key: node.kubernetes.io/not-ready
           operator: Exists
           tolerationSeconds: 900
         volumes:
         - name: snapshots
           emptyDir: {}
         - name: service-ca-bundle
           configMap:
             name: service-ca-bundle
             optional: true
         initContainers:
         - name: insights-operator
           image: quay.io/openshift/origin-insights-operator:latest
           terminationMessagePolicy: FallbackToLogsOnError
           volumeMounts:
           - name: snapshots
             mountPath: /var/lib/insights-operator
           - name: service-ca-bundle
             mountPath: /var/run/configmaps/service-ca-bundle
             readOnly: true
           ports:
           - containerPort: 8443
             name: https
           resources:
             requests:
               cpu: 10m
               memory: 70Mi
           args:
           - gather
           - -v=4
           - --config=/etc/insights-operator/server.yaml
         containers:
           - name: sleepy
             image: quay.io/openshift/origin-base:latest
             args:
               - /bin/sh
               - -c
               - sleep 10m
             volumeMounts: [{name: snapshots, mountPath: /var/lib/insights-operator}]

2. Copy your insights-operator image version:

   $ oc get -n openshift-insights deployment insights-operator -o yaml

3. Paste your image version in gather-job.yaml:

   initContainers:
         - name: insights-operator
           image: <your_insights_operator_image_version>
           terminationMessagePolicy: FallbackToLogsOnError
           volumeMounts:

4. Create the gather job:

   $ oc apply -n openshift-insights -f gather-job.yaml

5. Find the name of the job pod:

   $ oc describe -n openshift-insights job/insights-operator-job

   Example output

   Events:
     Type    Reason            Age    From            Message
     ----    ------            ----   ----            -------
     Normal  SuccessfulCreate  7m18s  job-controller  Created pod: insights-operator-job-<your_job>

   where insights-operator-job-<your_job> is the name of the pod.

6. Verify that the operation has finished:

   $ oc logs -n openshift-insights insights-operator-job-<your_job> insights-operator

   Example output

   I0407 11:55:38.192084       1 diskrecorder.go:34] Wrote 108 records to disk in 33ms

7. Save the created archive:

   $ oc cp openshift-insights/insights-operator-job-<your_job>:/var/lib/insights-operator ./insights-data

8. Clean up the job:

   $ oc delete -n openshift-insights job insights-operator-job

Procedure

1. Download the dockerconfig.json file:

   $ oc extract secret/pull-secret -n openshift-config --to=.

2. Copy your "cloud.openshift.com" "auth" token from the dockerconfig.json file:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "<your_token>",
         "email": "asd@redhat.com"
       }
   }

3. Upload the archive to console.redhat.com:

   $ curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/<cluster_id>" -H "Authorization: Bearer <your_token>" -F "upload=@<path_to_archive>; type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload

   where <cluster_id> is your cluster ID, <your_token> is the token from your pull secret, and <path_to_archive> is the path to the Insights Operator archive.

   If the operation is successful, the command returns a "request_id" and "account_number":

   Example output

   * Connection #0 to host console.redhat.com left intact
   {"request_id":"393a7cf1093e434ea8dd4ab3eb28884c","upload":{"account_number":"6274079"}}%

Verification steps

1. Log in to https://console.redhat.com/openshift.
2. Click the Clusters menu in the left pane.
3. To display the details of the cluster, click the cluster name.

4. Open the Insights Advisor tab of the cluster.

   If the upload was successful, the tab displays one of the following:

   • Your cluster passed all recommendations, if Insights Advisor did not identify any issues.
   • A list of issues that Insights Advisor has detected, prioritized by risk (low, moderate, important, and critical).

Procedure

1. Navigate to Workloads → Secrets.
2. Select the openshift-config project.
3. Search for the support secret using the Search by name field. If it does not exist, click Create → Key/value secret to create it.
4. Click the Options menu , and then click Edit Secret.
5. Click Add Key/Value.
6. Create a key named enableGlobalObfuscation with a value of true, and click Save.
7. Navigate to Workloads → Pods
8. Select the openshift-insights project.
9. Find the insights-operator pod.
10. To restart the insights-operator pod, click the Options menu , and then click Delete Pod.

Verification

1. Navigate to Workloads → Secrets.
2. Select the openshift-insights project.
3. Search for the obfuscation-translation-table secret using the Search by name field.

Procedure

1. Navigate to Workloads → Secrets.
2. Select the openshift-config project.
3. Search for the support secret using the Search by name field. If it does not exist, click Create → Key/value secret to create it.
4. Click the Options menu , and then click Edit Secret.
5. Click Add Key/Value.

6. Create a key named scaInterval with a value of 1h, and click Save.

   Note

   The interval 1h can also be entered as 60m for 60 minutes.

Procedure

1. Navigate to Workloads → Secrets.
2. Select the openshift-config project.
3. Search for the support secret using the Search by name field. If it does not exist, click Create → Key/value secret to create it.
4. Click the Options menu , and then click Edit Secret.
5. Click Add Key/Value.

6. Create a key named scaPullDisabled with a value of true, and click Save.

   The simple content access entitlement import is now disabled.

   Note

   To enable the simple content access import again, edit the support secret and delete the scaPullDisabled key.