Chapter 4. Remote health monitoring with connected clusters

Procedure

1. Log in to a cluster.

2. Run the following command, which queries a cluster’s Prometheus service and returns the full set of time series data captured by Telemetry:

   $ curl -G -k -H "Authorization: Bearer $(oc whoami -t)" \
   https://$(oc get route prometheus-k8s-federate -n \
   openshift-monitoring -o jsonpath="{.spec.host}")/federate \
   --data-urlencode 'match[]={__name__=~"cluster:usage:.*"}' \
   --data-urlencode 'match[]={__name__="count:up0"}' \
   --data-urlencode 'match[]={__name__="count:up1"}' \
   --data-urlencode 'match[]={__name__="cluster_version"}' \
   --data-urlencode 'match[]={__name__="cluster_version_available_updates"}' \
   --data-urlencode 'match[]={__name__="cluster_version_capability"}' \
   --data-urlencode 'match[]={__name__="cluster_operator_up"}' \
   --data-urlencode 'match[]={__name__="cluster_operator_conditions"}' \
   --data-urlencode 'match[]={__name__="cluster_version_payload"}' \
   --data-urlencode 'match[]={__name__="cluster_installer"}' \
   --data-urlencode 'match[]={__name__="cluster_infrastructure_provider"}' \
   --data-urlencode 'match[]={__name__="cluster_feature_set"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_object_counts:sum"}' \
   --data-urlencode 'match[]={__name__="ALERTS",alertstate="firing"}' \
   --data-urlencode 'match[]={__name__="code:apiserver_request_total:rate:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:capacity_cpu_cores:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:capacity_memory_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="workload:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="workload:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:virt_platform_nodes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:node_instance_type_count:sum"}' \
   --data-urlencode 'match[]={__name__="cnv:vmi_status_running:count"}' \
   --data-urlencode 'match[]={__name__="cluster:vmi_request_cpu_cores:sum"}' \
   --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_cores:sum"}' \
   --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_sockets:sum"}' \
   --data-urlencode 'match[]={__name__="subscription_sync_total"}' \
   --data-urlencode 'match[]={__name__="olm_resolution_duration_seconds"}' \
   --data-urlencode 'match[]={__name__="csv_succeeded"}' \
   --data-urlencode 'match[]={__name__="csv_abnormal"}' \
   --data-urlencode 'match[]={__name__="cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:kubelet_volume_stats_used_bytes:provisioner:sum"}' \
   --data-urlencode 'match[]={__name__="ceph_cluster_total_bytes"}' \
   --data-urlencode 'match[]={__name__="ceph_cluster_total_used_raw_bytes"}' \
   --data-urlencode 'match[]={__name__="ceph_health_status"}' \
   --data-urlencode 'match[]={__name__="odf_system_raw_capacity_total_bytes"}' \
   --data-urlencode 'match[]={__name__="odf_system_raw_capacity_used_bytes"}' \
   --data-urlencode 'match[]={__name__="odf_system_health_status"}' \
   --data-urlencode 'match[]={__name__="job:ceph_osd_metadata:count"}' \
   --data-urlencode 'match[]={__name__="job:kube_pv:count"}' \
   --data-urlencode 'match[]={__name__="job:odf_system_pvs:count"}' \
   --data-urlencode 'match[]={__name__="job:ceph_pools_iops:total"}' \
   --data-urlencode 'match[]={__name__="job:ceph_pools_iops_bytes:total"}' \
   --data-urlencode 'match[]={__name__="job:ceph_versions_running:count"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_total_unhealthy_buckets:sum"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_bucket_count:sum"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_total_object_count:sum"}' \
   --data-urlencode 'match[]={__name__="odf_system_bucket_count", system_type="OCS", system_vendor="Red Hat"}' \
   --data-urlencode 'match[]={__name__="odf_system_objects_total", system_type="OCS", system_vendor="Red Hat"}' \
   --data-urlencode 'match[]={__name__="noobaa_accounts_num"}' \
   --data-urlencode 'match[]={__name__="noobaa_total_usage"}' \
   --data-urlencode 'match[]={__name__="console_url"}' \
   --data-urlencode 'match[]={__name__="cluster:ovnkube_master_egress_routing_via_host:max"}' \
   --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_instances:max"}' \
   --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_enabled_instance_up:max"}' \
   --data-urlencode 'match[]={__name__="cluster:ingress_controller_aws_nlb_active:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:min"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:max"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:avg"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:median"}' \
   --data-urlencode 'match[]={__name__="cluster:openshift_route_info:tls_termination:sum"}' \
   --data-urlencode 'match[]={__name__="insightsclient_request_send_total"}' \
   --data-urlencode 'match[]={__name__="cam_app_workload_migrations"}' \
   --data-urlencode 'match[]={__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}' \
   --data-urlencode 'match[]={__name__="cluster:alertmanager_integrations:max"}' \
   --data-urlencode 'match[]={__name__="cluster:telemetry_selected_series:count"}' \
   --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_series:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_samples_appended_total:sum"}' \
   --data-urlencode 'match[]={__name__="monitoring:container_memory_working_set_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="namespace_job:scrape_series_added:topk3_sum1h"}' \
   --data-urlencode 'match[]={__name__="namespace_job:scrape_samples_post_metric_relabeling:topk3"}' \
   --data-urlencode 'match[]={__name__="monitoring:haproxy_server_http_responses_total:sum"}' \
   --data-urlencode 'match[]={__name__="rhmi_status"}' \
   --data-urlencode 'match[]={__name__="status:upgrading:version:rhoam_state:max"}' \
   --data-urlencode 'match[]={__name__="state:rhoam_critical_alerts:max"}' \
   --data-urlencode 'match[]={__name__="state:rhoam_warning_alerts:max"}' \
   --data-urlencode 'match[]={__name__="rhoam_7d_slo_percentile:max"}' \
   --data-urlencode 'match[]={__name__="rhoam_7d_slo_remaining_error_budget:max"}' \
   --data-urlencode 'match[]={__name__="cluster_legacy_scheduler_policy"}' \
   --data-urlencode 'match[]={__name__="cluster_master_schedulable"}' \
   --data-urlencode 'match[]={__name__="che_workspace_status"}' \
   --data-urlencode 'match[]={__name__="che_workspace_started_total"}' \
   --data-urlencode 'match[]={__name__="che_workspace_failure_total"}' \
   --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_sum"}' \
   --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_count"}' \
   --data-urlencode 'match[]={__name__="cco_credentials_mode"}' \
   --data-urlencode 'match[]={__name__="cluster:kube_persistentvolume_plugin_type_counts:sum"}' \
   --data-urlencode 'match[]={__name__="visual_web_terminal_sessions_total"}' \
   --data-urlencode 'match[]={__name__="acm_managed_cluster_info"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_vcenter_info:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_esxi_version_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_node_hw_version_total:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:build_by_strategy:sum"}' \
   --data-urlencode 'match[]={__name__="rhods_aggregate_availability"}' \
   --data-urlencode 'match[]={__name__="rhods_total_users"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="jaeger_operator_instances_storage_types"}' \
   --data-urlencode 'match[]={__name__="jaeger_operator_instances_strategies"}' \
   --data-urlencode 'match[]={__name__="jaeger_operator_instances_agent_strategies"}' \
   --data-urlencode 'match[]={__name__="appsvcs:cores_by_product:sum"}' \
   --data-urlencode 'match[]={__name__="nto_custom_profiles:count"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_configmap"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_secret"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_mount_failures_total"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_mount_requests_total"}' \
   --data-urlencode 'match[]={__name__="cluster:velero_backup_total:max"}' \
   --data-urlencode 'match[]={__name__="cluster:velero_restore_total:max"}' \
   --data-urlencode 'match[]={__name__="eo_es_storage_info"}' \
   --data-urlencode 'match[]={__name__="eo_es_redundancy_policy_info"}' \
   --data-urlencode 'match[]={__name__="eo_es_defined_delete_namespaces_total"}' \
   --data-urlencode 'match[]={__name__="eo_es_misconfigured_memory_resources_info"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_data_nodes_total:max"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_documents_created_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_documents_deleted_total:sum"}' \
   --data-urlencode 'match[]={__name__="pod:eo_es_shards_total:max"}' \
   --data-urlencode 'match[]={__name__="eo_es_cluster_management_state_info"}' \
   --data-urlencode 'match[]={__name__="imageregistry:imagestreamtags_count:sum"}' \
   --data-urlencode 'match[]={__name__="imageregistry:operations_count:sum"}' \
   --data-urlencode 'match[]={__name__="log_logging_info"}' \
   --data-urlencode 'match[]={__name__="log_collector_error_count_total"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_pipeline_info"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_input_info"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_output_info"}' \
   --data-urlencode 'match[]={__name__="cluster:log_collected_bytes_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:log_logged_bytes_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:kata_monitor_running_shim_count:sum"}' \
   --data-urlencode 'match[]={__name__="platform:hypershift_hostedclusters:max"}' \
   --data-urlencode 'match[]={__name__="platform:hypershift_nodepools:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_bucket_claims:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_buckets_claims:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_resources:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_resources:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_buckets:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_buckets:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_accounts:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_usage:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_system_health_status:max"}' \
   --data-urlencode 'match[]={__name__="ocs_advanced_feature_usage"}' \
   --data-urlencode 'match[]={__name__="os_image_url_override:sum"}'

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

   Depending on the result, Insights Advisor displays one of the following:

   • No matching recommendations found, if Insights did not identify any issues.
   • A list of issues Insights has detected, grouped by risk (low, moderate, important, and critical).
   • No clusters yet, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.

2. If any issues are displayed, click the > icon in front of the entry for more details.

   Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Click the X icons next to the Clusters Impacted and Status filters.

   You can now browse through all of the potential recommendations for your cluster.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.
2. Optional: Use the Clusters Impacted and Status filters as needed.

3. Disable an alert by using one of the following methods:

   • To disable an alert:

     1. Click the Options menu for that alert, and then click Disable recommendation.
     2. Enter a justification note and click Save.

   • To view the clusters affected by this alert before disabling the alert:

     1. Click the name of the recommendation to disable. You are directed to the single recommendation page.
     2. Review the list of clusters in the Affected clusters section.
     3. Click Actions → Disable recommendation to disable the alert for all of your clusters.
     4. Enter a justification note and click Save.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Filter the recommendations to display on the disabled recommendations:

   1. From the Status drop-down menu, select Status.
   2. From the Filter by status drop-down menu, select Disabled.
   3. Optional: Clear the Clusters impacted filter.
3. Locate the recommendation to enable.
4. Click the Options menu , and then click Enable recommendation.

Procedure

1. Navigate to Home → Overview in the OpenShift Dedicated web console.

2. Click Insights on the Status card.

   The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Insights Advisor to display more details.

Procedure

1. Go to Workloads → ConfigMaps and select Project: openshift-insights.
2. Click on the insights-config ConfigMap object to open it.
3. Click Actions and select Edit ConfigMap.
4. Click the YAML view radio button.

5. In the file, set the obfuscation attribute with the workload_names value.

   apiVersion: v1
   kind: ConfigMap
   # ...
   data:
     config.yaml: |
       dataReporting:
         obfuscation:
           - workload_names
   # ...

6. Click Save. The insights-config config-map details page opens.
7. Verify that the value of the config.yaml obfuscation attribute is set to - workload_names.