Prometheus requires a view role to scrape metrics from applications in customer defined namespace

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
  • Azure Red Hat OpenShift
  • OpenShift Dedicated (OSD)

Issue

  • Default OpenShift monitoring based on Prometheus, that is shipped with the monitoring stack of OpenShift Container Platformis not receiving metrics. The metrics are not present in the Prometheus dashboard.
  • How to scrape metrics in Prometheus provided by OpenShift Monitoring for a user defined namespace?
  • Some applications or services require a Service Account with view role to be able to scrape metrics
    and send it to Prometheus

Resolution

  • The command below allows Service Account prometheus-k8s in the openshift-monitoring namespace to list resources in the user defined namespace:
$ oc policy add-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s -n user_defined

Root Cause

  • The Service Account used by Prometheus requires list permissions in the configured namespace to be able to scrape metrics.

Diagnostic Steps

  • Check logs of Prometheus pods in openshift-monitoring namespace. Cannot list resource and endpoints is forbidden access errors are detected.
$ oc get pods -n openshift-monitoring | grep -i 'prometheus-k8s'

prometheus-k8s-0                                         6/6     Running     0             10h
prometheus-k8s-1                                         6/6     Running     0             10h

$ oc logs prometheus-k8s-0 -n openshift-monitoring

ts=2023-06-22T02:22:03.282Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:541: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"xxxxxx\""
ts=2023-06-22T02:22:08.925Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"xxxxxx\""
ts=2023-06-22T02:22:08.926Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"xxxxxxx\""

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments