Prometheus serviceaccount missing permissions to monitor services in user-defined namespaces

Resolution

There are 3 situations associated with this problem.

Situation 1:
The label openshift.io/cluster-monitoring: "true" is applied on user-defined namespace because of which prometheus-k8s pods from openshift-monitoring tries to scrape metrics from user-defined namespace.
This label openshift.io/cluster-monitoring: "true" should not be applied on user-defined namespaces as per support considerations

To fix this, remove the label from user-defined namespace and enable user-workload monitoring using this documentation. To remove the label, refer below command:

$ oc label namespace <name-of-namespace> openshift.io/cluster-monitoring-

Situation 2:
Additional user-defined ServiceMonitors are created in the openshift-* and kube-* projects.
Additional user-defined ServiceMonitors should not be created in the openshift-* and kube-* projects as per support considerations

To fix this, remove the additional user-defined ServiceMonitors from the openshift-* and kube-* projects:

$ oc -n openshift-monitoring delete servicemonitor <name-of-servicemonitor>

Situation 3:
The namespace for which the log is streaming is actually a namespace hosting core OpenShift Container Platform components or Red Hat certified component.

If core OpenShift component / Red Hat certified component's namespace is missing the expected roles or rolebindings, please open a support case with Red Hat. These resources are expected to be present by default and their absence could be due to a bug in the relevant component.

Workaround

• Make sure a role exists granting the correct permissions in the namespace:

  $ cat role.yaml
  --
  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: prometheus-k8s
    namespace: <name-of-namespace>
  rules:
  - apiGroups:
    - ""
    resources:
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
  

• Make sure the rolebinding exists binding the previously created role to the serviceaccount system:serviceaccount:openshift-monitoring:prometheus-k8s in the namespace:

  $ cat rolebinding.yaml
  --
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: prometheus-k8s
    namespace: <name-of-namespace> 
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: prometheus-k8s
  subjects:
  - kind: ServiceAccount
    name: prometheus-k8s
    namespace: openshift-monitoring
  

• Run the following commands and check the output to validate the role and rolebinding

  $ oc get role,rolebinding -n <namespace> | egrep "NAME|prometheus"
  NAME                                            CREATED AT
  role.rbac.authorization.k8s.io/prometheus-k8s   2023-07-31T20:44:45Z
  NAME                                                          ROLE                               AGE
  rolebinding.rbac.authorization.k8s.io/prometheus-k8s          Role/prometheus-k8s                15m
  
  $ oc get svc -n dummyapp --as=system:serviceaccount:openshift-monitoring:prometheus-k8s
  NAME    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
  httpd   ClusterIP   172.30.184.120   <none>        8080/TCP,8443/TCP   20m
  nginx   ClusterIP   None             <none>        80/TCP              18h