Prometheus alert "many-to-many matching not allowed matching labels must be unique on one side" in OCP 4
Issue
The PrometheusRuleFailures alert is firing after installed 3scale. I see a lot of messages in the Prometheus logs like this:
level=warn ts=2020-11-17T14:02:36.835Z caller=manager.go:586 component="rule manager" group=kubernetes.rules msg="Evaluating rule failed" rule="record: cluster:kubelet_volume_stats_used_bytes:provisioner:sum\nexpr: sum by(provisioner) (kubelet_volume_stats_used_bytes * on(namespace, persistentvolumeclaim) group_right() (kube_persistentvolumeclaim_info * on(storageclass) group_left(provisioner) kube_storageclass_info))\n" err="found duplicate series for the match group {namespace=\"3scale-app\", persistentvolumeclaim=\"system-storage\"} on the left hand-side of the operation: [{__name__=\"kubelet_volume_stats_used_bytes\", endpoint=\"https-metrics\", instance=\"10.12.13.14:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"3scale-app\", node=\"prodocp1-my-openshift-node\", persistentvolumeclaim=\"system-storage\", service=\"kubelet\"}, {__name__=\"kubelet_volume_stats_used_bytes\", endpoint=\"https-metrics\", instance=\"10.12.13.14:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"3scale-app\", node=\"prodocp1-my-openshift-node\", persistentvolumeclaim=\"system-storage\", service=\"kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
### PVs
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
system-redis-storage Bound pvc-abc-def-ghi 1Gi RWO managed-premium 3d21h
system-storage Bound 3scale-azfile-pv 1Gi RWX azure-file-sc 3d21h
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4.6
- 4.7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.