Prometheus alert "many-to-many matching not allowed matching labels must be unique on one side" in OCP 4

Solution Verified - Updated -

Issue

The PrometheusRuleFailures alert is firing after installed 3scale. I see a lot of messages in the Prometheus logs like this:

level=warn ts=2020-11-17T14:02:36.835Z caller=manager.go:586 component="rule manager" group=kubernetes.rules msg="Evaluating rule failed" rule="record: cluster:kubelet_volume_stats_used_bytes:provisioner:sum\nexpr: sum by(provisioner) (kubelet_volume_stats_used_bytes * on(namespace, persistentvolumeclaim) group_right() (kube_persistentvolumeclaim_info * on(storageclass) group_left(provisioner) kube_storageclass_info))\n" err="found duplicate series for the match group {namespace=\"3scale-app\", persistentvolumeclaim=\"system-storage\"} on the left hand-side of the operation: [{__name__=\"kubelet_volume_stats_used_bytes\", endpoint=\"https-metrics\", instance=\"10.12.13.14:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"3scale-app\", node=\"prodocp1-my-openshift-node\", persistentvolumeclaim=\"system-storage\", service=\"kubelet\"}, {__name__=\"kubelet_volume_stats_used_bytes\", endpoint=\"https-metrics\", instance=\"10.12.13.14:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"3scale-app\", node=\"prodocp1-my-openshift-node\", persistentvolumeclaim=\"system-storage\", service=\"kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"

### PVs

NAME                    STATUS   VOLUME                           CAPACITY   ACCESS MODES   STORAGECLASS      AGE
system-redis-storage    Bound    pvc-abc-def-ghi   1Gi        RWO            managed-premium   3d21h
system-storage          Bound    3scale-azfile-pv         1Gi        RWX            azure-file-sc     3d21h

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.6
    • 4.7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content