KubeVirtComponentExceedsRequestedMemory Prometheus Rule is Failing to Evaluate in RHOCP 4

Solution Verified - Updated -

Issue

  • After installing Red Hat OpenShift Virtualization operator in both Prometheus pods are visible errors where it's indicating that one Prometheus rule is failing to be evaluated with error:

    level=warn ts=2022-02-24T08:29:05.200Z caller=manager.go:603 component="rule manager" group=kubevirt.rules msg="Evaluating rule failed" rule="alert: KubeVirtComponentExceedsRequestedMemory\nexpr: ((kube_pod_container_resource_requests{container=~\"virt-controller|virt-api|virt-handler|virt-operator\",namespace=\"openshift-cnv\",resource=\"memory\"})\n  - on(pod) group_left(node) container_memory_usage_bytes{namespace=\"openshift-cnv\"})\n  < 0\nfor: 5m\nlabels:\n  severity: warning\nannotations:\n  description: Container {{ $labels.container }} in pod {{ $labels.pod }} memory usage\n    exceeds the memory requested\n  summary: The container is using more memory than what is defined in the containers\n    resource requests\n" err="found duplicate series for the match group {pod=\"bridge-marker-btm6q\"} on the right hand-side of the operation: [{__name__=\"container_memory_usage_bytes\", container=\"bridge-marker\", endpoint=\"https-metrics\", id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab00f667_6ba0_4dd2_9e38_7e2cb0d2199c.slice/crio-5f438921d6bfa58f36db7d6e263a8d4e6b5c103b35e91f0c45a9f9c4f66af11d.scope\", image=\"registry.redhat.io/container-native-virtualization/bridge-marker@sha256:ac602257a8d36cf11713f507dc931254b49f5892af460808cc57fec5e94ecebc\", instance=\"172.31.8.114:10250\", job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", name=\"k8s_bridge-marker_bridge-marker-btm6q_openshift-cnv_ab00f667-6ba0-4dd2-9e38-7e2cb0d2199c_1\", namespace=\"openshift-cnv\", node=\"node.example.com\", pod=\"bridge-marker-btm6q\", service=\"kubelet\"}, {__name__=\"container_memory_usage_bytes\", container=\"POD\", endpoint=\"https-metrics\", id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab00f667_6ba0_4dd2_9e38_7e2cb0d2199c.slice/crio-e43d8cb67a4a777e1e30f476c0b11b3caa3c9d50c3ae1c795052c008a0b3b66d.scope\", instance=\"172.31.8.114:10250\", job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", name=\"k8s_POD_bridge-marker-btm6q_openshift-cnv_ab00f667-6ba0-4dd2-9e38-7e2cb0d2199c_0\", namespace=\"openshift-cnv\", node=\"node.example.com\", pod=\"bridge-marker-btm6q\", service=\"kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
    

Environment

  • RedHat OpenShift Container Platform (RHOCP)
    • 4.9
    • 4.10
    • 4.11
  • Red Hat OpenShift Virtualization
    • 4.9
    • 4.10
    • 4.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content