KubeVirtComponentExceedsRequestedMemory Prometheus Rule is Failing to Evaluate in RHOCP 4
Issue
-
After installing Red Hat OpenShift Virtualization operator in both Prometheus pods are visible errors where it's indicating that one Prometheus rule is failing to be evaluated with error:
level=warn ts=2022-02-24T08:29:05.200Z caller=manager.go:603 component="rule manager" group=kubevirt.rules msg="Evaluating rule failed" rule="alert: KubeVirtComponentExceedsRequestedMemory\nexpr: ((kube_pod_container_resource_requests{container=~\"virt-controller|virt-api|virt-handler|virt-operator\",namespace=\"openshift-cnv\",resource=\"memory\"})\n - on(pod) group_left(node) container_memory_usage_bytes{namespace=\"openshift-cnv\"})\n < 0\nfor: 5m\nlabels:\n severity: warning\nannotations:\n description: Container {{ $labels.container }} in pod {{ $labels.pod }} memory usage\n exceeds the memory requested\n summary: The container is using more memory than what is defined in the containers\n resource requests\n" err="found duplicate series for the match group {pod=\"bridge-marker-btm6q\"} on the right hand-side of the operation: [{__name__=\"container_memory_usage_bytes\", container=\"bridge-marker\", endpoint=\"https-metrics\", id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab00f667_6ba0_4dd2_9e38_7e2cb0d2199c.slice/crio-5f438921d6bfa58f36db7d6e263a8d4e6b5c103b35e91f0c45a9f9c4f66af11d.scope\", image=\"registry.redhat.io/container-native-virtualization/bridge-marker@sha256:ac602257a8d36cf11713f507dc931254b49f5892af460808cc57fec5e94ecebc\", instance=\"172.31.8.114:10250\", job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", name=\"k8s_bridge-marker_bridge-marker-btm6q_openshift-cnv_ab00f667-6ba0-4dd2-9e38-7e2cb0d2199c_1\", namespace=\"openshift-cnv\", node=\"node.example.com\", pod=\"bridge-marker-btm6q\", service=\"kubelet\"}, {__name__=\"container_memory_usage_bytes\", container=\"POD\", endpoint=\"https-metrics\", id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab00f667_6ba0_4dd2_9e38_7e2cb0d2199c.slice/crio-e43d8cb67a4a777e1e30f476c0b11b3caa3c9d50c3ae1c795052c008a0b3b66d.scope\", instance=\"172.31.8.114:10250\", job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", name=\"k8s_POD_bridge-marker-btm6q_openshift-cnv_ab00f667-6ba0-4dd2-9e38-7e2cb0d2199c_0\", namespace=\"openshift-cnv\", node=\"node.example.com\", pod=\"bridge-marker-btm6q\", service=\"kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
Environment
- RedHat OpenShift Container Platform (RHOCP)
- 4.9
- 4.10
- 4.11
- Red Hat OpenShift Virtualization
- 4.9
- 4.10
- 4.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.