KubeVirtComponentExceedsRequestedMemory Warning' alert generated for virt-controller pod

Solution In Progress - Updated -

Environment

  • OpenShift Container Platform (RHOCP)- 4.13.7

Issue

  • The following alerts are generated.
'KubeVirtComponentExceedsRequestedMemory Warning' : 'Container virt-controller in pod virt-controller-XXXXXX memory usage exceeds the memory requested'

Resolution

  • Monitor the pod with the following command and if neccessary increase the memory size for the virt-controller
$ oc adm top pods -n openshift-cnv | egrep 'NAME|virt-con'
  • You can run the following to increase the memory to 400 as an example:
$ oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
  kubevirt.kubevirt.io/jsonpatch='[
    {
      "op": "add",
      "path": "/spec/customizeComponents/patches",
      "value": [{
          "patch": "[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/resources/requests\",\"value\":{\"cpu\": \"10m\",\"memory\": \"400Mi\"}}]",
          "resourceName": "virt-controller",
          "resourceType": "Deployment",
          "type": "json"
      }]
    }
  ]'

  • If you wait a few minutes your virt-controller pod should restart with a higher limit.

  • You will get a new alert after updating the memory using jsonpatch.
    This alert is expected per JIRA CNV-28429.

KubevirtHyperconvergedClusterOperatorUSModification
Severity:   Info
Description:    unsafe modification for the kubevirt.kubevirt.io/jsonpatch annotation in the HyperConverged resource.
  • You can remove the annotation if you wish to revert with:
$ oc annotate -n openshift-cnv hyperconverged kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch-
  • Or simply 'oc edit hco -n openshift-cnv' and remove the entire kubevirt.kubevirt.io/jsonpatch section

Root Cause

  • The container is using more memory than what is defined in the containers resource requests

Diagnostic Steps

  • virt-controller deployment set to 275Mi
└╼ $ oc get deployment virt-controller -o yaml -n openshift-cnv

        resources:
          requests:
            cpu: 10m
            memory: 275Mi
  • virt-controller-XXXX pod is set at 275Mi
└╼ $ oc get pods virt-controller-XXXXXX -o yaml -n openshift-cnv 

    resources:
      requests:
        cpu: 10m
        memory: 275Mi  <<<=======
  • Component
  • cnv

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments