Why draining a node is stuck with error 'failed calling webhook "migration-create-validator.kubevirt.io"'?

Solution In Progress - Updated -

Issue

  • The OpenShift node was having many virtual machines running (50+ VMs) and I am unable to drain the node. The creation of VirtualMachineInstanceMigration(VMIM) is failing with following error:
8s (x4 over 39m)         Warning   FailedCreate             VirtualMachineInstance/test-vm-29                           Error creating a Migration: Internal error occurred: failed calling webhook "migration-create-validator.kubevirt.io": failed to call webhook: Post "https://virt-api.openshift-cnv.svc:443/migration-validate-create?timeout=10s": context deadline exceeded
  • The virt-api pod has the following logs, with the "wait time" continuously increasing:
I0121 20:17:22.042821       1 request.go:697] Waited for 3m33.071923903s due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/default/pods/virt-launcher-testvm-146-jbw27

Environment

  • OpenShift Virtualization 4.16, 4.17.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content