Chapter 14. Logging, events, and monitoring

Procedure

1. In the Administrator perspective in the OpenShift Container Platform web console, navigate to Virtualization → Overview.
2. Click the Top consumers tab.
3. Optional: You can filter the results by selecting a time period or by selecting the 5 or 10 top consumers.

Procedure

1. To set log verbosity for specific components, open the HyperConverged CR in your default text editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Set the log level for one or more components by editing the spec.logVerbosityConfig stanza. For example:

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
   spec:
     logVerbosityConfig:
       kubevirt:
         virtAPI: 5 1
         virtController: 4
         virtHandler: 3
         virtLauncher: 2
         virtOperator: 6

   1

   The log verbosity value must be an integer in the range 1–9, where a higher number indicates a more detailed log. In this example, the virtAPI component logs are exposed if their priority level is 5 or higher.

3. Apply your changes by saving and exiting the editor.

4. View a list of pods in the OpenShift Virtualization namespace by running the following command:

   $ oc get pods -n openshift-cnv

   Example 14.1. Example output

   NAME                               READY   STATUS    RESTARTS   AGE
   disks-images-provider-7gqbc        1/1     Running   0          32m
   disks-images-provider-vg4kx        1/1     Running   0          32m
   virt-api-57fcc4497b-7qfmc          1/1     Running   0          31m
   virt-api-57fcc4497b-tx9nc          1/1     Running   0          31m
   virt-controller-76c784655f-7fp6m   1/1     Running   0          30m
   virt-controller-76c784655f-f4pbd   1/1     Running   0          30m
   virt-handler-2m86x                 1/1     Running   0          30m
   virt-handler-9qs6z                 1/1     Running   0          30m
   virt-operator-7ccfdbf65f-q5snk     1/1     Running   0          32m
   virt-operator-7ccfdbf65f-vllz8     1/1     Running   0          32m

5. To view logs for a component pod, run the following command:

   $ oc logs -n openshift-cnv <pod_name>

   For example:

   $ oc logs -n openshift-cnv virt-handler-2m86x

   Note

   If a pod fails to start, you can use the --previous option to view logs from the last attempt.

   To monitor log output in real time, use the -f option.

   Example 14.2. Example output

   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"}
   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"}
   {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"}
   {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"}
   {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"}
   {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}

Procedure

1. In the OpenShift Container Platform console, click Virtualization → VirtualMachines from the side menu.
2. Select a virtual machine to open the VirtualMachine details page.
3. Click the Details tab.
4. Click the virt-launcher-<name> pod in the Pod section to open the Pod details page.
5. Click the Logs tab to view the pod logs.

Procedure

1. Click Virtualization → VirtualMachines from the side menu.
2. Select a virtual machine to open the VirtualMachine details page.

3. Click the Events tab to view streaming events for the virtual machine.

   • The ▮▮ button pauses the events stream.
   • The ▶ button resumes a paused events stream.

Procedure

1. Include details of the readiness probe in the VMI configuration file.

   Sample readiness probe with an HTTP GET test

   # ...
   spec:
     readinessProbe:
       httpGet: 1
         port: 1500 2
         path: /healthz 3
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       initialDelaySeconds: 120 4
       periodSeconds: 20 5
       timeoutSeconds: 10 6
       failureThreshold: 3 7
       successThreshold: 3 8
   # ...

   1

   The HTTP GET request to perform to connect to the VMI.

   2

   The port of the VMI that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VMI is considered to be healthy. If the handler returns a failure code, the VMI is removed from the list of available endpoints.

   4

   The time, in seconds, after the VMI starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the TCP readiness probe in the VMI configuration file.

   Sample readiness probe with a TCP socket test

   ...
   spec:
     readinessProbe:
       initialDelaySeconds: 120 1
       periodSeconds: 20 2
       tcpSocket: 3
         port: 1500 4
       timeoutSeconds: 10 5
   ...

   1

   The time, in seconds, after the VMI starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VMI that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the HTTP liveness probe in the VMI configuration file.

   Sample liveness probe with an HTTP GET test

   # ...
   spec:
     livenessProbe:
       initialDelaySeconds: 120 1
       periodSeconds: 20 2
       httpGet: 3
         port: 1500 4
         path: /healthz 5
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       timeoutSeconds: 10 6
   # ...

   1

   The time, in seconds, after the VMI starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VMI.

   4

   The port of the VMI that the probe queries. In the above example, the probe queries port 1500. The VMI installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VMI is considered to be healthy. If the handler returns a failure code, the VMI is deleted and a new instance is created.

   6

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the guest agent ping probe in the VMI configuration file. For example:

   Sample guest agent ping probe

   # ...
   spec:
     readinessProbe:
       guestAgentPing: {} 1
       initialDelaySeconds: 120 2
       periodSeconds: 20 3
       timeoutSeconds: 10 4
       failureThreshold: 3 5
       successThreshold: 3 6
   # ...

   1

   The guest agent ping probe to connect to the VMI.

   2

   Optional: The time, in seconds, after the VMI starts before the guest agent probe is initiated.

   3

   Optional: The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   4

   Optional: The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   5

   Optional: The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   6

   Optional: The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. In the Administrator perspective in the OpenShift Virtualization web console, navigate to Observe → Dashboards.
2. Select the KubeVirt/Infrastructure Resources/Top Consumers dashboard from the Dashboard list.
3. Select a predefined time period from the drop-down menu for Period. You can review the data for top consumers in the tables.
4. Optional: Click Inspect to view or edit the Prometheus Query Language (PromQL) query associated with the top consumers for a table.

Procedure

1. Create a manifest file that contains the ServiceAccount, Role, and RoleBinding objects with permissions that the checkup requires for cluster access:

   Example 14.3. Example role manifest file

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: vm-latency-checkup-sa
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kubevirt-vm-latency-checker
   rules:
   - apiGroups: ["kubevirt.io"]
     resources: ["virtualmachineinstances"]
     verbs: ["get", "create", "delete"]
   - apiGroups: ["subresources.kubevirt.io"]
     resources: ["virtualmachineinstances/console"]
     verbs: ["get"]
   - apiGroups: ["k8s.cni.cncf.io"]
     resources: ["network-attachment-definitions"]
     verbs: ["get"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kubevirt-vm-latency-checker
   subjects:
   - kind: ServiceAccount
     name: vm-latency-checkup-sa
   roleRef:
     kind: Role
     name: kubevirt-vm-latency-checker
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kiagnose-configmap-access
   rules:
   - apiGroups: [ "" ]
     resources: [ "configmaps" ]
     verbs: ["get", "update"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kiagnose-configmap-access
   subjects:
   - kind: ServiceAccount
     name: vm-latency-checkup-sa
   roleRef:
     kind: Role
     name: kiagnose-configmap-access
     apiGroup: rbac.authorization.k8s.io

2. Apply the checkup roles manifest:

   $ oc apply -n <target_namespace> -f <latency_roles>.yaml 1

   1

   <target_namespace> is the namespace where the checkup is to be run. This must be an existing namespace where the NetworkAttachmentDefinition object resides.

3. Create a ConfigMap manifest that contains the input parameters for the checkup. The config map provides the input for the framework to run the checkup and also stores the results of the checkup.

   Example input config map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup-config
   data:
     spec.timeout: 5m
     spec.param.network_attachment_definition_namespace: <target_namespace>
     spec.param.network_attachment_definition_name: "blue-network" 1
     spec.param.max_desired_latency_milliseconds: "10" 2
     spec.param.sample_duration_seconds: "5" 3
     spec.param.source_node: "worker1" 4
     spec.param.target_node: "worker2" 5

   1

   The name of the NetworkAttachmentDefinition object.

   2

   Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the checkup fails.

   3

   Optional: The duration of the latency check, in seconds.

   4

   Optional: When specified, latency is measured from this node to the target node. If the source node is specified, the spec.param.target_node field cannot be empty.

   5

   Optional: When specified, latency is measured from the source node to this node.

4. Apply the config map manifest in the target namespace:

   $ oc apply -n <target_namespace> -f <latency_config_map>.yaml

5. Create a Job object to run the checkup:

   Example job manifest

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: kubevirt-vm-latency-checkup
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccountName: vm-latency-checkup-sa
         restartPolicy: Never
         containers:
           - name: vm-latency-checkup
             image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup:v4.12.0
             securityContext:
               allowPrivilegeEscalation: false
               capabilities:
                 drop: ["ALL"]
               runAsNonRoot: true
               seccompProfile:
                 type: "RuntimeDefault"
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: <target_namespace>
               - name: CONFIGMAP_NAME
                 value: kubevirt-vm-latency-checkup-config

6. Apply the Job manifest. The checkup uses the ping utility to verify connectivity and measure latency.

   $ oc apply -n <target_namespace> -f <latency_job>.yaml

7. Wait for the job to complete:

   $ oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m

8. Review the results of the latency checkup by running the following command. If the maximum measured latency is greater than the value of the spec.param.max_desired_latency_milliseconds attribute, the checkup fails and returns an error.

   $ oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml

   Example output config map (success)

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup-config
     namespace: <target_namespace>
   data:
     spec.timeout: 5m
     spec.param.network_attachment_definition_namespace: <target_namespace>
     spec.param.network_attachment_definition_name: "blue-network"
     spec.param.max_desired_latency_milliseconds: "10"
     spec.param.sample_duration_seconds: "5"
     spec.param.source_node: "worker1"
     spec.param.target_node: "worker2"
     status.succeeded: "true"
     status.failureReason: ""
     status.completionTimestamp: "2022-01-01T09:00:00Z"
     status.startTimestamp: "2022-01-01T09:00:07Z"
     status.result.avgLatencyNanoSec: "177000"
     status.result.maxLatencyNanoSec: "244000" 1
     status.result.measurementDurationSec: "5"
     status.result.minLatencyNanoSec: "135000"
     status.result.sourceNode: "worker1"
     status.result.targetNode: "worker2"

   1

   The maximum measured latency in nanoseconds.

9. Optional: To view the detailed job log in case of checkup failure, use the following command:

   $ oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace>

10. Delete the job and config map resources that you previously created by running the following commands:

    $ oc delete job -n <target_namespace> kubevirt-vm-latency-checkup

    $ oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config

11. Optional: If you do not plan to run another checkup, delete the checkup role and framework manifest files.

    $ oc delete -f <file_name>.yaml

Procedure

1. Create the Service YAML file. In the following example, the file is called node-exporter-service.yaml.

   kind: Service
   apiVersion: v1
   metadata:
     name: node-exporter-service 1
     namespace: dynamation 2
     labels:
       servicetype: metrics 3
   spec:
     ports:
       - name: exmet 4
         protocol: TCP
         port: 9100 5
         targetPort: 9100 6
     type: ClusterIP
     selector:
       monitor: metrics 7

   1

   The node-exporter service that exposes the metrics from the virtual machines.

   2

   The namespace where the service is created.

   3

   The label for the service. The ServiceMonitor uses this label to match this service.

   4

   The name given to the port that exposes metrics on port 9100 for the ClusterIP service.

   5

   The target port used by node-exporter-service to listen for requests.

   6

   The TCP port number of the virtual machine that is configured with the monitor label.

   7

   The label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label monitor and a value of metrics will be matched.

2. Create the node-exporter service:

   $ oc create -f node-exporter-service.yaml

Procedure

1. Log on to the virtual machine.

2. Download the node-exporter file on to the virtual machine by using the directory path that applies to the version of node-exporter file.

   $ wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

3. Extract the executable and place it in the /usr/bin directory.

   $ sudo tar xvf node_exporter-1.3.1.linux-amd64.tar.gz \
       --directory /usr/bin --strip 1 "*/node_exporter"

4. Create a node_exporter.service file in this directory path: /etc/systemd/system. This systemd service file runs the node-exporter service when the virtual machine reboots.

   [Unit]
   Description=Prometheus Metrics Exporter
   After=network.target
   StartLimitIntervalSec=0
   
   [Service]
   Type=simple
   Restart=always
   RestartSec=1
   User=root
   ExecStart=/usr/bin/node_exporter
   
   [Install]
   WantedBy=multi-user.target

5. Enable and start the systemd service.

   $ sudo systemctl enable node_exporter.service
   $ sudo systemctl start node_exporter.service

Procedure

1. Edit the template spec of your virtual machine configuration file. In this example, the label monitor has the value metrics.

   spec:
     template:
       metadata:
         labels:
           monitor: metrics

2. Stop and restart the virtual machine to create a new pod with the label name given to the monitor label.

Procedure

1. Create a YAML file for the ServiceMonitor resource configuration. In this example, the service monitor matches any service with the label metrics and queries the exmet port every 30 seconds.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: node-exporter-metrics-monitor
     name: node-exporter-metrics-monitor 1
     namespace: dynamation 2
   spec:
     endpoints:
     - interval: 30s 3
       port: exmet 4
       scheme: http
     selector:
       matchLabels:
         servicetype: metrics

   1

   The name of the ServiceMonitor.

   2

   The namespace where the ServiceMonitor is created.

   3

   The interval at which the port will be queried.

   4

   The name of the port that is queried every 30 seconds

2. Create the ServiceMonitor configuration for the node-exporter service.

   $ oc create -f node-exporter-metrics-monitor.yaml

1. Check the cluster for a default storage class:

   $ oc get sc

   The output displays the storage classes with (default) beside the name of the default storage class. You must set a default storage class, either on the cluster or in the DataImportCron specification, in order for the DataImportCron to poll and import golden images. If no storage class is defined, the DataVolume controller fails to create PVCs and the following event is displayed: DataVolume.storage spec is missing accessMode and no storageClass to choose profile.

2. Obtain the DataImportCron namespace and name:

   $ oc get dataimportcron -A -o json | jq -r '.items[] | \
     select(.status.conditions[] | select(.type == "UpToDate" and \
     .status == "False")) | .metadata.namespace + "/" + .metadata.name'

3. If a default storage class is not defined on the cluster, check the DataImportCron specification for a default storage class:

   $ oc get dataimportcron <dataimportcron> -o yaml | \
     grep -B 5 storageClassName

   Example output

         url: docker://.../cdi-func-test-tinycore
       storage:
         resources:
           requests:
             storage: 5Gi
       storageClassName: rook-ceph-block

4. Obtain the name of the DataVolume associated with the DataImportCron object:

   $ oc -n <namespace> get dataimportcron <dataimportcron> -o json | \
     jq .status.lastImportedPVC.name

5. Check the DataVolume log for error messages:

   $ oc -n <namespace> get dv <datavolume> -o yaml

6. Set the CDI_NAMESPACE environment variable:

   $ export CDI_NAMESPACE="$(oc get deployment -A | \
     grep cdi-operator | awk '{print $1}')"

7. Check the cdi-deployment log for error messages:

   $ oc logs -n $CDI_NAMESPACE deployment/cdi-deployment

1. Set a default storage class, either on the cluster or in the DataImportCron specification, to poll and import golden images. The updated Containerized Data Importer (CDI) will resolve the issue within a few seconds.
2. If the issue does not resolve itself, delete the data volumes associated with the affected DataImportCron objects. The CDI will recreate the data volumes with the default storage class.

3. If your cluster is installed in a restricted network environment, disable the enableCommonBootImageImport feature gate in order to opt out of automatic updates:

   $ oc patch hco kubevirt-hyperconverged -n $CDI_NAMESPACE --type json \
     -p '[{"op": "replace", "path": \
     "/spec/featureGates/enableCommonBootImageImport", "value": false}]'

1. Obtain the name and namespace of the data volume:

   $ oc get dv -A -o json | jq -r '.items[] | \
     select(.status.restartCount>3)' | jq '.metadata.name, .metadata.namespace'

2. Check the status of the pods associated with the data volume:

   $ oc get pods -n <namespace> -o json | jq -r '.items[] | \
     select(.metadata.ownerReferences[] | \
     select(.name=="<dv_name>")).metadata.name'

3. Obtain the details of the pods:

   $ oc -n <namespace> describe pods <pod>

4. Check the pod logs for error messages:

   $ oc -n <namespace> describe logs <pod>

1. Set the CDI_NAMESPACE environment variable:

   $ export CDI_NAMESPACE="$(oc get deployment -A | \
     grep cdi-operator | awk '{print $1}')"

2. Check the CDI deployment for components that are not ready:

   $ oc -n $CDI_NAMESPACE get deploy -l cdi.kubevirt.io

3. Check the details of the failing pod:

   $ oc -n $CDI_NAMESPACE describe pods <pod>

4. Check the logs of the failing pod:

   $ oc -n $CDI_NAMESPACE logs <pod>

1. Set the CDI_NAMESPACE environment variable:

   $ export CDI_NAMESPACE="$(oc get deployment -A | grep cdi-operator | \
     awk '{print $1}')"

2. Check whether the cdi-operator pod is currently running:

   $ oc -n $CDI_NAMESPACE get pods -l name=cdi-operator

3. Obtain the details of the cdi-operator pod:

   $ oc -n $CDI_NAMESPACE describe pods -l name=cdi-operator

4. Check the log of the cdi-operator pod for errors:

   $ oc -n $CDI_NAMESPACE logs -l name=cdi-operator

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get deployment -A | \
     grep cluster-network-addons-operator | awk '{print $1}')"

2. Check the status of the cluster-network-addons-operator pod:

   $ oc -n $NAMESPACE get pods -l name=cluster-network-addons-operator

3. Check the cluster-network-addons-operator logs for error messages:

   $ oc -n $NAMESPACE logs -l name=cluster-network-addons-operator

4. Obtain the details of the cluster-network-addons-operator pods:

   $ oc -n $NAMESPACE describe pods -l name=cluster-network-addons-operator

1. Set the HPP_NAMESPACE environment variable:

   $ export HPP_NAMESPACE="$(oc get deployment -A | \
     grep hostpath-provisioner-operator | awk '{print $1}')"

2. Check for HPP components that are currently not ready:

   $ oc -n $HPP_NAMESPACE get all -l k8s-app=hostpath-provisioner

3. Obtain the details of the failing pod:

   $ oc -n $HPP_NAMESPACE describe pods <pod>

4. Check the logs of the failing pod:

   $ oc -n $HPP_NAMESPACE logs <pod>

1. Configure the HPP_NAMESPACE environment variable:

   $ HPP_NAMESPACE="$(oc get deployment -A | grep \
     hostpath-provisioner-operator | awk '{print $1}')"

2. Check whether the hostpath-provisioner-operator pod is currently running:

   $ oc -n $HPP_NAMESPACE get pods -l name=hostpath-provisioner-operator

3. Obtain the details of the hostpath-provisioner-operator pod:

   $ oc -n $HPP_NAMESPACE describe pods -l name=hostpath-provisioner-operator

4. Check the log of the hostpath-provisioner-operator pod for errors:

   $ oc -n $HPP_NAMESPACE logs -l name=hostpath-provisioner-operator

1. Configure the HPP_NAMESPACE environment variable:

   $ export HPP_NAMESPACE="$(oc get deployment -A | \
     grep hostpath-provisioner-operator | awk '{print $1}')"

2. Obtain the status of the hostpath-provisioner-csi daemon set pods:

   $ oc -n $HPP_NAMESPACE get pods | grep hostpath-provisioner-csi

3. Check the hostpath-provisioner-csi logs to identify the shared pool and path:

   $ oc -n $HPP_NAMESPACE logs <csi_daemonset> -c hostpath-provisioner

   Example output

   I0208 15:21:03.769731       1 utils.go:221] pool (<legacy, csi-data-dir>/csi),
   shares path with OS which can lead to node disk pressure

1. Set the KMP_NAMESPACE environment variable:

   $ export KMP_NAMESPACE="$(oc get pod -A --no-headers -l \
     control-plane=mac-controller-manager | awk '{print $1}')"

2. Set the KMP_NAME environment variable:

   $ export KMP_NAME="$(oc get pod -A --no-headers -l \
     control-plane=mac-controller-manager | awk '{print $2}')"

3. Obtain the KubeMacPool-manager pod details:

   $ oc describe pod -n $KMP_NAMESPACE $KMP_NAME

4. Check the KubeMacPool-manager logs for error messages:

   $ oc logs -n $KMP_NAMESPACE $KMP_NAME

1. Obtain the namespace and the name of the kubemacpool-mac-controller pod:

   $ oc get pod -A -l control-plane=mac-controller-manager --no-headers \
     -o custom-columns=":metadata.namespace,:metadata.name"

2. Obtain the duplicate MAC addresses from the kubemacpool-mac-controller logs:

   $ oc logs -n <namespace> <kubemacpool_mac_controller> | \
     grep "already allocated"

   Example output

   mac address 02:00:ff:ff:ff:ff already allocated to
   vm/kubemacpool-test/testvm, br1,
   conflict with: vm/kubemacpool-test/testvm2, br1

1. Update the VMs to remove the duplicate MAC addresses.

2. Restart the kubemacpool-mac-controller pod:

   $ oc delete pod -n <namespace> <kubemacpool_mac_controller>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the component’s CPU request limit:

   $ oc -n $NAMESPACE get deployment <component> -o yaml | grep requests: -A 2

3. Check the actual CPU usage by using a PromQL query:

   node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate
   {namespace="$NAMESPACE",container="<component>"}

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the component’s memory request limit:

   $ oc -n $NAMESPACE get deployment <component> -o yaml | \
     grep requests: -A 2

3. Check the actual memory usage by using a PromQL query:

   container_memory_usage_bytes{namespace="$NAMESPACE",container="<component>"}

1. Obtain the virt-launcher pod details:

   $ oc get pod <virt-launcher> -o yaml

2. Identify compute container processes with high memory usage in the virt-launcher pod:

   $ oc exec -it <virt-launcher> -c compute -- top

1. Verify that the worker node has sufficient resources:

   $ oc get nodes -l node-role.kubernetes.io/worker= -o json | \
     jq .items[].status.allocatable

   Example output

   {
     "cpu": "3500m",
     "devices.kubevirt.io/kvm": "1k",
     "devices.kubevirt.io/sev": "0",
     "devices.kubevirt.io/tun": "1k",
     "devices.kubevirt.io/vhost-net": "1k",
     "ephemeral-storage": "38161122446",
     "hugepages-1Gi": "0",
     "hugepages-2Mi": "0",
     "memory": "7000128Ki",
     "pods": "250"
   }

2. Check the status of the worker node:

   $ oc get nodes -l node-role.kubernetes.io/worker= -o json | \
     jq .items[].status.conditions

   Example output

   {
     "lastHeartbeatTime": "2022-05-26T07:36:01Z",
     "lastTransitionTime": "2022-05-23T08:12:02Z",
     "message": "kubelet has sufficient memory available",
     "reason": "KubeletHasSufficientMemory",
     "status": "False",
     "type": "MemoryPressure"
   },
   {
     "lastHeartbeatTime": "2022-05-26T07:36:01Z",
     "lastTransitionTime": "2022-05-23T08:12:02Z",
     "message": "kubelet has no disk pressure",
     "reason": "KubeletHasNoDiskPressure",
     "status": "False",
     "type": "DiskPressure"
   },
   {
     "lastHeartbeatTime": "2022-05-26T07:36:01Z",
     "lastTransitionTime": "2022-05-23T08:12:02Z",
     "message": "kubelet has sufficient PID available",
     "reason": "KubeletHasSufficientPID",
     "status": "False",
     "type": "PIDPressure"
   },
   {
     "lastHeartbeatTime": "2022-05-26T07:36:01Z",
     "lastTransitionTime": "2022-05-23T08:24:15Z",
     "message": "kubelet is posting ready status",
     "reason": "KubeletReady",
     "status": "True",
     "type": "Ready"
   }

3. Log in to the worker node and verify that the kubelet service is running:

   $ systemctl status kubelet

4. Check the kubelet journal log for error messages:

   $ journalctl -r -u kubelet

1. Check the virtual machine instance (VMI) details:

   $ oc describe vmi <vmi> -n <namespace>

   Example output

   Name:          testvmi-hxghp
   Namespace:     kubevirt-test-default1
   Labels:        name=testvmi-hxghp
   Annotations:   kubevirt.io/latest-observed-api-version: v1
                  kubevirt.io/storage-observed-api-version: v1alpha3
   API Version:   kubevirt.io/v1
   Kind:          VirtualMachineInstance
   ...
   Spec:
     Domain:
   ...
       Resources:
         Requests:
           Cpu:     5000000Gi
           Memory:  5130000240Mi
   ...
   Status:
   ...
     Conditions:
       Last Probe Time:       2022-10-03T11:11:07Z
       Last Transition Time:  2022-10-03T11:11:07Z
       Message:               Guest VM is not reported as running
       Reason:                GuestNotRunning
       Status:                False
       Type:                  Ready
       Last Probe Time:       <nil>
       Last Transition Time:  2022-10-03T11:11:07Z
       Message:               0/2 nodes are available: 2 Insufficient cpu, 2
         Insufficient memory.
       Reason:                Unschedulable
       Status:                False
       Type:                  PodScheduled
     Guest OS Info:
     Phase:  Scheduling
     Phase Transition Timestamps:
       Phase:                        Pending
       Phase Transition Timestamp:   2022-10-03T11:11:07Z
       Phase:                        Scheduling
       Phase Transition Timestamp:   2022-10-03T11:11:07Z
     Qos Class:                       Burstable
     Runtime User:                    0
     Virtual Machine Revision Name:   revision-start-vm-3503e2dc-27c0-46ef-9167-7ae2e7d93e6e-1
   Events:
     Type    Reason            Age   From                       Message
     ----    ------            ----  ----                       -------
     Normal  SuccessfulCreate  27s   virtualmachine-controller  Created virtual
       machine pod virt-launcher-testvmi-hxghp-xh9qn

2. Check the node resources:

   $ oc get nodes -l node-role.kubernetes.io/worker= -o json | jq '.items | \
     .[].status.allocatable'

   Example output

   {
     "cpu": "5",
     "devices.kubevirt.io/kvm": "1k",
     "devices.kubevirt.io/sev": "0",
     "devices.kubevirt.io/tun": "1k",
     "devices.kubevirt.io/vhost-net": "1k",
     "ephemeral-storage": "33812468066",
     "hugepages-1Gi": "0",
     "hugepages-2Mi": "128Mi",
     "memory": "3783496Ki",
     "pods": "110"
   }

3. Check the node for error conditions:

   $ oc get nodes -l node-role.kubernetes.io/worker= -o json | jq '.items | \
     .[].status.conditions'

   Example output

   [
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:20Z",
       "message": "kubelet has sufficient memory available",
       "reason": "KubeletHasSufficientMemory",
       "status": "False",
       "type": "MemoryPressure"
     },
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:20Z",
       "message": "kubelet has no disk pressure",
       "reason": "KubeletHasNoDiskPressure",
       "status": "False",
       "type": "DiskPressure"
     },
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:20Z",
       "message": "kubelet has sufficient PID available",
       "reason": "KubeletHasSufficientPID",
       "status": "False",
       "type": "PIDPressure"
     },
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:30Z",
       "message": "kubelet is posting ready status",
       "reason": "KubeletReady",
       "status": "True",
       "type": "Ready"
     }
   ]

1. Check the node resources:

   $ oc get nodes -l node-role.kubernetes.io/worker= -o json | jq '.items | \
     .[].status.allocatable'

   Example output

   {
      "cpu": "5",
      "devices.kubevirt.io/kvm": "1k",
      "devices.kubevirt.io/sev": "0",
      "devices.kubevirt.io/tun": "1k",
      "devices.kubevirt.io/vhost-net": "1k",
      "ephemeral-storage": "33812468066",
      "hugepages-1Gi": "0",
      "hugepages-2Mi": "128Mi",
      "memory": "3783496Ki",
      "pods": "110"
   }

2. Check the node status conditions:

   $ oc get nodes -l node-role.kubernetes.io/worker= -o json | jq '.items | \
     .[].status.conditions'

   Example output

   [
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:20Z",
       "message": "kubelet has sufficient memory available",
       "reason": "KubeletHasSufficientMemory",
       "status": "False",
       "type": "MemoryPressure"
     },
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:20Z",
       "message": "kubelet has no disk pressure",
       "reason": "KubeletHasNoDiskPressure",
       "status": "False",
       "type": "DiskPressure"
     },
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:20Z",
       "message": "kubelet has sufficient PID available",
       "reason": "KubeletHasSufficientPID",
       "status": "False",
       "type": "PIDPressure"
     },
     {
       "lastHeartbeatTime": "2022-10-03T11:13:34Z",
       "lastTransitionTime": "2022-10-03T10:14:30Z",
       "message": "kubelet is posting ready status",
       "reason": "KubeletReady",
       "status": "True",
       "type": "Ready"
     }
   ]

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Verify a virt-controller device is available:

   $ oc get deployment -n $NAMESPACE virt-controller \
     -o jsonpath='{.status.readyReplicas}'

3. Check the status of the virt-controller deployment:

   $ oc -n $NAMESPACE get deploy virt-controller -o yaml

4. Obtain the details of the virt-controller deployment to check for status conditions, such as crashing pods or failures to pull images:

   $ oc -n $NAMESPACE describe deploy virt-controller

5. Check if any problems occurred with the nodes. For example, they might be in a NotReady state:

   $ oc get nodes

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Obtain the name of the virt-operator deployment:

   $ oc -n $NAMESPACE get deploy virt-operator -o yaml

3. Obtain the details of the virt-operator deployment:

   $ oc -n $NAMESPACE describe deploy virt-operator

4. Check for node issues, such as a NotReady state:

   $ oc get nodes

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the number of available virt-api pods:

   $ oc get deployment -n $NAMESPACE virt-api \
     -o jsonpath='{.status.readyReplicas}'

3. Check the status of the virt-api deployment for error conditions:

   $ oc -n $NAMESPACE get deploy virt-api -o yaml

4. Check the nodes for issues such as nodes in a NotReady state:

   $ oc get nodes

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Verify that running virt-controller pods are available:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-controller

3. Check the virt-launcher logs for error messages:

   $ oc -n $NAMESPACE logs <virt-launcher>

4. Obtain the details of the virt-launcher pod to check for status conditions such as unexpected termination or a NotReady state.

   $ oc -n $NAMESPACE describe pod/<virt-launcher>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the states of the virt-operator pods:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-operator

3. Review the logs of the affected virt-operator pods:

   $ oc -n $NAMESPACE logs <virt-operator>

4. Obtain the details of the affected virt-operator pods:

   $ oc -n $NAMESPACE describe pod <virt-operator>

1. Check the status conditions of the NetworkAddonsConfig CR to identify the deployment or daemon set that is not ready:

   $ oc get networkaddonsconfig \
     -o custom-columns="":.status.conditions[*].message

   Example output

   DaemonSet "cluster-network-addons/macvtap-cni" update is being processed...

2. Check the component’s pod for errors:

   $ oc -n cluster-network-addons get daemonset <pod> -o yaml

3. Check the component’s logs:

   $ oc -n cluster-network-addons logs <pod>

4. Check the component’s details for error conditions:

   $ oc -n cluster-network-addons describe <pod>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A -o \
     custom-columns="":.metadata.namespace)"

2. Obtain the status of the virt-operator pods:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-operator

3. Check the virt-operator pod logs to determine the leader status:

   $ oc -n $NAMESPACE logs | grep lead

   Leader pod example:

   {"component":"virt-operator","level":"info","msg":"Attempting to acquire
   leader status","pos":"application.go:400","timestamp":"2021-11-30T12:15:18.635387Z"}
   I1130 12:15:18.635452       1 leaderelection.go:243] attempting to acquire
   leader lease <namespace>/virt-operator...
   I1130 12:15:19.216582       1 leaderelection.go:253] successfully acquired
   lease <namespace>/virt-operator
   {"component":"virt-operator","level":"info","msg":"Started leading",
   "pos":"application.go:385","timestamp":"2021-11-30T12:15:19.216836Z"}

   Non-leader pod example:

   {"component":"virt-operator","level":"info","msg":"Attempting to acquire
   leader status","pos":"application.go:400","timestamp":"2021-11-30T12:15:20.533696Z"}
   I1130 12:15:20.533792       1 leaderelection.go:243] attempting to acquire
   leader lease <namespace>/virt-operator...

4. Obtain the details of the affected virt-operator pods:

   $ oc -n $NAMESPACE describe pod <virt-operator>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Verify the number of virt-controller devices:

   $ oc get deployment -n $NAMESPACE virt-controller \
     -o jsonpath='{.status.readyReplicas}'

3. Check the status of the virt-controller deployment:

   $ oc -n $NAMESPACE get deploy virt-controller -o yaml

4. Obtain the details of the virt-controller deployment to check for status conditions such as crashing pods or failure to pull images:

   $ oc -n $NAMESPACE describe deploy virt-controller

5. Obtain the details of the virt-controller pods:

   $ get pods -n $NAMESPACE | grep virt-controller

6. Check the logs of the virt-controller pods for error messages:

   $ oc logs -n $NAMESPACE <virt-controller>

7. Check the nodes for problems, such as a NotReady state:

   $ oc get nodes

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Obtain the name of the virt-operator deployment:

   $ oc -n $NAMESPACE get deploy virt-operator -o yaml

3. Generate the description of the virt-operator deployment:

   $ oc -n $NAMESPACE describe deploy virt-operator

4. Check for node issues, such as a NotReady state:

   $ oc get nodes

1. Check the status of the virt-handler pods to view the nodes on which they are running:

   $ oc get pods --all-namespaces -o wide -l kubevirt.io=virt-handler

2. Check the status of the VMIs to identify VMIs running on nodes that do not have a running virt-handler pod:

   $ oc get vmis --all-namespaces

3. Check the status of the virt-handler daemon:

   $ oc get daemonset virt-handler --all-namespaces

   Example output

   NAME          DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE ...
   virt-handler  2        2        2      2           2         ...

   The daemon set is considered healthy if the Desired, Ready, and Available columns contain the same value.

4. If the virt-handler daemon set is not healthy, check the virt-handler daemon set for pod deployment issues:

   $ oc get daemonset virt-handler --all-namespaces -o yaml | jq .status

5. Check the nodes for issues such as a NotReady status:

   $ oc get nodes

6. Check the spec.workloads stanza of the KubeVirt custom resource (CR) for a workloads placement policy:

   $ oc get kubevirt kubevirt --all-namespaces -o yaml

1. Identify the outdated VMIs:

   $ oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces

2. Check the KubeVirt custom resource (CR) to determine whether workloadUpdateMethods is configured in the workloadUpdateStrategy stanza:

   $ oc get kubevirt kubevirt --all-namespaces -o yaml

3. Check each outdated VMI to determine whether it is live-migratable:

   $ oc get vmi <vmi> -o yaml

   Example output

   apiVersion: kubevirt.io/v1
   kind: VirtualMachineInstance
   ...
     status:
       conditions:
       - lastProbeTime: null
         lastTransitionTime: null
         message: cannot migrate VMI which does not use masquerade
         to connect to the pod network
         reason: InterfaceNotLiveMigratable
         status: "False"
         type: LiveMigratable

1. Create a VirtualMachineInstanceMigration manifest and save it as migration.yaml:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachineInstanceMigration
   metadata:
     name: <migration_name>
     namespace: <namespace>
   spec:
     vmiName: <vmi_name>

2. Create a VirtualMachineInstanceMigration object to trigger the migration:

   $ oc create -f migration.yaml

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get deployment -A | grep ssp-operator | \
     awk '{print $1}')"

2. Check the ssp-operator logs for templates with reverted changes:

   $ oc -n $NAMESPACE logs --tail=-1 -l control-plane=ssp-operator | \
     grep 'common template' -C 3

1. Export the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get deployment -A | grep ssp-operator | \
     awk '{print $1}')"

2. Obtain the details of the ssp-operator pods:

   $ oc -n $NAMESPACE describe pods -l control-plane=ssp-operator

3. Check the ssp-operator logs for errors:

   $ oc -n $NAMESPACE logs --tail=-1 -l control-plane=ssp-operator

4. Obtain the status of the virt-template-validator pods:

   $ oc -n $NAMESPACE get pods -l name=virt-template-validator

5. Obtain the details of the virt-template-validator pods:

   $ oc -n $NAMESPACE describe pods -l name=virt-template-validator

6. Check the virt-template-validator logs for errors:

   $ oc -n $NAMESPACE logs --tail=-1 -l name=virt-template-validator

1. Export the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get deployment -A | grep ssp-operator | \
     awk '{print $1}')"

2. Check the virt-template-validator logs for errors that might indicate the cause:

   $ oc -n $NAMESPACE logs --tail=-1 -l name=virt-template-validator

   Example output

   {"component":"kubevirt-template-validator","level":"info","msg":"evalution
   summary for ubuntu-3166wmdbbfkroku0:\nminimal-required-memory applied: FAIL,
   value 1073741824 is lower than minimum [2147483648]\n\nsucceeded=false",
   "pos":"admission.go:25","timestamp":"2021-09-28T17:59:10.934470Z"}

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get deployment -A | grep ssp-operator | \
     awk '{print $1}')"

2. Check the status of the ssp-operator pods.

   $ oc -n $NAMESPACE get pods -l control-plane=ssp-operator

3. Obtain the details of the ssp-operator pods:

   $ oc -n $NAMESPACE describe pods -l control-plane=ssp-operator

4. Check the ssp-operator logs for error messages:

   $ oc -n $NAMESPACE logs --tail=-1 -l control-plane=ssp-operator

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get deployment -A | grep ssp-operator | \
     awk '{print $1}')"

2. Obtain the status of the virt-template-validator pods:

   $ oc -n $NAMESPACE get pods -l name=virt-template-validator

3. Obtain the details of the virt-template-validator pods:

   $ oc -n $NAMESPACE describe pods -l name=virt-template-validator

4. Check the virt-template-validator logs for error messages:

   $ oc -n $NAMESPACE logs --tail=-1 -l name=virt-template-validator

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-api pods:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-api

3. Check the status of the virt-api deployment:

   $ oc -n $NAMESPACE get deploy virt-api -o yaml

4. Check the virt-api deployment details for issues such as crashing pods or image pull failures:

   $ oc -n $NAMESPACE describe deploy virt-api

5. Check for issues such as nodes in a NotReady state:

   $ oc get nodes

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Obtain the list of virt-api pods on your deployment:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-api

3. Check the virt-api logs for error messages:

   $ oc logs -n $NAMESPACE <virt-api>

4. Obtain the details of the virt-api pods:

   $ oc describe -n $NAMESPACE <virt-api>

5. Check if any problems occurred with the nodes. For example, they might be in a NotReady state:

   $ oc get nodes

6. Check the status of the virt-api deployment:

   $ oc -n $NAMESPACE get deploy virt-api -o yaml

7. Obtain the details of the virt-api deployment:

   $ oc -n $NAMESPACE describe deploy virt-api

1. Set the NAMESPACE environment variable as follows:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-api pods:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-api

3. Check the virt-api logs:

   $ oc logs -n  $NAMESPACE <virt-api>

4. Obtain the details of the virt-api pods:

   $ oc describe -n $NAMESPACE <virt-api>

5. Check if any problems occurred with the nodes. For example, they might be in a NotReady state:

   $ oc get nodes

6. Check the status of the virt-api deployment:

   $ oc -n $NAMESPACE get deploy virt-api -o yaml

7. Obtain the details of the virt-api deployment:

   $ oc -n $NAMESPACE describe deploy virt-api

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-controller deployment:

   $ oc get deployment -n $NAMESPACE virt-controller -o yaml

3. Review the logs of the virt-controller pod:

   $ oc get logs <virt-controller>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. List the available virt-controller pods:

   $ oc get pods -n $NAMESPACE -l=kubevirt.io=virt-controller

3. Check the virt-controller logs for error messages when connecting to the API server:

   $ oc logs -n  $NAMESPACE <virt-controller>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. List the available virt-controller pods:

   $ oc get pods -n $NAMESPACE -l=kubevirt.io=virt-controller

3. Check the virt-controller logs for error messages when connecting to the API server:

   $ oc logs -n  $NAMESPACE <virt-controller>

1. Export the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-handler pods to identify pods that have not deployed:

   $ oc get pods -n $NAMESPACE -l=kubevirt.io=virt-handler

3. Obtain the name of the worker node of the virt-handler pod:

   $ oc -n $NAMESPACE get pod <virt-handler> -o jsonpath='{.spec.nodeName}'

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-handler pod:

   $ oc get pods -n $NAMESPACE -l=kubevirt.io=virt-handler

3. Check the virt-handler logs for error messages when connecting to the API server:

   $ oc logs -n  $NAMESPACE <virt-handler>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-handler pod:

   $ oc get pods -n $NAMESPACE -l=kubevirt.io=virt-handler

3. Check the virt-handler logs for error messages when connecting to the API server:

   $ oc logs -n $NAMESPACE <virt-handler>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-operator deployment:

   $ oc -n $NAMESPACE get deploy virt-operator -o yaml

3. Obtain the details of the virt-operator deployment:

   $ oc -n $NAMESPACE describe deploy virt-operator

4. Check the status of the virt-operator pods:

   $ oc get pods -n $NAMESPACE -l=kubevirt.io=virt-operator

5. Check for node issues, such as a NotReady state:

   $ oc get nodes

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-operator pods:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-operator

3. Check the virt-operator logs for error messages when connecting to the API server:

   $ oc -n $NAMESPACE logs <virt-operator>

4. Obtain the details of the virt-operator pod:

   $ oc -n $NAMESPACE describe pod <virt-operator>

1. Set the NAMESPACE environment variable:

   $ export NAMESPACE="$(oc get kubevirt -A \
     -o custom-columns="":.metadata.namespace)"

2. Check the status of the virt-operator pods:

   $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-operator

3. Check the virt-operator logs for error messages when connecting to the API server:

   $ oc -n $NAMESPACE logs <virt-operator>

4. Obtain the details of the virt-operator pod:

   $ oc -n $NAMESPACE describe pod <virt-operator>

1. Check the VMI configuration to determine whether the value of evictionStrategy is LiveMigrate:

   $ oc get vmis -o yaml

2. Check for a False status in the LIVE-MIGRATABLE column to identify VMIs that are not migratable:

   $ oc get vmis -o wide

3. Obtain the details of the VMI and check spec.conditions to identify the issue:

   $ oc get vmi <vmi> -o yaml

   Example output

   status:
     conditions:
     - lastProbeTime: null
       lastTransitionTime: null
       message: cannot migrate VMI which does not use masquerade to connect
       to the pod network
       reason: InterfaceNotLiveMigratable
       status: "False"
       type: LiveMigratable

Procedure

1. Collect must-gather data for the cluster by using the default must-gather image.
2. Collect must-gather data for Red Hat OpenShift Data Foundation, if necessary.
3. Collect must-gather data for OpenShift Virtualization by using the OpenShift Virtualization must-gather image.
4. Collect Prometheus metrics for the cluster.

Procedure

1. Collect detailed must-gather data about the malfunctioning VMs.
2. Collect screenshots of VMs that have crashed before you restart them.
3. Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network.