hypervisor showing cpu used and 16 instances
Issue
-
We cant instantiate any new VM'S
-
Can you help us with this urgent case, the hypervisor is showing used resources even though they have all been evacuated.
-
overcloud-compue-0has no instances running but is showing that it is consumed in the hypervisor info.
(overcloud) [stack@undercloud ~]$ openstack hypervisor show overcloud-compute-0
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| aggregates | [u'sriov_group'] |
| cpu_info | {"vendor": "Intel", "model": "Skylake-Server-IBRS", "arch": "x86_64", "features": ["pku", "rtm", "tsc_adjust", "vme", "pge", "xsaveopt", "smep", "fpu", "monitor", "lm", "tsc", "adx", "fxsr", "tm", "pclmuldq", "xgetbv1", "tsc-deadline", "arat", "de", "aes", "pse", "sse", "f16c", "ds", "mpx", "avx512f", "avx2", "pbe", "cx16", "ds_cpl", "movbe", "ospke", "vmx", "sep", "avx512dq", "stibp", "xsave", "erms", "hle", "est", "smx", "abm", "sse4.1", "sse4.2", "ssbd", "acpi", "mmx", "osxsave", "clwb", "dca", "popcnt", "invtsc", "tm2", "pcid", "pdcm", "avx512vl", "x2apic", "smap", "clflush", "dtes64", "xtpr", "avx512bw", "msr", "fma", "cx8", "mce", "avx512cd", "mca", "lahf_lm", "rdseed", "apic", "fsgsbase", "rdtscp", "ssse3", "pse36", "mtrr", "avx", "syscall", "invpcid", "cmov", "spec-ctrl", "clflushopt", "pat", "3dnowprefetch", "nx", "pae", "md-clear", "ht", "pdpe1gb", "rdrand", "xsavec", "pni", "sse2", "ss", "bmi1", "bmi2", "xsaves", "arch-facilities"], "topology": {"cores": 18, "cells": 2, "threads": 2, "sockets": 1}} |
| current_workload | 1 |
| disk_available_least | 362 |
| free_disk_gb | 393 |
| free_ram_mb | 145024 |
| host_ip | 192.168.3.112 |
| host_time | 16:08:37 |
| hypervisor_hostname | overcloud-compute-0.localdomain |
| hypervisor_type | QEMU |
| hypervisor_version | 2012000 |
| id | 30 |
| load_average | 0.90, 0.92, 0.76 |
| local_gb | 449 |
| local_gb_used | 56 |
| memory_mb | 391808 |
| memory_mb_used | 246784 |
| running_vms | 16 |
| service_host | overcloud-compute-0.localdomain |
| service_id | 180 |
| state | up |
| status | enabled |
| uptime | 21 min |
| users | 0 |
| vcpus | 68 |
| vcpus_used | 102 |
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
virsh list --allshows a running VM:
#virsh list --all
Id Name State
----------------------------------------------------
- instance-000006c5 shut off
- The following log is seen in `/var/log/containers/nova/nova-compute.log:
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager [req-d325cf39-c30d-48f6-aa5a-c2592ba77b3a - - - - -] Error updating resources for node overcloud-compute-0.: DiskNotFound: No disk at /var/lib/nova/instances/480c39b2-7568-4616-b167-864c0eeb65ce/disk.swap
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager Traceback (most recent call last):
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7447, in update_available_resource_for_node
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 690, in update_available_resource
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6485, in get_available_resource
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8048, in _get_disk_over_committed_size_total
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager err_ctxt.reraise = False
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager self.force_reraise()
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7989, in _get_disk_over_committed_size_total
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager config, block_device_info)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7891, in _get_instance_disk_info_from_config
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager qemu_img_info = disk_api.get_disk_info(path)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 136, in get_disk_info
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager return images.qemu_img_info(path)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 57, in qemu_img_info
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager raise exception.DiskNotFound(location=path)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager DiskNotFound: No disk at /var/lib/nova/instances/480c39b2-7568-4616-b167-864c0eeb65ce/disk.swap
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
