hypervisor showing cpu used and 16 instances

Solution In Progress - Updated -

Issue

  • We cant instantiate any new VM'S

  • Can you help us with this urgent case, the hypervisor is showing used resources even though they have all been evacuated.

  • overcloud-compue-0 has no instances running but is showing that it is consumed in the hypervisor info.

(overcloud) [stack@undercloud ~]$ openstack hypervisor show overcloud-compute-0
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| aggregates           | [u'sriov_group']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| cpu_info             | {"vendor": "Intel", "model": "Skylake-Server-IBRS", "arch": "x86_64", "features": ["pku", "rtm", "tsc_adjust", "vme", "pge", "xsaveopt", "smep", "fpu", "monitor", "lm", "tsc", "adx", "fxsr", "tm", "pclmuldq", "xgetbv1", "tsc-deadline", "arat", "de", "aes", "pse", "sse", "f16c", "ds", "mpx", "avx512f", "avx2", "pbe", "cx16", "ds_cpl", "movbe", "ospke", "vmx", "sep", "avx512dq", "stibp", "xsave", "erms", "hle", "est", "smx", "abm", "sse4.1", "sse4.2", "ssbd", "acpi", "mmx", "osxsave", "clwb", "dca", "popcnt", "invtsc", "tm2", "pcid", "pdcm", "avx512vl", "x2apic", "smap", "clflush", "dtes64", "xtpr", "avx512bw", "msr", "fma", "cx8", "mce", "avx512cd", "mca", "lahf_lm", "rdseed", "apic", "fsgsbase", "rdtscp", "ssse3", "pse36", "mtrr", "avx", "syscall", "invpcid", "cmov", "spec-ctrl", "clflushopt", "pat", "3dnowprefetch", "nx", "pae", "md-clear", "ht", "pdpe1gb", "rdrand", "xsavec", "pni", "sse2", "ss", "bmi1", "bmi2", "xsaves", "arch-facilities"], "topology": {"cores": 18, "cells": 2, "threads": 2, "sockets": 1}} |
| current_workload     | 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| disk_available_least | 362                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| free_disk_gb         | 393                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| free_ram_mb          | 145024                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| host_ip              | 192.168.3.112                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| host_time            | 16:08:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| hypervisor_hostname  | overcloud-compute-0.localdomain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| hypervisor_type      | QEMU                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| hypervisor_version   | 2012000                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| id                   | 30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| load_average         | 0.90, 0.92, 0.76                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| local_gb             | 449                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| local_gb_used        | 56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| memory_mb            | 391808                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| memory_mb_used       | 246784                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| running_vms          | 16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| service_host         | overcloud-compute-0.localdomain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| service_id           | 180                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| state                | up                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| status               | enabled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| uptime               | 21 min                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| users                | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| vcpus                | 68                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| vcpus_used           | 102                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  • virsh list --all shows a running VM:
#virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     instance-000006c5              shut off
  • The following log is seen in `/var/log/containers/nova/nova-compute.log:
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager [req-d325cf39-c30d-48f6-aa5a-c2592ba77b3a - - - - -] Error updating resources for node overcloud-compute-0.: DiskNotFound: No disk at /var/lib/nova/instances/480c39b2-7568-4616-b167-864c0eeb65ce/disk.swap
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager Traceback (most recent call last):
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7447, in update_available_resource_for_node
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 690, in update_available_resource
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6485, in get_available_resource
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8048, in _get_disk_over_committed_size_total
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     err_ctxt.reraise = False
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     self.force_reraise()
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7989, in _get_disk_over_committed_size_total
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     config, block_device_info)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7891, in _get_instance_disk_info_from_config
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     qemu_img_info = disk_api.get_disk_info(path)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 136, in get_disk_info
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     return images.qemu_img_info(path)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 57, in qemu_img_info
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager     raise exception.DiskNotFound(location=path)
2020-10-30 18:31:53.397 1 ERROR nova.compute.manager DiskNotFound: No disk at /var/lib/nova/instances/480c39b2-7568-4616-b167-864c0eeb65ce/disk.swap

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content