Unknown memory utilization for VM

Issue

• VM could not be rebooted, following error reported in /var/log/messages:

Mar 19 04:10:49 overcloud-compute-24 libvirtd: 2020-03-19T04:10:49.021584Z qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory

• VM Flavor needs 64gb memory which is not available on this overcloud-compute-24:

[stack@undercloud ~]$ openstack flavor show pcrf-ipam-oam
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| access_project_ids         | None                                 |
| disk                       | 100                                  |
| id                         | 3405278d-ec3b-4333-ba13-446909359b9c |
| name                       | pcrf-ipam-oam                        |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 65536                                |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 8                                    |
+----------------------------+--------------------------------------+

• Checking memory utilization on compute-24:

[heat-admin@overcloud-compute-24 ~]$ free -h
                     total        used        free      shared  buff/cache   available
Mem:           251G        194G         47G        2.9M        9.9G         54G
Swap:            0B          0B          0B

• If we add 4 instances + 8 other high utilization processes, the consumption doesn’t add up to 194gb.

[heat-admin@overcloud-compute-24 ~]$ top -o %MEM
top - 06:11:51 up 359 days,  5:54,  2 users,  load average: 3.35, 3.28, 2.91
Tasks: 564 total,   1 running, 563 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.6 us,  1.7 sy,  0.0 ni, 93.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26392284+total, 49640396 free, 20406246+used, 10219984 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 57140816 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                              
 414490 qemu      20   0   97.4g  30.2g   4604 S  31.9 12.0 138723:33 qemu-kvm                                                                                                                                                             
 412624 qemu      20   0   17.3g  15.9g   3132 S  55.8  6.3 279513:48 qemu-kvm                                                                                                                                                             
 411702 qemu      20   0   17.1g  15.9g   4724 S   1.0  6.3  26242:38 qemu-kvm                                                                                                                                                             
 415201 qemu      20   0   17.3g  14.9g   3120 S  83.7  5.9 172726:31 qemu-kvm                                                                                                                                                             
 298582 openvsw+  10 -10 4279568 493020  13340 S   1.7  0.2   0:59.09 ovs-vswitchd                                                                                                                                                         
 409084 nova      20   0 2406488 125196   5476 S   0.0  0.0   4388:11 nova-compute                                                                                                                                                         
 298885 neutron   20   0  385600  88520   6092 S   0.7  0.0   0:37.93 neutron-openvsw                                                                                                                                                      
   1021 root      20   0  121420  56800  56368 S   0.0  0.0  31:28.77 systemd-journal                                                                                                                                                      
   6357 root      20   0  333808  41024   3392 S   0.0  0.0 878:33.13 os-collect-conf                                                                                                                                                      
   6036 root      20   0  731552  39144  33512 S   0.0  0.0  31:50.39 rsyslogd                                                                                                                                                             
 409141 root      20   0  267856  22952   1476 S   0.0  0.0   0:00.96 privsep-helper                                                                                                                                                       
 219748 root      20   0  310008  18900   3020 S   0.0  0.0   0:00.13 rhsmd                                                                                                                                                                
 408781 root      20   0 1602604  18516   6972 S   0.0  0.0 984:20.12 libvirtd  

• Something is consuming memory, and we require assistance.