Strange memory usage

Posted on

Hi all,
I've encountered a strange issue with memory usage with CentOS 8 and containers.
I've deployed a containerized (podman) openstack 16 environment, with tripleo, and after some weeks/months all of my servers runs out of memory, but the strange thing is that no process is using this memory.
Checking memory with free, I can see the memory "available" very low, also for top, but I can't find the process (or the container) that are using the memory.
But the very strange thing is that, after a compute restart, in the case of openstack but also with a simple rhel environment with podman, I can see the memory usage back to normal (free output is similar of the sum of process usage).
So, I've start digging in the problem and found that in the /proc/meminfo, a parameter "percpu" keep increasing but after a reboot come back to normal:
- before a reboot -> Percpu: 394674688 kB
- after a reboot -> Percpu: 77312 kB
after some research I've found this issue: https://access.redhat.com/solutions/6369421 , but I can't understand the meaning of this "percpu" parameter.

I've attached some screen of the typical issue scenario on a compute, but it's the same for the ceph-storage, controller or any centos+podman environment.

free -h

horizon-hypervisorsmeminfo

top

Could this be the reason of my strange ram issue? If not, what are your opinion about this issue?