podman stats shows incorrect stats for containers

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • podman

Issue

  • When an already running podman container is made some changes(like memory for example), running podman stats does not show the changes made

Resolution

  • A bugzilla has been created to track this and made podman stats show the live changes made to the running container

Root Cause

  • The changes made are actually reflected in the running container but running podman stats doesn't just show the changes made
  • This is because podman containers are designed to be immutable and hence these changes are valid only until the container is restarted. Once it is restarted, the configuration is set back to how it was created initially
  • Since these changes are short lived, it is designed not be show the changes in stats

Diagnostic Steps

  • Check for the stats of a running container
[root@rhel9 ~]# podman run -itd --name memory-test ubi8
f9b4354d84fae97e95bd04d12213e1296b6f064dda092a4238095eb7a65f46fc
[root@rhel9 ~]# podman stats memory-test
ID            NAME         CPU %       MEM USAGE / LIMIT  MEM %       NET IO          BLOCK IO      PIDS        CPU TIME    AVG CPU %
f9b4354d84fa  memory-test  0.50%       544.8kB / 1.793GB  0.03%       2.016kB / 586B  7.045MB / 0B  1           59.092ms    0.50%
  • Modify the memory(or any desired parameter) and check the updated stats
[root@rhel9 ~]# podman update --memory 1G memory-test 
f9b4354d84fae97e95bd04d12213e1296b6f064dda092a4238095eb7a65f46fc
[root@rhel9 ~]# podman stats memory-test 
ID            NAME         CPU %       MEM USAGE / LIMIT  MEM %       NET IO          BLOCK IO      PIDS        CPU TIME    AVG CPU %
f9b4354d84fa  memory-test  0.08%       544.8kB / 1.793GB  0.03%       2.226kB / 796B  7.045MB / 0B  1           59.092ms    0.08%
  • The changes are not reflected and can be seen from the above command
  • The changes are however reflected inside the running container and is evident from the below command
[root@rhel9 ~]# podman exec -it memory-test cat /sys/fs/cgroup/memory.max
1073741824

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments