RHEL 8/9: The num_cgroups for blkio in cgroups keeps increasing

Issue

• The num_cgroups for blkio in cgroups keeps increasing:

$ cat /proc/cgroups | grep -e subsys -e blkio | column -t
   #subsys_name  hierarchy  num_cgroups  enabled
   blkio         4          1099         1
   memory        7          28           1

• kmalloc-* slab is consuming more memory.

0xffff8d8c80003500 kmalloc-2k                       745376 K    2048
0xffff8d8c800036c0 kmalloc-4k                       614240 K    4096
0xffff8d8c80003340 kmalloc-1k                       442416 K    1024

• Sometimes Percpu in /proc/meminfo increase is slightly in observation.
• Reading the /sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes_recursive file can take up to several minutes and cause soft lockups (and panic/crash when enabled). The same possibly also applies to files blkio.throttle.io_serviced_recursive, blkio.bfq.io_serviced_recursive and blkio.bfq.io_service_bytes_recursive.