RHEL7: While operating on a Veritas filesystem, kmalloc-1024 slab caches take all resources
Issue
-
When the customer does rman backups (on veritas filesystem) and some SQL queries (in databases ontop of veritas filesystem), The "kmalloc-1024" buffer (part of "cat /proc/slabinfo" output) grows until the system has no memory left. So under usage, the buffer constantly grows, and after some weeks memory is used up.
-
The issue is seen on multiple systems.
-
The kmalloc-1024 cache only seems to grow with pread and pwrite.
-
Memory is not shrinking when veritas filesystems are umounted.
-
Unloading the veritas modules is not possible
-
Dropping caches using /proc/sys/vm/drop_caches does not influence that memory.
-
"-/+ buffers/cache" in the "free" output is decreasing when the filesystem is used and kmalloc-1024 is growing.
-
Simpler tools like dd, cp, tar etc. were not able to reproduce the issue.
-
Normal tools like cp, tar etc. use read/write. Tools like rman use "pread" and "pwrite", these seem to cause the issue
Environment
-
Red Hat Enterprise Linux (RHEL) 7.0 and 7.1
-
VERITAS Storage Foundation HA 6.2.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.