RHEL7: While operating on a Veritas filesystem, kmalloc-1024 slab caches take all resources

Solution Verified - Updated -

Issue

  • When the customer does rman backups (on veritas filesystem) and some SQL queries (in databases ontop of veritas filesystem), The "kmalloc-1024" buffer (part of "cat /proc/slabinfo" output) grows until the system has no memory left. So under usage, the buffer constantly grows, and after some weeks memory is used up.

  • The issue is seen on multiple systems.

  • The kmalloc-1024 cache only seems to grow with pread and pwrite.

  • Memory is not shrinking when veritas filesystems are umounted.

  • Unloading the veritas modules is not possible

  • Dropping caches using /proc/sys/vm/drop_caches does not influence that memory.

  • "-/+ buffers/cache" in the "free" output is decreasing when the filesystem is used and kmalloc-1024 is growing.

  • Simpler tools like dd, cp, tar etc. were not able to reproduce the issue.

  • Normal tools like cp, tar etc. use read/write. Tools like rman use "pread" and "pwrite", these seem to cause the issue

Environment

  • Red Hat Enterprise Linux (RHEL) 7.0 and 7.1

  • VERITAS Storage Foundation HA 6.2.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In