RHEL7: While operating on a Veritas filesystem, kmalloc-1024 slab caches take all resources

Solution Verified - Updated -

Issue

  • When the customer does rman backups (on veritas filesystem) and some SQL queries (in databases ontop of veritas filesystem), The "kmalloc-1024" buffer (part of "cat /proc/slabinfo" output) grows until the system has no memory left. So under usage, the buffer constantly grows, and after some weeks memory is used up.

  • The issue is seen on multiple systems.

  • The kmalloc-1024 cache only seems to grow with pread and pwrite.

  • Memory is not shrinking when veritas filesystems are umounted.

  • Unloading the veritas modules is not possible

  • Dropping caches using /proc/sys/vm/drop_caches does not influence that memory.

  • "-/+ buffers/cache" in the "free" output is decreasing when the filesystem is used and kmalloc-1024 is growing.

  • Simpler tools like dd, cp, tar etc. were not able to reproduce the issue.

  • Normal tools like cp, tar etc. use read/write. Tools like rman use "pread" and "pwrite", these seem to cause the issue

Environment

  • Red Hat Enterprise Linux (RHEL) 7.0 and 7.1

  • VERITAS Storage Foundation HA 6.2.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content