Application uses a lot of memory under RHEL 6

Issue

• Application uses lot of memory on rhel6.3 as well as 6.4. It works fine under 5.6. Problems with unacceptably large memory growth (both RSS and VSZ) . Is part of this problem is likely due to the new memory arena allocator including in glibc 2.10 and later referenced on sourceware.org and IBM developerworks

  • With the default transparent huge page(THP) support enabled, memory RSS would grow for the application [compared to a RHEL 5 build on a RH5 box of the same software].
  • Disabling transparent huge page support seemed to bring the memory back in line after setting the kernel command line option [appending transparent_hugepage=never] or dynamically setting echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
  • In some cases, even with THP [Anon Huge pages] disabled, the virtual memory would balloon [compared to a rhel5 build on a rhel5 box of the same software]
  • Enabling transparent hugepages and setting the arenas seems to have the virtual memory growth.
  • Setting MALLOC_ARENA_MAX environment variable to a 4 to start, we did some preliminary testing and it appeared to improve the virtual memory growth.[ +5 or 10% wouldn't be quite a concern, but 10 times as large getting into 250G
  • Virtual memory (VM) range causes some concern about the kernel's management of the Virtual memory (VM).

• While testing a Java app under RHEL 6 using jdk 7u5_x64, we ran into a situation where VM usage increased a lot. We did some research and based on HADOOP-7154 we set MALLOC_ARENA_MAX=4. That brought our VM usage back to levels we would expect.
  We read KB article 61778 which helped, but we're looking for more information about how an arena functions. We are currently testing value of 4, which seems to have desired effect, but we are not certain why? Some google articles note that value of 1 should not be used, but no clear explanation as to why.

• We are witnessing JBoss instances process size getting larger and larger (using a lot more physical heap than is configured). But this only seems to happen on RHEL servers. What could cause this and what is the best way to analyse something like this