Issue

• We've experienced a failure with one of our production machines. This machine was running RHEL6.3 with kernel 2.6.32-279.el6.x86_64. It has 48GB of RAM and 66GB of Swap.
• Initially we were running on kernel 2.6.32-279.el6 but updated to 2.6.32-358.6.2.el6 and still see the problem
• There were large number of processes running on the box and at some point in time we run out of both physical and virtual(swap) memory and oom killer kicked in.
• After that box was very slow with load around 100 and constant swapping.
• We've killed all the processes that were using significant amount of memory (>200MB) but this did not result in any significant release of memory. Even after killing most processes and switching to single user mode.
• slabtop showed that sizes size-192, size-256 are using about 20GB each.

slabtop | grep -n 20 

 Active / Total Objects (% used)    : 206100532 / 207069757 (99.5%)
 Active / Total Slabs (% used)      : 12058828 / 12058901 (100.0%)
 Active / Total Caches (% used)     : 119 / 195 (61.0%)
 Active / Total Size (% used)       : 45136325.42K / 45252935.30K (99.7%)
 Minimum / Average / Maximum Object : 0.02K / 0.22K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
102813400 102813219  16%    0.19K 5140670       20  20562680K size-192          
102810420 102810092  16%    0.25K 6854028       15  27416112K size-256