- We've experienced a failure with one of our production machines. This machine was running RHEL6.3 with kernel 2.6.32-279.el6.x86_64. It has 48GB of RAM and 66GB of Swap.
- Initially we were running on kernel 2.6.32-279.el6 but updated to 2.6.32-358.6.2.el6 and still see the problem
- There were large number of processes running on the box and at some point in time we run out of both physical and virtual(swap) memory and oom killer kicked in.
- After that box was very slow with load around 100 and constant swapping.
- We've killed all the processes that were using significant amount of memory (>200MB) but this did not result in any significant release of memory. Even after killing most processes and switching to single user mode.
- slabtop showed that sizes size-192, size-256 are using about 20GB each.
slabtop | grep -n 20 Active / Total Objects (% used) : 206100532 / 207069757 (99.5%) Active / Total Slabs (% used) : 12058828 / 12058901 (100.0%) Active / Total Caches (% used) : 119 / 195 (61.0%) Active / Total Size (% used) : 45136325.42K / 45252935.30K (99.7%) Minimum / Average / Maximum Object : 0.02K / 0.22K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 102813400 102813219 16% 0.19K 5140670 20 20562680K size-192 102810420 102810092 16% 0.25K 6854028 15 27416112K size-256
- Red Hat Enterprise Linux 6
- kernels from at least 2.6.32-279.el6 and less than 2.6.32-358.23.2.el6
- NFSv3 Client
- NFS Server: NetApp storage with NIS to do the automount
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.