ns-slapd died with an out of memory error message on a RHDS 9.1 installation
Environment
- Red Hat Enterprise Linux Server release 6.5 (Santiago)
- Red Hat Directory Server 9.1
Issue
- ns-slapd died with an out of memory error message.
- Relevant information from /var/log/messages about the error:
Feb 6 19:06:26 rhdsprd0 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Feb 6 19:06:26 rhdsprd0 kernel: java cpuset=/ mems_allowed=0
Feb 6 19:06:26 rhdsprd0 kernel: Pid: 51324, comm: java Not tainted 2.6.32-431.3.1.el6.x86_64 #1
Feb 6 19:06:26 rhdsprd0 kernel: Call Trace:
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff810d05c1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81122970>] ? dump_header+0x90/0x1b0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8122799c>] ? security_real_capable_noaudit+0x3c/0x70
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81122df2>] ? oom_kill_process+0x82/0x2a0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81122d31>] ? select_bad_process+0xe1/0x120
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81123230>] ? out_of_memory+0x220/0x3c0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8112fb4c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81167aaa>] ? alloc_pages_current+0xaa/0x110
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8111fd67>] ? __page_cache_alloc+0x87/0x90
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8111f74e>] ? find_get_page+0x1e/0xa0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81120d07>] ? filemap_fault+0x1a7/0x500
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8114a094>] ? __do_fault+0x54/0x530
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8114a667>] ? handle_pte_fault+0xf7/0xb00
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff810af1de>] ? futex_wake+0x10e/0x120
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8114b29a>] ? handle_mm_fault+0x22a/0x300
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff81288e0c>] ? rb_erase+0x1bc/0x310
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8152d46e>] ? do_page_fault+0x3e/0xa0
Feb 6 19:06:26 rhdsprd0 kernel: [<ffffffff8152a825>] ? page_fault+0x25/0x30
Feb 6 19:06:26 rhdsprd0 kernel: Mem-Info:
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 DMA per-cpu:
Feb 6 19:06:26 rhdsprd0 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Feb 6 19:06:26 rhdsprd0 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 DMA32 per-cpu:
Feb 6 19:06:26 rhdsprd0 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Feb 6 19:06:26 rhdsprd0 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 Normal per-cpu:
Feb 6 19:06:26 rhdsprd0 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Feb 6 19:06:26 rhdsprd0 kernel: CPU 1: hi: 186, btch: 31 usd: 30
Feb 6 19:06:26 rhdsprd0 kernel: active_anon:1613532 inactive_anon:317634 isolated_anon:7
Feb 6 19:06:26 rhdsprd0 kernel: active_file:44 inactive_file:220 isolated_file:0
Feb 6 19:06:26 rhdsprd0 kernel: unevictable:0 dirty:0 writeback:4 unstable:0
Feb 6 19:06:26 rhdsprd0 kernel: free:25845 slab_reclaimable:3911 slab_unreclaimable:16068
Feb 6 19:06:26 rhdsprd0 kernel: mapped:386 shmem:26613 pagetables:19711 bounce:0
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 DMA free:15564kB min:120kB low:148kB high:180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15168kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb 6 19:06:26 rhdsprd0 kernel: lowmem_reserve[]: 0 3000 8106 8106
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 DMA32 free:45368kB min:24964kB low:31204kB high:37444kB active_anon:2175932kB inactive_anon:557424kB active_file:0kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:16kB mapped:312kB shmem:28528kB slab_reclaimable:2280kB slab_unreclaimable:2368kB kernel_stack:280kB pagetables:16632kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? yes
Feb 6 19:06:26 rhdsprd0 kernel: lowmem_reserve[]: 0 0 5106 5106
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 Normal free:42448kB min:42492kB low:53112kB high:63736kB active_anon:4278196kB inactive_anon:713112kB active_file:176kB inactive_file:864kB unevictable:0kB isolated(anon):28kB isolated(file):0kB present:5228544kB mlocked:0kB dirty:0kB writeback:0kB mapped:1232kB shmem:77924kB slab_reclaimable:13364kB slab_unreclaimable:61904kB kernel_stack:2744kB pagetables:62212kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? yes
Feb 6 19:06:26 rhdsprd0 kernel: lowmem_reserve[]: 0 0 0 0
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 DMA: 1*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15564kB
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 DMA32: 504*4kB 355*8kB 287*16kB 187*32kB 126*64kB 76*128kB 26*256kB 7*512kB 0*1024kB 1*2048kB 0*4096kB = 45512kB
Feb 6 19:06:26 rhdsprd0 kernel: Node 0 Normal: 1136*4kB 602*8kB 454*16kB 267*32kB 123*64kB 36*128kB 7*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 42512kB
Feb 6 19:06:26 rhdsprd0 kernel: 64692 total pagecache pages
Feb 6 19:06:26 rhdsprd0 kernel: 37775 pages in swap cache
Feb 6 19:06:26 rhdsprd0 kernel: Swap cache stats: add 2169440, delete 2131665, find 271783/313080
Feb 6 19:06:26 rhdsprd0 kernel: Free swap = 0kB
Feb 6 19:06:26 rhdsprd0 kernel: Total swap = 2064376kB
Feb 6 19:06:26 rhdsprd0 kernel: 2097136 pages RAM
Feb 6 19:06:26 rhdsprd0 kernel: 69427 pages reserved
Feb 6 19:06:26 rhdsprd0 kernel: 1193 pages shared
Feb 6 19:06:26 rhdsprd0 kernel: 1982537 pages non-shared
Feb 6 19:06:26 rhdsprd0 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Feb 6 19:06:26 rhdsprd0 kernel: [ 405] 0 405 2768 2 1 -17 -1000 udevd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1120] 0 1120 44803 121 1 0 0 vmtoolsd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1273] 0 1273 23294 28 0 -17 -1000 auditd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1291] 0 1291 1540 2 1 0 0 portreserve
Feb 6 19:06:26 rhdsprd0 kernel: [ 1298] 0 1298 62272 103 1 0 0 rsyslogd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1325] 32 1325 4744 32 1 0 0 rpcbind
Feb 6 19:06:26 rhdsprd0 kernel: [ 1339] 0 1339 49887 43 0 0 0 sssd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1340] 0 1340 57880 384 1 0 0 sssd_be
Feb 6 19:06:26 rhdsprd0 kernel: [ 1341] 0 1341 50597 86 0 0 0 sssd_nss
Feb 6 19:06:26 rhdsprd0 kernel: [ 1342] 0 1342 48157 66 0 0 0 sssd_pam
Feb 6 19:06:26 rhdsprd0 kernel: [ 1359] 29 1359 5837 2 1 0 0 rpc.statd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1599] 0 1599 36510 16 0 0 0 httpd.worker
Feb 6 19:06:26 rhdsprd0 kernel: [ 1602] 0 1602 36505 2 1 0 0 httpd.worker
Feb 6 19:06:26 rhdsprd0 kernel: [ 1603] 99 1603 451816 78 0 0 0 httpd.worker
Feb 6 19:06:26 rhdsprd0 kernel: [ 1699] 81 1699 8464 17 0 0 0 dbus-daemon
Feb 6 19:06:26 rhdsprd0 kernel: [ 1729] 68 1729 9955 128 0 0 0 hald
Feb 6 19:06:26 rhdsprd0 kernel: [ 1730] 0 1730 5081 2 0 0 0 hald-runner
Feb 6 19:06:26 rhdsprd0 kernel: [ 1759] 0 1759 5611 2 0 0 0 hald-addon-inpu
Feb 6 19:06:26 rhdsprd0 kernel: [ 1769] 68 1769 4483 2 0 0 0 hald-addon-acpi
Feb 6 19:06:26 rhdsprd0 kernel: [ 1807] 0 1807 1568 1 1 0 0 mcelog
Feb 6 19:06:26 rhdsprd0 kernel: [ 1821] 0 1821 16652 25 0 -17 -1000 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1831] 0 1831 5545 2 1 0 0 xinetd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1839] 38 1839 8200 33 0 0 0 ntpd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1917] 0 1917 20320 23 0 0 0 master
Feb 6 19:06:26 rhdsprd0 kernel: [ 1925] 89 1925 20903 22 0 0 0 qmgr
Feb 6 19:06:26 rhdsprd0 kernel: [ 1941] 0 1941 27580 2 0 0 0 abrtd
Feb 6 19:06:26 rhdsprd0 kernel: [ 1949] 0 1949 27053 17 1 0 0 abrt-dump-oops
Feb 6 19:06:26 rhdsprd0 kernel: [ 1984] 0 1984 51160 890 0 0 0 osad
Feb 6 19:06:26 rhdsprd0 kernel: [ 1992] 0 1992 29333 26 0 0 0 crond
Feb 6 19:06:26 rhdsprd0 kernel: [ 2003] 0 2003 5385 4 0 0 0 atd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2025] 0 2025 233982 151 0 0 0 nsrexecd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2086] 0 2086 25231 1 0 0 0 rhnsd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2094] 0 2094 26004 23 1 0 0 rhsmcertd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2107] 0 2107 15580 10 0 0 0 certmonger
Feb 6 19:06:26 rhdsprd0 kernel: [ 2130] 0 2130 1016 2 1 0 0 mingetty
Feb 6 19:06:26 rhdsprd0 kernel: [ 2133] 0 2133 1016 2 1 0 0 mingetty
Feb 6 19:06:26 rhdsprd0 kernel: [ 2135] 0 2135 1016 2 1 0 0 mingetty
Feb 6 19:06:26 rhdsprd0 kernel: [ 2138] 0 2138 1016 2 1 0 0 mingetty
Feb 6 19:06:26 rhdsprd0 kernel: [ 2144] 0 2144 1016 2 0 0 0 mingetty
Feb 6 19:06:26 rhdsprd0 kernel: [ 2148] 0 2148 1016 2 0 0 0 mingetty
Feb 6 19:06:26 rhdsprd0 kernel: [ 2375] 0 2375 27792 57 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2379] 253255 2379 27829 139 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2385] 253255 2385 29895 2 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [ 2756] 0 2756 27792 57 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2760] 14143 2760 27829 141 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [ 2767] 14143 2767 29936 20 1 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [12522] 0 12522 27792 56 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [12526] 14143 12526 28175 59 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [12527] 14143 12527 29936 2 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [12557] 0 12557 47849 2 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [12558] 0 12558 37420 2 0 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [12559] 0 12559 27076 9 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [20325] 0 20325 156418 65 0 0 0 automount
Feb 6 19:06:26 rhdsprd0 kernel: [21321] 253255 21321 887633 15834 0 0 0 java
Feb 6 19:06:26 rhdsprd0 kernel: [29812] 0 29812 27793 10 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [29816] 253255 29816 27901 41 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [29817] 253255 29817 29791 2 1 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [29854] 0 29854 47849 2 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [29856] 0 29856 37420 2 1 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [29857] 0 29857 27076 2 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [29933] 0 29933 6480 83 0 0 0 top
Feb 6 19:06:26 rhdsprd0 kernel: [34664] 0 34664 27792 74 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [34668] 253255 34668 27792 159 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [34677] 253255 34677 29792 2 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [34716] 0 34716 47849 2 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [34717] 0 34717 37420 2 1 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [34720] 0 34720 27076 2 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [37804] 99 37804 8422558 1808070 1 0 0 ns-slapd
Feb 6 19:06:26 rhdsprd0 kernel: [39331] 0 39331 27793 243 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [39336] 14143 39336 27793 238 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [39337] 14143 39337 29975 270 1 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [39364] 0 39364 47849 165 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [39365] 0 39365 37420 101 0 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [39366] 0 39366 29825 68 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [40303] 0 40303 25236 25 1 0 0 tail
Feb 6 19:06:26 rhdsprd0 kernel: [40501] 0 40501 27793 119 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [40505] 14143 40505 27941 291 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [40506] 14143 40506 29975 271 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [40534] 0 40534 47849 165 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [40535] 0 40535 37420 101 0 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [40536] 0 40536 29799 91 1 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [43532] 0 43532 25236 3 1 0 0 tail
Feb 6 19:06:26 rhdsprd0 kernel: [48440] 0 48440 27793 60 1 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [48449] 855487 48449 28198 133 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [48460] 855487 48460 29799 13 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [49354] 855487 49354 886038 20298 1 0 0 java
Feb 6 19:06:26 rhdsprd0 kernel: [49481] 0 49481 47852 16 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [49482] 0 49482 37420 15 1 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [49485] 0 49485 27101 84 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [50444] 0 50444 27793 245 1 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [50449] 855487 50449 27793 255 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [50462] 855487 50462 29799 102 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [50490] 0 50490 47852 160 1 0 0 sudo
Feb 6 19:06:26 rhdsprd0 kernel: [50491] 0 50491 37420 102 1 0 0 su
Feb 6 19:06:26 rhdsprd0 kernel: [50494] 0 50494 27102 112 1 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [51209] 0 51209 27792 67 0 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [51216] 14143 51216 27829 152 1 0 0 sshd
Feb 6 19:06:26 rhdsprd0 kernel: [51231] 14143 51231 29983 6 0 0 0 bash
Feb 6 19:06:26 rhdsprd0 kernel: [51262] 14143 51262 886871 16449 1 0 0 java
Feb 6 19:06:26 rhdsprd0 kernel: [55741] 0 55741 2767 9 1 -17 -1000 udevd
Feb 6 19:06:26 rhdsprd0 kernel: [55745] 0 55745 2767 6 0 -17 -1000 udevd
Feb 6 19:06:26 rhdsprd0 kernel: [57308] 89 57308 20860 221 0 0 0 pickup
Feb 6 19:06:26 rhdsprd0 kernel: Out of memory: Kill process 37804 (ns-slapd) score 899 or sacrifice child
Feb 6 19:06:26 rhdsprd0 kernel: Killed process 37804, UID 99, (ns-slapd) total-vm:33690232kB, anon-rss:7232208kB, file-rss:72kB
Resolution
On a server that is experiencing the memory usage growth, edit /etc/sysconfig/dirsrv-
export SLAPD_MXFAST=0
export MALLOC_TRIM_THRESHOLD_=4069
export MALLOC_MMAP_THRESHOLD_=33554432
- Then restart slapd.
#service dirsrv restart
- This makes the memory fragmentation very, very small, at the expense of a slight performance hit, therefore this should be tried first on a staging or dev environment.
Root Cause
The memory growth is due to memory fragmentation.
Diagnostic Steps
Examine the directory server log, possible indications of this issue are queries that:
1. return large datasets
2. work with un-indexed data
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments