ns-slapd died with an out of memory error message on a RHDS 9.1 installation

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux Server release 6.5 (Santiago)
  • Red Hat Directory Server 9.1

Issue

  • ns-slapd died with an out of memory error message.
  • Relevant information from /var/log/messages about the error:
Feb  6 19:06:26 rhdsprd0 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Feb  6 19:06:26 rhdsprd0 kernel: java cpuset=/ mems_allowed=0
Feb  6 19:06:26 rhdsprd0 kernel: Pid: 51324, comm: java Not tainted 2.6.32-431.3.1.el6.x86_64 #1
Feb  6 19:06:26 rhdsprd0 kernel: Call Trace:
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff810d05c1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81122970>] ? dump_header+0x90/0x1b0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8122799c>] ? security_real_capable_noaudit+0x3c/0x70
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81122df2>] ? oom_kill_process+0x82/0x2a0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81122d31>] ? select_bad_process+0xe1/0x120
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81123230>] ? out_of_memory+0x220/0x3c0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8112fb4c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81167aaa>] ? alloc_pages_current+0xaa/0x110
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8111fd67>] ? __page_cache_alloc+0x87/0x90
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8111f74e>] ? find_get_page+0x1e/0xa0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81120d07>] ? filemap_fault+0x1a7/0x500
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8114a094>] ? __do_fault+0x54/0x530
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8114a667>] ? handle_pte_fault+0xf7/0xb00
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff810af1de>] ? futex_wake+0x10e/0x120
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8114b29a>] ? handle_mm_fault+0x22a/0x300
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff81288e0c>] ? rb_erase+0x1bc/0x310
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8152d46e>] ? do_page_fault+0x3e/0xa0
Feb  6 19:06:26 rhdsprd0 kernel: [<ffffffff8152a825>] ? page_fault+0x25/0x30
Feb  6 19:06:26 rhdsprd0 kernel: Mem-Info:
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 DMA per-cpu:
Feb  6 19:06:26 rhdsprd0 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Feb  6 19:06:26 rhdsprd0 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 DMA32 per-cpu:
Feb  6 19:06:26 rhdsprd0 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Feb  6 19:06:26 rhdsprd0 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 Normal per-cpu:
Feb  6 19:06:26 rhdsprd0 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Feb  6 19:06:26 rhdsprd0 kernel: CPU    1: hi:  186, btch:  31 usd:  30
Feb  6 19:06:26 rhdsprd0 kernel: active_anon:1613532 inactive_anon:317634 isolated_anon:7
Feb  6 19:06:26 rhdsprd0 kernel: active_file:44 inactive_file:220 isolated_file:0
Feb  6 19:06:26 rhdsprd0 kernel: unevictable:0 dirty:0 writeback:4 unstable:0
Feb  6 19:06:26 rhdsprd0 kernel: free:25845 slab_reclaimable:3911 slab_unreclaimable:16068
Feb  6 19:06:26 rhdsprd0 kernel: mapped:386 shmem:26613 pagetables:19711 bounce:0
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 DMA free:15564kB min:120kB low:148kB high:180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15168kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb  6 19:06:26 rhdsprd0 kernel: lowmem_reserve[]: 0 3000 8106 8106
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 DMA32 free:45368kB min:24964kB low:31204kB high:37444kB active_anon:2175932kB inactive_anon:557424kB active_file:0kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:16kB mapped:312kB shmem:28528kB slab_reclaimable:2280kB slab_unreclaimable:2368kB kernel_stack:280kB pagetables:16632kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? yes
Feb  6 19:06:26 rhdsprd0 kernel: lowmem_reserve[]: 0 0 5106 5106
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 Normal free:42448kB min:42492kB low:53112kB high:63736kB active_anon:4278196kB inactive_anon:713112kB active_file:176kB inactive_file:864kB unevictable:0kB isolated(anon):28kB isolated(file):0kB present:5228544kB mlocked:0kB dirty:0kB writeback:0kB mapped:1232kB shmem:77924kB slab_reclaimable:13364kB slab_unreclaimable:61904kB kernel_stack:2744kB pagetables:62212kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? yes
Feb  6 19:06:26 rhdsprd0 kernel: lowmem_reserve[]: 0 0 0 0
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 DMA: 1*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15564kB
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 DMA32: 504*4kB 355*8kB 287*16kB 187*32kB 126*64kB 76*128kB 26*256kB 7*512kB 0*1024kB 1*2048kB 0*4096kB = 45512kB
Feb  6 19:06:26 rhdsprd0 kernel: Node 0 Normal: 1136*4kB 602*8kB 454*16kB 267*32kB 123*64kB 36*128kB 7*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 42512kB
Feb  6 19:06:26 rhdsprd0 kernel: 64692 total pagecache pages
Feb  6 19:06:26 rhdsprd0 kernel: 37775 pages in swap cache
Feb  6 19:06:26 rhdsprd0 kernel: Swap cache stats: add 2169440, delete 2131665, find 271783/313080
Feb  6 19:06:26 rhdsprd0 kernel: Free swap  = 0kB
Feb  6 19:06:26 rhdsprd0 kernel: Total swap = 2064376kB
Feb  6 19:06:26 rhdsprd0 kernel: 2097136 pages RAM
Feb  6 19:06:26 rhdsprd0 kernel: 69427 pages reserved
Feb  6 19:06:26 rhdsprd0 kernel: 1193 pages shared
Feb  6 19:06:26 rhdsprd0 kernel: 1982537 pages non-shared
Feb  6 19:06:26 rhdsprd0 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Feb  6 19:06:26 rhdsprd0 kernel: [  405]     0   405     2768        2   1     -17         -1000 udevd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1120]     0  1120    44803      121   1       0             0 vmtoolsd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1273]     0  1273    23294       28   0     -17         -1000 auditd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1291]     0  1291     1540        2   1       0             0 portreserve
Feb  6 19:06:26 rhdsprd0 kernel: [ 1298]     0  1298    62272      103   1       0             0 rsyslogd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1325]    32  1325     4744       32   1       0             0 rpcbind
Feb  6 19:06:26 rhdsprd0 kernel: [ 1339]     0  1339    49887       43   0       0             0 sssd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1340]     0  1340    57880      384   1       0             0 sssd_be
Feb  6 19:06:26 rhdsprd0 kernel: [ 1341]     0  1341    50597       86   0       0             0 sssd_nss
Feb  6 19:06:26 rhdsprd0 kernel: [ 1342]     0  1342    48157       66   0       0             0 sssd_pam
Feb  6 19:06:26 rhdsprd0 kernel: [ 1359]    29  1359     5837        2   1       0             0 rpc.statd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1599]     0  1599    36510       16   0       0             0 httpd.worker
Feb  6 19:06:26 rhdsprd0 kernel: [ 1602]     0  1602    36505        2   1       0             0 httpd.worker
Feb  6 19:06:26 rhdsprd0 kernel: [ 1603]    99  1603   451816       78   0       0             0 httpd.worker
Feb  6 19:06:26 rhdsprd0 kernel: [ 1699]    81  1699     8464       17   0       0             0 dbus-daemon
Feb  6 19:06:26 rhdsprd0 kernel: [ 1729]    68  1729     9955      128   0       0             0 hald
Feb  6 19:06:26 rhdsprd0 kernel: [ 1730]     0  1730     5081        2   0       0             0 hald-runner
Feb  6 19:06:26 rhdsprd0 kernel: [ 1759]     0  1759     5611        2   0       0             0 hald-addon-inpu
Feb  6 19:06:26 rhdsprd0 kernel: [ 1769]    68  1769     4483        2   0       0             0 hald-addon-acpi
Feb  6 19:06:26 rhdsprd0 kernel: [ 1807]     0  1807     1568        1   1       0             0 mcelog
Feb  6 19:06:26 rhdsprd0 kernel: [ 1821]     0  1821    16652       25   0     -17         -1000 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1831]     0  1831     5545        2   1       0             0 xinetd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1839]    38  1839     8200       33   0       0             0 ntpd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1917]     0  1917    20320       23   0       0             0 master
Feb  6 19:06:26 rhdsprd0 kernel: [ 1925]    89  1925    20903       22   0       0             0 qmgr
Feb  6 19:06:26 rhdsprd0 kernel: [ 1941]     0  1941    27580        2   0       0             0 abrtd
Feb  6 19:06:26 rhdsprd0 kernel: [ 1949]     0  1949    27053       17   1       0             0 abrt-dump-oops
Feb  6 19:06:26 rhdsprd0 kernel: [ 1984]     0  1984    51160      890   0       0             0 osad
Feb  6 19:06:26 rhdsprd0 kernel: [ 1992]     0  1992    29333       26   0       0             0 crond
Feb  6 19:06:26 rhdsprd0 kernel: [ 2003]     0  2003     5385        4   0       0             0 atd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2025]     0  2025   233982      151   0       0             0 nsrexecd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2086]     0  2086    25231        1   0       0             0 rhnsd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2094]     0  2094    26004       23   1       0             0 rhsmcertd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2107]     0  2107    15580       10   0       0             0 certmonger
Feb  6 19:06:26 rhdsprd0 kernel: [ 2130]     0  2130     1016        2   1       0             0 mingetty
Feb  6 19:06:26 rhdsprd0 kernel: [ 2133]     0  2133     1016        2   1       0             0 mingetty
Feb  6 19:06:26 rhdsprd0 kernel: [ 2135]     0  2135     1016        2   1       0             0 mingetty
Feb  6 19:06:26 rhdsprd0 kernel: [ 2138]     0  2138     1016        2   1       0             0 mingetty
Feb  6 19:06:26 rhdsprd0 kernel: [ 2144]     0  2144     1016        2   0       0             0 mingetty
Feb  6 19:06:26 rhdsprd0 kernel: [ 2148]     0  2148     1016        2   0       0             0 mingetty
Feb  6 19:06:26 rhdsprd0 kernel: [ 2375]     0  2375    27792       57   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2379] 253255  2379    27829      139   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2385] 253255  2385    29895        2   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [ 2756]     0  2756    27792       57   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2760] 14143  2760    27829      141   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [ 2767] 14143  2767    29936       20   1       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [12522]     0 12522    27792       56   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [12526] 14143 12526    28175       59   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [12527] 14143 12527    29936        2   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [12557]     0 12557    47849        2   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [12558]     0 12558    37420        2   0       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [12559]     0 12559    27076        9   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [20325]     0 20325   156418       65   0       0             0 automount
Feb  6 19:06:26 rhdsprd0 kernel: [21321] 253255 21321   887633    15834   0       0             0 java
Feb  6 19:06:26 rhdsprd0 kernel: [29812]     0 29812    27793       10   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [29816] 253255 29816    27901       41   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [29817] 253255 29817    29791        2   1       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [29854]     0 29854    47849        2   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [29856]     0 29856    37420        2   1       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [29857]     0 29857    27076        2   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [29933]     0 29933     6480       83   0       0             0 top
Feb  6 19:06:26 rhdsprd0 kernel: [34664]     0 34664    27792       74   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [34668] 253255 34668    27792      159   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [34677] 253255 34677    29792        2   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [34716]     0 34716    47849        2   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [34717]     0 34717    37420        2   1       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [34720]     0 34720    27076        2   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [37804]    99 37804  8422558  1808070   1       0             0 ns-slapd
Feb  6 19:06:26 rhdsprd0 kernel: [39331]     0 39331    27793      243   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [39336] 14143 39336    27793      238   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [39337] 14143 39337    29975      270   1       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [39364]     0 39364    47849      165   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [39365]     0 39365    37420      101   0       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [39366]     0 39366    29825       68   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [40303]     0 40303    25236       25   1       0             0 tail
Feb  6 19:06:26 rhdsprd0 kernel: [40501]     0 40501    27793      119   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [40505] 14143 40505    27941      291   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [40506] 14143 40506    29975      271   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [40534]     0 40534    47849      165   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [40535]     0 40535    37420      101   0       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [40536]     0 40536    29799       91   1       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [43532]     0 43532    25236        3   1       0             0 tail
Feb  6 19:06:26 rhdsprd0 kernel: [48440]     0 48440    27793       60   1       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [48449] 855487 48449    28198      133   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [48460] 855487 48460    29799       13   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [49354] 855487 49354   886038    20298   1       0             0 java
Feb  6 19:06:26 rhdsprd0 kernel: [49481]     0 49481    47852       16   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [49482]     0 49482    37420       15   1       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [49485]     0 49485    27101       84   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [50444]     0 50444    27793      245   1       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [50449] 855487 50449    27793      255   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [50462] 855487 50462    29799      102   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [50490]     0 50490    47852      160   1       0             0 sudo
Feb  6 19:06:26 rhdsprd0 kernel: [50491]     0 50491    37420      102   1       0             0 su
Feb  6 19:06:26 rhdsprd0 kernel: [50494]     0 50494    27102      112   1       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [51209]     0 51209    27792       67   0       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [51216] 14143 51216    27829      152   1       0             0 sshd
Feb  6 19:06:26 rhdsprd0 kernel: [51231] 14143 51231    29983        6   0       0             0 bash
Feb  6 19:06:26 rhdsprd0 kernel: [51262] 14143 51262   886871    16449   1       0             0 java
Feb  6 19:06:26 rhdsprd0 kernel: [55741]     0 55741     2767        9   1     -17         -1000 udevd
Feb  6 19:06:26 rhdsprd0 kernel: [55745]     0 55745     2767        6   0     -17         -1000 udevd
Feb  6 19:06:26 rhdsprd0 kernel: [57308]    89 57308    20860      221   0       0             0 pickup
Feb  6 19:06:26 rhdsprd0 kernel: Out of memory: Kill process 37804 (ns-slapd) score 899 or sacrifice child
Feb  6 19:06:26 rhdsprd0 kernel: Killed process 37804, UID 99, (ns-slapd) total-vm:33690232kB, anon-rss:7232208kB, file-rss:72kB

Resolution

On a server that is experiencing the memory usage growth, edit /etc/sysconfig/dirsrv- and add the following lines:

export SLAPD_MXFAST=0
export MALLOC_TRIM_THRESHOLD_=4069
export MALLOC_MMAP_THRESHOLD_=33554432
  • Then restart slapd.
#service dirsrv restart
  • This makes the memory fragmentation very, very small, at the expense of a slight performance hit, therefore this should be tried first on a staging or dev environment.

Root Cause

The memory growth is due to memory fragmentation.

Diagnostic Steps

Examine the directory server log, possible indications of this issue are queries that:
1. return large datasets
2. work with un-indexed data

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.