Red Hat Directory Server 8 and high excessive cpu usage, what can I do?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Directory Server 8
    redhat-ds-8.1.0-1
    redhat-ds-base-8.1.0-0.14
    

Issue

What can be done when Red Hat Directory Server (RHDS) version 8 seem to have several high cpu spikes a day, and/or more than 1 hour, too long high and excessive cpu usage?
This may affect application performances.

Resolution

Do not set nsslapd-lookthroughlimit to -1
General tips: refine tuning configuration parameters depending on application environment like , cache settings, and:
nsslapd-sizelimit under cn=config
nsslapd-lookthroughlimit under cn=config,cn=ldbm database,cn=plugins,cn=config
nsslapd-idlistscanlimit  under cn=config,cn=ldbm database,cn=plugins,cn=config
review RHDS logs, system monitoring logs, refine tuning configuration parameters depending on application environment, revert prior configuration changes.

Root Cause

One example is when nsslapd-lookthroughlimit is set to a value of -1.
Changing to a value of a few thousands fixed the reported problem, and depends on the traffic served, this setting limits the amount of data ns-slapd will look for into the db.
Adding some indexes also helped.
But there are many possible reasons on why RHDS's ns-slapd process cpu usage can be excessive in a long period of time on a system, here are some possible causes:

  • unforseen side effects of recent prior ns-slapd or system configuration changes
  • unforseen extended LDAP traffic load spike from applications for existing RHDS infrastructure sizing and ns-slapd tuning parameters
  • unforseen change and evolution of the data set, more entries possibly affecting limit settings, more or larger attribute values possibly affecting cache settings
  • many newer unindexed searches from applications
  • many attributes or many newer wild chars searches from applications
  • more TCP traffic, and possible latency snow ball effects

Diagnostic Steps

General tip:
Review and list possible recent changes, or changes made after the applications were working fine, like for example maximum TCP connections, file descriptors, binds per connections, "look through limit", memcache.
Try to note cpu hike time stamps iand durations to match in RHDS logs
Then collect some environment details, such as:

cat /etc/redhat-release; uname -a; rpm -q redhat-ds-base
/var/log/messages*
/var/log/sa/*
/var/log/dirsrv/slapd-*/access*
/var/log/dirsrv/slapd-*/errors*

or a sosreport
For example:

df -h > /tmp/fd-h.out
vmstat 1 40 > /tmp/vmstat.out
ps auxwwH > /tmp/process.out
io wait, mem, cpu swap - top -d 1 -b -n 40 > /tmp/top.out
page faults - sar -A 1 40 > /tmp/sar.out
iostat -x 1 40 > /tmp/iostat.out
lsof > /tmp/lsof.out
tar cvjf dirsrv.tar.bz2 /tmp/*.out

Eexample of excessive cpu use from ns-slapd:

PID  USER  PR  NI VIRT  RES   SHR S %CPU %MEM   TIME+     COMMAND
1234 rhds  15   0 1092m 623m  21m S 98.2 12.6   240:30.40 ns-slapd
1234 rhds  15   0 1092m 623m  21m S 99.7 12.6   240:31.40 ns-slapd
1234 rhds  15   0 1092m 623m  21m S 99.5 12.6   240:32.40 ns-slapd

The utility RHDS log analysis tool /usr/bin/logconv.pl from the rpm redhat-ds-base also provides with valuable information, it is recommended to run it on a separate system when many log entries are parsed.

/usr/bin/logconv.pl -h
/usr/bin/logconv.pl -V access > access.logconv.txt

Review the output for too long etime, connection latency, unindexed searches, search filters, most requested attributes, error codes like err=4 Size Limit Exceeded or err=10 Unwilling To Perform, file descriptor usage, peak connection, add/del/mod/srch operations.
Review tuning of nsslapd-lookthroughlimit, nsslapd-sizelimit, and test, example:

nsslapd-lookthroughlimit: 5000
nsslapd-sizelimit: 2000

Note: nsslapd-sizelimit can be overwritten with the optin -z from the openldap-clients's ldapsearch client.

nsslapd-sizelimit which specifies the maximum number of entries to return from a search operation.
If this limit is reached, the directory returns any entries it has located that match the search request, as well as an exceeded size limit error.

nsslapd-lookthroughlimit which specifies the maximum number of entries that the directory will check when examining candidate entries in response to a search request.

See http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/User_Account_Management-Setting_Resource_Limits_Based_on_the_Bind_DN.html

Tune nsslapd-idlistscanlimit used by the search algorithm, it can make a search treated as an unindexed search.
The results of an unindexed search are not cached as they could potentially be the size of the whole directory.
See http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/Managing_Indexes.html

Comments

On line documentation for the logconv.pl utility:

http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.1/html/Configuration_and_Command_Reference/Configuration_Command_File_Reference-Command_Line_Scripts-Perl_Scripts.html#Configuration_Command_File_Reference-ldif2db.pl_Import-logconv.pl_Log_converter
http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/Managing_Indexes.html#Managing_Indexes-Managing_Indexes

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.