- Red Hat Enterprise Linux 5
- Red Hat Directory Server 8
What can be done when Red Hat Directory Server (RHDS) version 8 seem to have several high cpu spikes a day, and/or more than 1 hour, too long high and excessive cpu usage?
This may affect application performances.
Do not set nsslapd-lookthroughlimit to -1
General tips: refine tuning configuration parameters depending on application environment like , cache settings, and:
nsslapd-sizelimit under cn=config
nsslapd-lookthroughlimit under cn=config,cn=ldbm database,cn=plugins,cn=config
nsslapd-idlistscanlimit under cn=config,cn=ldbm database,cn=plugins,cn=config
review RHDS logs, system monitoring logs, refine tuning configuration parameters depending on application environment, revert prior configuration changes.
One example is when nsslapd-lookthroughlimit is set to a value of -1.
Changing to a value of a few thousands fixed the reported problem, and depends on the traffic served, this setting limits the amount of data ns-slapd will look for into the db.
Adding some indexes also helped.
But there are many possible reasons on why RHDS's ns-slapd process cpu usage can be excessive in a long period of time on a system, here are some possible causes:
- unforseen side effects of recent prior ns-slapd or system configuration changes
- unforseen extended LDAP traffic load spike from applications for existing RHDS infrastructure sizing and ns-slapd tuning parameters
- unforseen change and evolution of the data set, more entries possibly affecting limit settings, more or larger attribute values possibly affecting cache settings
- many newer unindexed searches from applications
- many attributes or many newer wild chars searches from applications
- more TCP traffic, and possible latency snow ball effects
Review and list possible recent changes, or changes made after the applications were working fine, like for example maximum TCP connections, file descriptors, binds per connections, "look through limit", memcache.
Try to note cpu hike time stamps iand durations to match in RHDS logs
Then collect some environment details, such as:
cat /etc/redhat-release; uname -a; rpm -q redhat-ds-base /var/log/messages* /var/log/sa/* /var/log/dirsrv/slapd-*/access* /var/log/dirsrv/slapd-*/errors*
or a sosreport
df -h > /tmp/fd-h.out vmstat 1 40 > /tmp/vmstat.out ps auxwwH > /tmp/process.out io wait, mem, cpu swap - top -d 1 -b -n 40 > /tmp/top.out page faults - sar -A 1 40 > /tmp/sar.out iostat -x 1 40 > /tmp/iostat.out lsof > /tmp/lsof.out tar cvjf dirsrv.tar.bz2 /tmp/*.out
Eexample of excessive cpu use from ns-slapd:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1234 rhds 15 0 1092m 623m 21m S 98.2 12.6 240:30.40 ns-slapd 1234 rhds 15 0 1092m 623m 21m S 99.7 12.6 240:31.40 ns-slapd 1234 rhds 15 0 1092m 623m 21m S 99.5 12.6 240:32.40 ns-slapd
The utility RHDS log analysis tool /usr/bin/logconv.pl from the rpm redhat-ds-base also provides with valuable information, it is recommended to run it on a separate system when many log entries are parsed.
/usr/bin/logconv.pl -h /usr/bin/logconv.pl -V access > access.logconv.txt
Review the output for too long etime, connection latency, unindexed searches, search filters, most requested attributes, error codes like err=4 Size Limit Exceeded or err=10 Unwilling To Perform, file descriptor usage, peak connection, add/del/mod/srch operations.
Review tuning of nsslapd-lookthroughlimit, nsslapd-sizelimit, and test, example:
nsslapd-lookthroughlimit: 5000 nsslapd-sizelimit: 2000
Note: nsslapd-sizelimit can be overwritten with the optin -z from the openldap-clients's ldapsearch client.
nsslapd-sizelimit which specifies the maximum number of entries to return from a search operation.
If this limit is reached, the directory returns any entries it has located that match the search request, as well as an exceeded size limit error.
nsslapd-lookthroughlimit which specifies the maximum number of entries that the directory will check when examining candidate entries in response to a search request.
Tune nsslapd-idlistscanlimit used by the search algorithm, it can make a search treated as an unindexed search.
The results of an unindexed search are not cached as they could potentially be the size of the whole directory.
On line documentation for the logconv.pl utility:
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.