Chapter 9. Tuning SSSD performance for large IdM-AD trust deployments

Retrieving user and group information is a very data-intensive operation for the System Security Services Daemon (SSSD), especially in an IdM deployment with a trust to a large Active Directory (AD) domain. You can improve this performance by adjusting which information SSSD retrieves from identity providers and for how long.

9.1. Tuning SSSD in IdM servers for large IdM-AD trust deployments

This procedure applies tuning options to the configuration of the SSSD service in an IdM server to improve its response time when retrieving information from a large AD environment.

Prerequisites

  • You need root permissions to edit the /etc/sssd/sssd.conf configuration file.

Procedure

  1. Open the /etc/sssd/sssd.conf configuration file in a text editor.

  2. Add the following options to the [domain] section for your Active Directory domain. If you do not already have a domain section for your AD domain, create one. 

    [domain/ad.example.com]
ignore_group_members = true
subdomain_inherit = ignore_group_members
...
  3. Save and close the /etc/sssd/sssd.conf file on the server.

  4. Restart the SSSD service to load the configuration changes. 

    [root@client ~]# systemctl restart sssd

Additional resources

9.2. Tuning the config timeout for the ipa-extdom plugin on IdM servers

IdM clients cannot receive information about users and groups from Active Directory (AD) directly, so IdM servers use the ipa-extdom plugin to receive information about AD users and groups, and that information is forwarded to the requesting client.

The ipa-extdom plug-in sends a request to SSSD for the data about AD users. If the information is not in the SSSD cache, SSSD requests the data from an AD domain controller (DC). You can adjust the config timeout value, which defines how long the ipa-extdom plug-in waits for a reply from SSSD before the plug-in cancels the connection and returns a timeout error to the caller. The default value is 10000 milliseconds (10 seconds).

The following example adjusts the config timeout to 20 seconds (20000 milliseconds).

Warning

Exercise caution when adjusting the config timeout:

  • If you set a value that is too small, such as 500 milliseconds, SSSD might not have enough time to reply and requests will always return a timeout.
  • If you set a value that is too large, such as 30000 milliseconds (30 seconds), a single request might block the connection to SSSD for this amount of time. Because only one thread can connect to SSSD at a time, all other requests from the plug-in have to wait.
  • If there are many requests sent by IdM clients, they can block all available workers configured for the Directory Server on the IdM server. As a consequence, the server might not be able to reply to any kind of request for some time.

Only change the config timeout in the following situations:

  • If IdM clients frequently receive timeout errors before their own search timeout is reached when requesting information about AD users and groups, the config timeout value is too small.
  • If the Directory Server on the IdM server is often locked and the pstack utility reports that many or all worker threads are handling ipa-extdom requests at this time, the value is too large.

Prerequisites

  • The LDAP Directory Manager password

Procedure

  • Use the following command to adjust the config timeout to 20000 milliseconds: 

    # ldapmodify -D "cn=directory manager" -W
dn: cn=ipa_extdom_extop,cn=plugins,cn=config
changetype: modify
replace: ipaExtdomMaxNssTimeout
ipaExtdomMaxNssTimeout: 20000

9.3. Tuning the maximum buffer size for the ipa-extdom plugin on IdM servers

IdM clients cannot receive information about users and groups from Active Directory (AD) directly, so IdM servers use the ipa-extdom plugin to receive information about AD users and groups, and that information is forwarded to the requesting client.

You can tune the maximum buffer size for the ipa-extdom plugin, which adjusts the size of the buffer where SSSD can store the data it receives. If the buffer is too small, SSSD returns an ERANGE error and the plug-in retries the request with a larger buffer. The default buffer size is 134217728 bytes (128 MB).

The following example adjusts the maximum buffer size to 256 MB (268435456 bytes).

Prerequisites

  • The LDAP Directory Manager password

Procedure

  • Use the following command to set the maximum buffer size to 268435456 bytes: 

    # ldapmodify -D "cn=directory manager" -W
dn: cn=ipa_extdom_extop,cn=plugins,cn=config
changetype: modify
replace: ipaExtdomMaxNssBufSize
ipaExtdomMaxNssBufSize: 268435456

9.4. Tuning the maximum number of instances for the ipa-extdom plugin on IdM servers

As IdM clients cannot receive information about users and groups from Active Directory (AD) directly, IdM servers use the ipa-extdom plugin to receive information about AD users and groups and then forward this information to the requesting client.

By default, the ipa-extdom plugin is configured to use up to 80% of the LDAP worker threads to handle requests from IdM clients. If the SSSD service on an IdM client has requested a large amount of information about AD trust users and groups, this operation can halt the LDAP service if it uses most of the LDAP threads. If you experience these issues, you might see similar errors in the SSSD log file for your AD domain, /var/log/sssd/sssd__your-ad-domain-name.com_.log: 

(2022-05-22  5:00:13): [be[ad.example.com]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(2022-05-22  5:00:13): [be[ad.example.com]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(2022-05-22  5:00:13): [be[ad.example.com]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Server is busy(51), Too many extdom instances running.

You can adjust the maximum number of ipa-extdom instances by setting the value for the ipaExtdomMaxInstances option, which must be an integer larger than 0 and less than the total number of worker threads.

Prerequisites

  • The LDAP Directory Manager password

Procedure

  1. Retrieve the total number of worker threads: 

    # ldapsearch -xLLLD cn=directory\ manager -W -b cn=config -s base nsslapd-threadnumber
Enter LDAP Password:
dn: cn=config
nsslapd-threadnumber: 16

    This means that the current value for ipaExtdomMaxInstances is 13.

  2. Adjust the maximum number of instances. This example changes the value to 14: 

    # ldapmodify -D "cn=directory manager" -W
dn: cn=ipa_extdom_extop,cn=plugins,cn=config
changetype: modify
replace: ipaExtdomMaxInstances
ipaExtdomMaxInstances: 14

  3. Retrieve the current value of ipaExtdomMaxInstances: 

    # ldapsearch -xLLLD "cn=directory manager" -W -b "cn=ipa_extdom_extop,cn=plugins,cn=config" |grep ipaextdommaxinstances

Enter LDAP Password:

ipaextdommaxinstances: 14
  4. Monitor the IdM directory server’s performance and if it does not improve, repeat this procedure and adjust the value of the ipaExtdomMaxInstances variable.

9.5. Tuning SSSD in IdM clients for large IdM-AD trust deployments

This procedure applies tuning options to SSSD service configuration in an IdM client to improve its response time when retrieving information from a large AD environment.

Prerequisites

  • You need root permissions to edit the /etc/sssd/sssd.conf configuration file.

Procedure

  1. Determine the number of seconds a single un-cached login takes.

    1. Clear the SSSD cache on the IdM client client.example.com. 

      [root@client ~]# sss_cache -E

    2. Measure how long it takes to log in as an AD user with the time command. In this example, from the IdM client client.example.com, log into the same host as the user ad-user from the ad.example.com AD domain. 

      [root@client ~]# time ssh ad-user@ad.example.com@client.example.com

    3. Type in the password as soon as possible. 

      Password:
Last login: Sat Jan 23 06:29:54 2021 from 10.0.2.15
[ad-user@ad.example.com@client ~]$

    4. Log out as soon as possible to display elapsed time. In this example, a single un-cached login takes about 9 seconds. 

      [ad-user@ad.example.com@client /]$ exit
logout
Connection to client.example.com closed.

real 0m8.755s
user    0m0.017s
sys     0m0.013s
  2. Open the /etc/sssd/sssd.conf configuration file in a text editor.

  3. Add the following options to the [domain] section for your Active Directory domain. Set the pam_id_timeout and krb5_auth_timeout options to the number of seconds an un-cached login takes. If you do not already have a domain section for your AD domain, create one. 

    [domain/example.com/ad.example.com]
krb5_auth_timeout = 9
ldap_deref_threshold = 0
...

  4. Add the following option to the [pam] section: 

    [pam]
pam_id_timeout = 9
  5. Save and close the /etc/sssd/sssd.conf file on the server.

  6. Restart the SSSD service to load the configuration changes. 

    [root@client ~]# systemctl restart sssd

Additional resources

9.6. Mounting the SSSD cache in tmpfs

The System Security Services Daemon (SSSD) constantly writes LDAP objects to its cache. These internal SSSD transactions write data to disk, which is much slower than reading and writing from Random-Access Memory (RAM).

To improve this performance, mount the SSSD cache in RAM.

Considerations

  • Cached information does not persist after a reboot if the SSSD cache is in RAM.
  • It is safe to perform this change on IdM servers, as the SSSD instance on an IdM server cannot lose connectivity with the Directory Server on the same host.
  • If you perform this adjustment on an IdM client and it loses connectivity to IdM servers, users will not be able to authenticate after a reboot until you reestablish connectivity.

Prerequisites

  • You need root permissions to edit the /etc/fstab configuration file.

Procedure

  1. Create a tmpfs temporary filesystem:

    1. On RHEL 8.6 and later, confirm that the SSSD user owns the config.ldb file: 

      # ls -al /var/lib/sss/db/config.ldb
-rw-------. 1 sssd sssd 1286144 Jun  8 16:41 /var/lib/sss/db/config.ldb

      Add the following entry to the /etc/fstab file as a single line: 

      tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,uid=sssd,gid=sssd,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0

    2. On RHEL 8 versions lower than 8.6, the config.ldb file is owned by the root user: 

      # ls -al /var/lib/sss/db/config.ldb
-rw-------. 1 root root 1286144 Jun  8 14:15 /var/lib/sss/db/config.ldb

      Add the following entry to the /etc/fstab file as a single line: 

      tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0

      This example creates a 300MB cache. Tune the size parameter according to your IdM and AD directory size, estimating 100 MBs per 10,000 LDAP entries.

  2. Mount the new SSSD cache directory. 

    [root@host ~]# mount /var/lib/sss/db/

  3. Restart SSSD to reflect this configuration change. 

    [root@host ~]# systemctl restart sssd

9.7. Options in sssd.conf for tuning IdM servers and clients for large IdM-AD trust deployments

You can use the following options in the /etc/sssd/sssd.conf configuration file to tune the performance of SSSD in IdM servers and clients when you have a large IdM-AD trust deployment.

9.7.1. Tuning options for IdM servers

ignore_group_members

Knowing which groups a user belongs to, as opposed to all the users that belong to a group, is important when authenticating and authorizing a user. When ignore_group_members is set to true, SSSD only retrieves information about the group objects themselves and not their members, providing a significant performance boost.

Note

The id user@ad-domain.com command still returns the correct list of groups, but getent group ad-group@ad-domain.com returns an empty list.

Default value

false

Recommended value

true

Note

You should not set this option to true when the deployment involves an IdM server with the compat tree.

subdomain_inherit

With the subdomain_inherit option, you can apply the ignore_group_members setting to the trusted AD domains’ configuration. Settings listed in the subdomain_inherit options apply to both the main (IdM) domain as well as the AD subdomain.

Default value

none

Recommended value

subdomain_inherit = ignore_group_members

9.7.2. Tuning options for IdM clients

pam_id_timeout

This parameter controls how long results from a PAM session are cached, to avoid excessive round-trips to the identity provider during an identity lookup. The default value of 5 seconds might not be enough in environments where complex group memberships are populated on the IdM Server and IdM client side. Red Hat recommends setting pam_id_timeout to the number of seconds a single un-cached login takes.

Default value

5

Recommended value

the number of seconds a single un-cached login takes

krb5_auth_timeout

Increasing krb5_auth_timeout allows more time to process complex group information in environments where users are members of a large number of groups. Red Hat recommends setting this value to the number of seconds a single un-cached login takes.

Default value

6

Recommended value

the number of seconds a single un-cached login takes

ldap_deref_threshold

A dereference lookup is a means of fetching all group members in a single LDAP call. The ldap_deref_threshold value specifies the number of group members that must be missing from the internal cache to trigger a dereference lookup. If less members are missing, they are looked up individually. Dereference lookups may take a long time in large environments and decrease performance. To disable dereference lookups, set this option to 0.

Default value

10

Recommended value

0

9.8. Additional resources