Chapter 11. Troubleshooting authentication with SSSD in IdM

Authentication in an Identity Management (IdM) environment involves many components:

On the IdM client:

  • The SSSD service.
  • The Name Services Switch (NSS).
  • Pluggable Authentication Modules (PAM).

On the IdM server:

  • The SSSD service.
  • The IdM Directory Server.
  • The IdM Kerberos Key Distribution Center (KDC).

If you are authenticating as an Active Directory (AD) user:

  • The Directory Server on an AD Domain Controller.
  • The Kerberos server on an AD Domain Controller.

To authenticate users, you must be able to perform the following functions with the SSSD service:

  • Retrieve user information from the authentication server.
  • Prompt the user for their credentials, pass those credentials to the authentication server, and process the outcome.

The following sections discuss how information flows between the SSSD service and servers that store user information, so you can troubleshoot failing authentication attempts in your environment:

11.1. Data flow when retrieving IdM user information with SSSD

The following diagram is a simplification of the information flow between an IdM client and an IdM server during a request for IdM user information with the command getent passwd <idm_user_name>.

A diagram with numbered arrows representing the flow of information between an IdM client and an IdM server. The following numbered list describes each step in the process.

  1. The getent command triggers the getpwnam call from the libc library.
  2. The libc library references the /etc/nsswitch.conf configuration file to check which service is responsible for providing user information, and discovers the entry sss for the SSSD service.
  3. The libc library opens the nss_sss module.
  4. The nss_sss module checks the memory-mapped cache for the user information. If the data is present in the cache, the nss_sss module returns it.
  5. If the user information is not in the memory-mapped cache, the request is passed to the SSSD sssd_nss responder process.
  6. The SSSD service checks its cache. If the data is present in the cache and valid, the sssd_nss responder reads the data from the cache and returns it to the application.
  7. If the data is not present in the cache or it is expired, the sssd_nss responder queries the appropriate back-end process and waits for a reply. The SSSD service uses the IPA backend in an IdM environment, enabled by the setting id_provider=ipa in the sssd.conf configuration file.
  8. The sssd_be back-end process connects to the IdM server and requests the information from the IdM LDAP Directory Server.
  9. The SSSD back-end on the IdM server responds to the SSSD back-end process on the IdM client.
  10. The SSSD back-end on the client stores the resulting data in the SSSD cache and alerts the responder process that the cache has been updated.
  11. The sssd_nss front-end responder process retrieves the information from the SSSD cache.
  12. The sssd_nss responder sends the user information to the nss_sss responder, completing the request.
  13. The libc library returns the user information to the application that requested it.

11.2. Data flow when retrieving AD user information with SSSD

If you have established a cross-forest trust between your IdM environment and an Active Directory (AD) domain, the information flow when retrieving AD user information on an IdM client is very similar to the information flow when retrieving IdM user information, with the additional step of contacting the AD user database.

The following diagram is a simplification of the information flow when a user requests information about an AD user with the command getent passwd <ad_user_name@ad.example.com>. This diagram does not include the internal details discussed in the Data flow when retrieving IdM user information with SSSD section. It focuses on the communication between the SSSD service on an IdM client, the SSSD service on an IdM server, and the LDAP database on an AD Domain Controller.

A diagram with numbered arrows representing the flow of information between an IdM client, an IdM server, and an AD Domain Controller. The following numbered list describes each step in the process.

  1. The IdM client looks to its local SSSD cache for AD user information.
  2. If the IdM client does not have the user information, or the information is stale, the SSSD service on the client contacts the extdom_extop plugin on the IdM server to perform an LDAP extended operation and requests the information.
  3. The SSSD service on the IdM server looks for the AD user information in its local cache.
  4. If the IdM server does not have the user information in its SSSD cache, or its information is stale, it performs an LDAP search to request the user information from an AD Domain Controller.
  5. The SSSD service on the IdM server receives the AD user information from the AD domain controller and stores it in its cache.
  6. The extdom_extop plugin receives the information from the SSSD service on the IdM server, which completes the LDAP extended operation.
  7. The SSSD service on the IdM client receives the AD user information from the LDAP extended operation.
  8. The IdM client stores the AD user information in its SSSD cache and returns the information to the application that requested it.

11.3. Data flow when authenticating as a user with SSSD in IdM

Authenticating as a user on an IdM server or client involves the following components:

  • The service that initiates the authentication request, such as the sshd service.
  • The Pluggable Authentication Module (PAM) library and its modules.
  • The SSSD service, its responders, and back-ends.
  • A smart card reader, if smart card authentication is configured.
  • The authentication server:

    • IdM users are authenticated against an IdM Kerberos Key Distribution Center (KDC).
    • Active Directory (AD) users are authenticated against an AD Domain Controller (DC).

The following diagram is a simplification of the information flow when a user needs to authenticate during an attempt to log in locally to a host via the SSH service on the command line.

A diagram with numbered arrows representing the flow of information between an IdM client and an IdM server or AD Domain Controller during an authentication attempt. The following numbered list describes each step in the process.

  1. The authentication attempt with the ssh command triggers the libpam library.
  2. The libpam library references the PAM file in the /etc/pam.d/ directory that corresponds to the service requesting the authentication attempt. In this example involving authenticating via the SSH service on the local host, the libpam library checks the /etc/pam.d/system-auth configuration file and discovers the pam_sss.so entry for the SSSD PAM:

    auth    sufficient    pam_sss.so
  3. To determine which authentication methods are available, the libpam library opens the pam_sss module and sends an SSS_PAM_PREAUTH request to the sssd_pam PAM responder of the SSSD service.
  4. If smart card authentication is configured, the SSSD service spawns a temporary p11_child process to check for a smart card and retrieve certificates from it.
  5. If smart card authentication is configured for the user, the sssd_pam responder attempts to match the certificate from the smart card with the user. The sssd_pam responder also performs a search for the groups that the user belongs to, since group membership might affect access control.
  6. The sssd_pam responder sends an SSS_PAM_PREAUTH request to the sssd_be back-end responder to see which authentication methods the server supports, such as passwords or 2-factor authentication. In an IdM environment, where the SSSD service uses the IPA responder, the default authentication method is Kerberos. For this example, the user authenticates with a simple Kerberos password.
  7. The sssd_be responder spawns a temporary krb5_child process.
  8. The krb5_child process contacts the KDC on the IdM server and checks for available authentication methods.
  9. The KDC responds to the request:

    1. The krb5_child process evaluates the reply and sends the results back to the sssd_be backend process.
    2. The sssd_be backend process receives the result.
    3. The sssd_pam responder receives the result.
    4. The pam_sss module receives the result.
  10. If password authentication is configured for the user, the pam_sss module prompts the user for their password. If smart card authentication is configured, the pam_sss module prompts the user for their smart card PIN.
  11. The module sends an SSS_PAM_AUTHENTICATE request with the user name and password, which travels to:

    1. The sssd_pam responder.
    2. The sssd_be back-end process.
  12. The sssd_be process spawns a temporary krb5_child process to contact the KDC.
  13. The krb5_child process attempts to retrieve a Kerberos Ticket Granting Ticket (TGT) from the KDC with the user name and password the user provided.
  14. The krb5_child process receives the result of the authentication attempt.
  15. The krb5_child process:

    1. Stores the TGT in a credential cache.
    2. Returns the authentication result to the sssd_be back-end process.
  16. The authentication result travels from the sssd_be process to:

    1. The sssd_pam responder.
    2. The pam_sss module.
  17. The pam_sss module sets an environment variable with the location of the user’s TGT so other applications can reference it.

11.4. Narrowing the scope of authentication issues

To successfully authenticate a user, you must be able to retrieve user information with the SSSD service from the database that stores user information. The following procedure describes steps to test different components of the authentication process so you can narrow the scope of authentication issues when a user is unable to log in.

Procedure

  1. Verify that the SSSD service and its processes are running.

    [root@client ~]# pstree -a | grep sssd
      |-sssd -i --logger=files
      |   |-sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
      |   |-sssd_be --domain example.com --uid 0 --gid 0 --logger=files
      |   |-sssd_ifp --uid 0 --gid 0 --logger=files
      |   |-sssd_nss --uid 0 --gid 0 --logger=files
      |   |-sssd_pac --uid 0 --gid 0 --logger=files
      |   |-sssd_pam --uid 0 --gid 0 --logger=files
      |   |-sssd_ssh --uid 0 --gid 0 --logger=files
      |   `-sssd_sudo --uid 0 --gid 0 --logger=files
      |-sssd_kcm --uid 0 --gid 0 --logger=files
  2. Verify that the client can contact the user database server via the IP address.

    [user@client ~]$ ping <IP_address_of_the_database_server>

    If this step fails, check that your network and firewall settings allow direct communication between IdM clients and servers. See Using and configuring firewalld.

  3. Verify that the client can discover and contact the IdM LDAP server (for IdM users) or AD domain controller (for AD users) via the fully qualified host name.

    [user@client ~]$ dig -t SRV _ldap._tcp.example.com @<name_server>
    [user@client ~]$ ping <fully_qualified_host_name_of_the_server>

    If this step fails, check your Dynamic Name Service (DNS) settings, including the /etc/resolv.conf file. See Configuring the order of DNS servers.

    Note

    By default, the SSSD service attempts to automatically discover LDAP servers and AD DCs through DNS service (SRV) records. Alternatively, you can restrict the SSSD service to use specific servers by setting the following options in the sssd.conf configuration file:

    • ipa_server = <fully_qualified_host_name_of_the_server>
    • ad_server = <fully_qualified_host_name_of_the_server>
    • ldap_uri = <fully_qualified_host_name_of_the_server>

    If you use these options, verify you can contact the servers listed in them.

  4. Verify that the client can authenticate to the LDAP server and retrieve user information with ldapsearch commands.

    1. If your LDAP server is an IdM server, like server.example.com, retrieve a Kerberos ticket for the host and perform the database search authenticating with the host Kerberos principal:

      [user@client ~]$ kinit -t 'host/client.example.com@EXAMPLE.COM'
      [user@client ~]$ ldapsearch -LLL -Y GSSAPI -h server.example.com -b “dc=example,dc=com” uid=<user_name>
    2. If your LDAP server is an Active Directory (AD) Domain Controller (DC), like server.ad.example.com, retrieve a Kerberos ticket for the host and perform the database search authenticating with the host Kerberos principal:

      [user@client ~]$ kinit -t 'CLIENT$@AD.EXAMPLE.COM'
      [user@client ~]$ ldapsearch -LLL -Y GSSAPI -h server.ad.example.com -b “dc=example,dc=com” sAMAccountname=<user_name>
    3. If your LDAP server is a plain LDAP server, and you have set the ldap_default_bind_dn and ldap_default_authtok options in the sssd.conf file, authenticate as the same ldap_default_bind_dn account:

      [user@client ~]$ ldapsearch -xLLL -D "cn=ldap_default_bind_dn_value" -W -h ldapserver.example.com -b “dc=example,dc=com” uid=<user_name>

    If this step fails, verify that your database settings allow your host to search the LDAP server.

  5. Since the SSSD service uses Kerberos encryption, verify you can obtain a Kerberos ticket as the user that is unable to log in.

    1. If your LDAP server is an IdM server:

      [user@client ~]$ kinit <user_name>
    2. If LDAP server database is an AD server:

      [user@client ~]$ kinit <user_name@AD.EXAMPLE.COM>

    If this step fails, verify that your Kerberos server is operating properly, all servers have their times synchronized, and that the user account is not locked.

  6. Verify you can retrieve user information on the command line.

    [user@client ~]$ getent passwd <user_name>
    [user@client ~]$ id <user_name>

    If this step fails, verify that the SSSD service on the client can receive information from the user database:

    1. Review errors in the /var/log/messages log file.
    2. Enable detailed logging in the SSSD service, collect debugging logs, and review the logs for indications to the source of the issue.
    3. (Optional) Open a Red Hat Technical Support case and provide the troubleshooting information you have gathered.
  7. Use the sssctl utility to verify the user is allowed to log in.

    [user@client ~]$ sssctl user-checks -a auth -s ssh <user_name>

    If this step fails, verify your authorization settings, such as your PAM configuration, IdM HBAC rules, and IdM RBAC rules:

    1. Review authorization errors in the /var/log/secure and /var/log/messages log files.
    2. Enable detailed logging in the SSSD service, collect debugging logs, and review the logs for indications to the source of the issue.
    3. (Optional) Open a Red Hat Technical Support case and provide the troubleshooting information you have gathered.

11.5. SSSD log files and logging levels

Each SSSD service logs into its own log file in the /var/log/sssd/ directory. For an IdM server in the example.com IdM domain, its log files might look like this:

[root@server ~]# ls -l /var/log/sssd/
total 620
-rw-------.  1 root root      0 Mar 29 09:21 krb5_child.log
-rw-------.  1 root root  14324 Mar 29 09:50 ldap_child.log
-rw-------.  1 root root 212870 Mar 29 09:50 sssd_example.com.log
-rw-------.  1 root root      0 Mar 29 09:21 sssd_ifp.log
-rw-------.  1 root root      0 Mar 29 09:21 sssd_implicit_files.log
-rw-------.  1 root root      0 Mar 29 09:21 sssd.log
-rw-------.  1 root root 219873 Mar 29 10:03 sssd_nss.log
-rw-------.  1 root root      0 Mar 29 09:21 sssd_pac.log
-rw-------.  1 root root  13105 Mar 29 09:21 sssd_pam.log
-rw-------.  1 root root   9390 Mar 29 09:21 sssd_ssh.log
-rw-------.  1 root root      0 Mar 29 09:21 sssd_sudo.log

11.5.1. SSSD log file purposes

krb5_child.log
Log file for the short-lived helper process involved in Kerberos authentication.
ldap_child.log
Log file for the short-lived helper process involved in getting a Kerberos ticket for the communication with the LDAP server.
sssd_<example.com>.log

For each domain section in the sssd.conf file, the SSSD service logs information about communication with the LDAP server to a separate log file. For example, in an environment with an IdM domain named example.com, the SSSD service logs its information in a file named sssd_example.com.log. If a host is directly integrated with an AD domain named ad.example.com, information is logged to a file named sssd_ad.example.com.log.

Note

If you have an IdM environment and a cross-forest trust with an AD domain, information about the AD domain is still logged to the log file for the IdM domain.

Similarly, if a host is directly integrated to an AD domain, information about any child domains is written in the log file for the primary domain.

selinux_child.log
Log file for the short-lived helper process that retrieves and sets SELinux information.
sssd.log
Log file for SSSD monitoring and communicating with its responder and backend processes.
sssd_ifp.log
Log file for the InfoPipe responder, which provides a public D-Bus interface accessible over the system bus.
sssd_nss.log
Log file for the Name Services Switch (NSS) responder that retrieves user and group information.
sssd_pac.log
Log file for the Microsoft Privilege Attribute Certificate (PAC) responder, which collects the PAC from AD Kerberos tickets and derives information about AD users from the PAC, which avoids requesting it directly from AD.
sssd_pam.log
Log file for the Pluggable Authentication Module (PAM) responder.
sssd_ssh.log
Log file for the SSH responder process.

11.5.2. SSSD logging levels

Setting a debug level also enables all debug levels below it. For example, setting the debug level at 6 also enables debug levels 0 through 5.

Table 11.1. SSSD logging levels

LevelDescription

0

Fatal failures. Errors that prevent the SSSD service from starting up or cause it to terminate. This is the default debug log level for RHEL 8.3 and earlier.

1

Critical failures. Errors that do not terminate the SSSD service, but at least one major feature is not working properly.

2

Serious failures. Errors announcing that a particular request or operation has failed. This is the default debug log level for RHEL 8.4 and later.

3

Minor failures. Errors that cause the operation failures captured at level 2.

4

Configuration settings.

5

Function data.

6

Trace messages for operation functions.

7

Trace messages for internal control functions.

8

Contents of function-internal variables.

9

Extremely low-level tracing information.

11.6. Enabling detailed logging for SSSD in the sssd.conf file

By default, the SSSD service in RHEL 8.4 and later only logs serious failures (debug level 2), but it does not log at the level of detail necessary to troubleshoot authentication issues.

To enable detailed logging persistently across SSSD service restarts, add the option debug_level=<integer> in each section of the /etc/sssd/sssd.conf configuration file, where the <integer> value is a number between 0 and 9. Debug levels up to 3 log larger failures, and levels 8 and higher provide a large number of detailed log messages. Level 6 is a good starting point for debugging authentication issues.

Prerequisites

  • You need the root password to edit the sssd.conf configuration file and restart the SSSD service.

Procedure

  1. Open the /etc/sssd/sssd.conf file in a text editor.
  2. Add the debug_level option to every section of the file, and set the debug level to the verbosity of your choice.

    [domain/example.com]
    debug_level = 6
    id_provider = ipa
    ...
    
    [sssd]
    debug_level = 6
    services = nss, pam, ifp, ssh, sudo
    domains = example.com
    
    [nss]
    debug_level = 6
    
    [pam]
    debug_level = 6
    
    [sudo]
    debug_level = 6
    
    [ssh]
    debug_level = 6
    
    [pac]
    debug_level = 6
    
    [ifp]
    debug_level = 6
  3. Save and close the sssd.conf file.
  4. Restart the SSSD service to load the new configuration settings.

    [root@server ~]# systemctl restart sssd

Additional resources

11.7. Enabling detailed logging for SSSD with the sssctl command

By default, the SSSD service in RHEL 8.4 and later only logs serious failures (debug level 2), but it does not log at the level of detail necessary to troubleshoot authentication issues.

You can change the debug level of the SSSD service on the command line with the sssctl debug-level <integer> command, where the <integer> value is a number between 0 and 9. Debug levels up to 3 log larger failures, and levels 8 and higher provide a large number of detailed log messages. Level 6 is a good starting point for debugging authentication issues.

Prerequisites

  • You need the root password to run the sssctl command.

Procedure

  • Use the sssctl debug-level command to set the debug level of your choiceto your desired verbosity.

    [root@server ~]# sssctl debug-level 6

Additional resources

11.8. Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM server

If you experience issues when attempting to authenticate as an IdM user to an IdM server, enable detailed debug logging in the SSSD service on the server and gather logs of an attempt to retrieve information about the user.

Prerequisites

  • You need the root password to run the sssctl command and restart the SSSD service.

Procedure

  1. Enable detailed SSSD debug logging on the IdM server.

    [root@server ~]# sssctl debug-level 6
  2. Invalidate objects in the SSSD cache for the user that is experiencing authentication issues, so you do not bypass the LDAP server and retrieve information SSSD has already cached.

    [root@server ~]# sssctl cache-expire -u idmuser
  3. Minimize the troubleshooting dataset by removing older SSSD logs.

    [root@server ~]# sssctl logs-remove
  4. Attempt to switch to the user experiencing authentication problems, while gathering timestamps before and after the attempt. These timestamps further narrow the scope of the dataset.

    [root@server sssd]# date; su idmuser; date
    Mon Mar 29 15:33:48 EDT 2021
    su: user idmuser does not exist
    Mon Mar 29 15:33:49 EDT 2021
  5. (Optional) Lower the debug level if you do not wish to continue gathering detailed SSSD logs.

    [root@server ~]# sssctl debug-level 2
  6. Review SSSD logs for information about the failed request. For example, reviewing the /var/log/sssd/sssd_example.com.log file shows that the SSSD service did not find the user in the cn=accounts,dc=example,dc=com LDAP subtree. This might indicate that the user does not exist, or exists in another location.

    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [dp_get_account_info_send] (0x0200): Got request for [0x1][BE_REQ_USER][name=idmuser@example.com]
    ...
    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(uid=idmuser)(objectclass=posixAccount)(uid=)(&(uidNumber=)(!(uidNumber=0))))][cn=accounts,dc=example,dc=com].
    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set
    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results.
    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_search_by_name] (0x0400): No such entry
    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory)
    (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_search_by_name] (0x0400): No such entry
    (Mon Mar 29 15:33:49 2021) [sssd[be[example.com]]] [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request
  7. If you are unable to determine the cause of the authentication issue:

    1. Collect the SSSD logs you recently generated.

      [root@server ~]# sssctl logs-fetch sssd-logs-Mar29.tar
    2. Open a Red Hat Technical Support case and provide:

      1. The SSSD logs: sssd-logs-Mar29.tar
      2. The console output, including the time stamps and user name, of the request that corresponds to the logs:

        [root@server sssd]# date; id idmuser; date
        Mon Mar 29 15:33:48 EDT 2021
        id: ‘idmuser’: no such user
        Mon Mar 29 15:33:49 EDT 2021

11.9. Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM client

If you experience issues when attempting to authenticate as an IdM user to an IdM client, verify that you can retrieve user information on the IdM server. If you cannot retrieve the user information on an IdM server, you will not be able to retrieve it on an IdM client (which retrieves information from the IdM server).

After you have confirmed that authentication issues do not originate from the IdM server, gather SSSD debugging logs from both the IdM server and IdM client.

Prerequisites

  • The user only has authentication issues on IdM clients, not IdM servers.
  • You need the root password to run the sssctl command and restart the SSSD service.

Procedure

  1. On the client: Open the /etc/sssd/sssd.conf file in a text editor.
  2. On the client: Add the ipa_server option to the [domain] section of the file and set it to an IdM server. This avoids the IdM client autodiscovering other IdM servers, thus limiting this test to just one client and one server.

    [domain/example.com]
    ipa_server = server.example.com
    ...
  3. On the client: Save and close the sssd.conf file.
  4. On the client: Restart the SSSD service to load the configuration changes.

    [root@client ~]# systemctl restart sssd
  5. On the server and client: Enable detailed SSSD debug logging.

    [root@server ~]# sssctl debug-level 6
    [root@client ~]# sssctl debug-level 6
  6. On the server and client: Invalidate objects in the SSSD cache for the user experiencing authentication issues, so you do not bypass the LDAP database and retrieve information SSSD has already cached.

    [root@server ~]# sssctl cache-expire -u idmuser
    [root@client ~]# sssctl cache-expire -u idmuser
  7. On the server and client: Minimize the troubleshooting dataset by removing older SSSD logs.

    [root@server ~]# sssctl logs-remove
    [root@server ~]# sssctl logs-remove
  8. On the client: Attempt to switch to the user experiencing authentication problems while gathering timestamps before and after the attempt. These timestamps further narrow the scope of the dataset.

    [root@client sssd]# date; su idmuser; date
    Mon Mar 29 16:20:13 EDT 2021
    su: user idmuser does not exist
    Mon Mar 29 16:20:14 EDT 2021
  9. (Optional) On the server and client: Lower the debug level if you do not wish to continue gathering detailed SSSD logs.

    [root@server ~]# sssctl debug-level 0
    [root@client ~]# sssctl debug-level 0
  10. On the server and client: Review SSSD logs for information about the failed request.

    1. Review the request from the client in the client logs.
    2. Review the request from the client in the server logs.
    3. Review the result of the request in the server logs.
    4. Review the outcome of the client receiving the results of the request from the server.
  11. If you are unable to determine the cause of the authentication issue:

    1. Collect the SSSD logs you recently generated on the IdM server and IdM client. Label them according to their hostname or role.

      [root@server ~]# sssctl logs-fetch sssd-logs-server-Mar29.tar
      [root@client ~]# sssctl logs-fetch sssd-logs-client-Mar29.tar
    2. Open a Red Hat Technical Support case and provide:

      1. The SSSD debug logs:

        1. sssd-logs-server-Mar29.tar from the server
        2. sssd-logs-client-Mar29.tar from the client
      2. The console output, including the time stamps and user name, of the request that corresponds to the logs:

        [root@client sssd]# date; su idmuser; date
        Mon Mar 29 16:20:13 EDT 2021
        su: user idmuser does not exist
        Mon Mar 29 16:20:14 EDT 2021

11.10. Tracking client requests in the SSSD backend

SSSD processes requests asynchronously and as messages from different requests are added to the same log file, you can use the unique request identifier and client ID to track client requests in the back-end logs. The unique request identifier is added to the debug logs in the form of RID#<integer> and the client ID in the form [CID #<integer]. This allows you to isolate logs pertaining to an individual request, and you can track requests from start to finish across log files from multiple SSSD components.

Prerequisites

  • You have enabled debug logging and a request has been submitted from an IdM client.
  • You must have root privileges to display the contents of the SSSD log files.

Procedure

  1. To review your SSSD log file, open the log file using the less utility. For example, to view the /var/log/sssd/sssd_example.com.log:

    [root@server ~]# less /var/log/sssd/sssd_example.com.log
  2. Review the SSSD logs for information about the client request.

    (2021-07-26 18:26:37): [be[testidm.com]] [dp_req_destructor] (0x0400): [RID#3] Number of active DP request: 0
    (2021-07-26 18:26:37): [be[testidm.com]] [dp_req_reply_std] (0x1000): [RID#3] DP Request AccountDomain #3: Returning [Internal Error]: 3,1432158301,GetAccountDomain() not supported
    (2021-07-26 18:26:37): [be[testidm.com]] [dp_attach_req] (0x0400): [RID#4] DP Request Account #4: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
    (2021-07-26 18:26:37): [be[testidm.com]] [dp_attach_req] (0x0400): [RID#4] Number of active DP request: 1

    This sample output from an SSSD log file shows the unique identifiers RID#3 and RID#4 for two different requests.

However, a single client request to the SSSD client interface often triggers multiple requests in the backend and as a result it is not a 1-to-1 correlation between client request and requests in the backend. Though the multiple requests in the backend have different RID numbers, each initial backend request includes the unique client ID so an administrator can track the multiple RID numbers to the single client request.

The following example shows one client request [sssd.nss CID #1] and the multiple requests generated in the backend, [RID#5] to [RID#13]:

(2021-10-29 13:24:16): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#5] DP Request [Account #5]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:16): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#6] DP Request [AccountDomain #6]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:16): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#7] DP Request [Account #7]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#8] DP Request [Initgroups #8]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#9] DP Request [Account #9]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#10] DP Request [Account #10]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#11] DP Request [Account #11]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#12] DP Request [Account #12]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
(2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#13] DP Request [Account #13]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].