Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

Appendix A. Troubleshooting

A.1. Troubleshooting SSSD

A.1.1. Setting Debug Logs for SSSD Domains

Each domain sets its own debug log level. Increasing the log level can provide more information about problems with SSSD or with the domain configuration.
To change the log level, set the debug_level parameter for each section in the sssd.conf file for which to produce extra logs. For example:
[domain/LDAP]
cache_credentials = true
debug_level = 9

Table A.1. Debug Log Levels

Level Description
0 Fatal failures. Anything that would prevent SSSD from starting up or causes it to cease running.
1 Critical failures. An error that does not kill the SSSD, but one that indicates that at least one major feature is not going to work properly.
2 Serious failures. An error announcing that a particular request or operation has failed.
3 Minor failures. These are the errors that would percolate down to cause the operation failure of 2.
4 Configuration settings.
5 Function data.
6 Trace messages for operation functions.
7 Trace messages for internal control functions.
8 Contents of function-internal variables that may be interesting.
9 Extremely low-level tracing information.
To change the debug level while SSSD is running, use the sss_debuglevel utility, which is part of the sssd-tools package. For more information about how it works, see the sss_debuglevel man page.

A.1.2. Checking SSSD Log Files

SSSD uses a number of log files to report information about its operation, located in the /var/log/sssd/ directory. SSSD produces a log file for each domain, as well as an sssd_pam.log and an sssd_nss.log file.
Additionally, the /var/log/secure file logs authentication failures and the reason for the failure.

A.1.3. Problems with SSSD Configuration

Q: SSSD fails to start
Q: I do not see any groups with id or group members with getent group.
Q: Authentication fails against LDAP.
Q: Connecting to LDAP servers on non-standard ports fail.
Q: NSS fails to return user information
Q: NSS returns incorrect user information
Q: Setting the password for the local SSSD user prompts twice for the password
Q: An Active Directory identity provider is properly configured in my sssd.conf file, but SSSD fails to connect to it, with GSS-API errors.
Q: I configured SSSD for central authentication, but now several of my applications (such as Firefox or Adobe) will not start.
Q: SSSD is showing an automount location that I removed.
Q:
SSSD fails to start
A:
SSSD requires that the configuration file be properly set up, with all the required entries, before the daemon will start.
  • SSSD requires at least one properly configured domain before the service will start. Without a domain, attempting to start SSSD returns an error that no domains are configured:
    # sssd -d4 -i
    
    [sssd] [ldb] (3): server_sort:Unable to register control with rootdse!
    [sssd] [confdb_get_domains] (0): No domains configured, fatal error!
    [sssd] [get_monitor_config] (0): No domains configured.
    
    Edit the /etc/sssd/sssd.conf file and create at least one domain.
  • SSSD also requires at least one available service provider before it will start. If the problem is with the service provider configuration, the error message indicates that there are no services configured:
    [sssd] [get_monitor_config] (0): No services configured!
    
    Edit the /etc/sssd/sssd.conf file and configure at least one service provider.

    Important

    SSSD requires that service providers be configured as a comma-separated list in a single services entry in the /etc/sssd/sssd.conf file. If services are listed in multiple entries, only the last entry is recognized by SSSD.
  • SSSD also requires the ownership and permissions of the /etc/sssd/sssd.conf to be set correctly. If the ownership or permissions are set incorrectly, attempt to start SSSD returns these error messages:
    [sssd] [confdb_ldif_from_ini_file] (0x0020): Permission check on config file failed.
    [sssd] [confdb_init_db] (0x0020): Cannot convert INI to LDIF [1]: [Operation not permitted]
    [sssd] [confdb_setup] (0x0010): ConfDB initialization has failed [1]: Operation not permitted
    [sssd] [load_configuration] (0x0010): Unable to setup ConfDB [1]: Operation not permitted
    [sssd] [main] (0x0020): Cannot read config file /etc/sssd/sssd.conf. Please check that the file is accessible only by the owner and owned by root.root.
    
    Set the correct ownership and permissions of the /etc/sssd/sssd.conf file:
    # chmod 600 /etc/sssd/sssd.conf
    # chown root:root /etc/sssd/sssd.conf
    
Q:
I do not see any groups with id or group members with getent group.
A:
This may be due to an incorrect ldap_schema setting in the [domain/DOMAINNAME] section of sssd.conf.
SSSD supports RFC 2307 and RFC 2307bis schema types. By default, SSSD uses the more common RFC 2307 schema.
The difference between RFC 2307 and RFC 2307bis is the way which group membership is stored in the LDAP server. In an RFC 2307 server, group members are stored as the multi-valued memberuid attribute, which contains the name of the users that are members. In an RFC2307bis server, group members are stored as the multi-valued member or uniqueMember attribute which contains the DN of the user or group that is a member of this group. RFC2307bis allows nested groups to be maintained as well.
If group lookups are not returning any information:
  1. Set ldap_schema to rfc2307bis.
  2. Delete /var/lib/sss/db/cache_DOMAINNAME.ldb.
  3. Restarting SSSD.
If that does not work, add this line to sssd.conf:
ldap_group_name = uniqueMember
Then delete the cache and restart SSSD again.
Q:
Authentication fails against LDAP.
A:
To perform authentication, SSSD requires that the communication channel be encrypted. This means that if sssd.conf is configured to connect over a standard protocol (ldap://), it attempts to encrypt the communication channel with Start TLS. If sssd.conf is configured to connect over a secure protocol (ldaps://), then SSSD uses SSL.
This means that the LDAP server must be configured to run in SSL or TLS. TLS must be enabled for the standard LDAP port (389) or SSL enabled on the secure LDAPS port (636). With either SSL or TLS, the LDAP server must also be configured with a valid certificate trust.
An invalid certificate trust is one of the most common issues with authenticating against LDAP. If the client does not have proper trust of the LDAP server certificate, it is unable to validate the connection, and SSSD refuses to send the password. The LDAP protocol requires that the password be sent in plain text to the LDAP server. Sending the password in plain text over an unencrypted connection is a security problem.
If the certificate is not trusted, a syslog message is written, indicating that TLS encryption could not be started. The certificate configuration can be tested by checking if the LDAP server is accessible apart from SSSD. For example, this tests an anonymous bind over a TLS connection to test.example.com:
$ ldapsearch -x -ZZ -h test.example.com -b dc=example,dc=com
If the certificate trust is not properly configured, the test fails with this error:
ldap_start_tls: Connect error (-11) additional info: TLS error -8179:Unknown code ___f 13
To trust the certificate:
  1. Obtain a copy of the public CA certificate for the certificate authority used to sign the LDAP server certificate and save it to the local system.
  2. Add a line to the sssd.conf file that points to the CA certificate on the filesystem.
    ldap_tls_cacert = /path/to/cacert
  3. If the LDAP server uses a self-signed certificate, remove the ldap_tls_reqcert line from the sssd.conf file.
    This parameter directs SSSD to trust any certificate issued by the CA certificate, which is a security risk with a self-signed CA certificate.
Q:
Connecting to LDAP servers on non-standard ports fail.
A:
When running SELinux in enforcing mode, the client's SELinux policy has to be modified to connect to the LDAP server over the non-standard port. For example:
# semanage port -a -t ldap_port_t -p tcp 1389
Q:
NSS fails to return user information
A:
This usually means that SSSD cannot connect to the NSS service.
  • Ensure that the NSS service is running:
    # service sssd status
    Redirecting to /bin/systemctl status sssd.service
      sssd.service - System Security Services Daemon
       Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled)
       Active: active (running) since Wed 2015-01-14 10:17:26 CET; 1min 30s ago
       Process: 683 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS)
     Main PID: 745 (sssd)
       CGroup: /system.slice/sssd.service
                ├─745 /usr/sbin/sssd -D -f
    	    ├─746 /usr/libexec/sssd/sssd_be --domain default --debug-to-files...
    	    ├─804 /usr/libexec/sssd/sssd_nss --debug-to-files
    	    └─805 /usr/libexec/sssd/sssd_pam --debug-to-files
    NSS service is running when SSSD is in the Active: active (running) state and when the output includes sssd_nss.
  • If NSS is running, make sure that the provider is properly configured in the [nss] section of the /etc/sssd/sssd.conf file. Especially check the filter_users and filter_groups attributes.
  • Make sure that NSS is included in the list of services that SSSD uses.
  • Check the configuration in the /etc/nsswitch.conf file. For more information, see the section called “Configure NSS Services to Use SSSD”.
Q:
NSS returns incorrect user information
A:
If searches are returning the incorrect user information, check that there are not conflicting user names in separate domains. When there are multiple domains, set the use_fully_qualified_domains attribute to true in the /etc/sssd/sssd.conf file. This differentiates between different users in different domains with the same name.
Q:
Setting the password for the local SSSD user prompts twice for the password
A:
When attempting to change a local SSSD user's password, it may prompt for the password twice:
[root@clientF11 tmp]# passwd user1000
Changing password for user user1000.
New password:
Retype new password:
New Password:
Reenter new Password:
passwd: all authentication tokens updated successfully.
This is the result of an incorrect PAM configuration. Ensure that the use_authtok option is correctly configured in your /etc/pam.d/system-auth file. For examples of the correct configuration, see Section 7.5.2, “Configuring Services: PAM”.
Q:
An Active Directory identity provider is properly configured in my sssd.conf file, but SSSD fails to connect to it, with GSS-API errors.
A:
SSSD can only connect with an Active Directory provider using its host name. If the host name is not given, the SSSD client cannot resolve the IP address to the host, and authentication fails.
For example, with this configuration:
[domain/ADEXAMPLE]
debug_level = 0xFFF0
id_provider = ad
ad_server = 172.16.0.1
ad_domain = example.com
krb5_canonicalize = False
The SSSD client returns this GSS-API failure, and the authentication request fails:
(Fri Jul 27 18:27:44 2012) [sssd[be[ADTEST]]] [sasl_bind_send] (0x0020): ldap_sasl_bind failed (-2)[Local error]
(Fri Jul 27 18:27:44 2012) [sssd[be[ADTEST]]] [sasl_bind_send] (0x0080): Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Cannot determine realm for numeric host address)]
To avoid this error, set the ad_server to the name of the Active Directory host, or use the _srv_ keyword to use the DNS service discovery, as described in Section 7.4.3, “Configuring DNS Service Discovery”.
Q:
I configured SSSD for central authentication, but now several of my applications (such as Firefox or Adobe) will not start.
A:
Even on 64-bit systems, 32-bit applications require a 32-bit version of SSSD client libraries to use to access the password and identity cache. If a 32-bit version of SSSD is not available, but the system is configured to use the SSSD cache, then 32-bit applications can fail to start.
For example, Firefox can fail with permission denied errors:
Failed to contact configuration server. See http://www.gnome.org/projects/gconf/
for information. (Details -  1: IOR file '/tmp/gconfd-somebody/lock/ior'
not opened successfully, no gconfd located: Permission denied 2: IOR
file '/tmp/gconfd-somebody/lock/ior' not opened successfully, no gconfd
located: Permission denied)
For Adobe Reader, the error shows that the current system user is not recognized:
[jsmith@server ~]$ acroread
(acroread:12739): GLib-WARNING **: getpwuid_r(): failed due to unknown
user id (366)
Other applications may show similar user or permissions errors.
Q:
SSSD is showing an automount location that I removed.
A:
The SSSD cache for the automount location persists even if the location is subsequently changed or removed. To update the autofs information in SSSD:
  1. Restart SSSD:
    # systemctl restart sssd

A.1.4. A User Cannot Log In After UID or GID Changed

After a user or group ID changed, SSSD prevents a user from logging in.

What this means:

SSSD recognizes users and groups based on user IDs (UID) and group IDs (GID). When the UID or GID of an existing user or group changes, SSSD does not recognize the user or group.

To fix the problem:

Clear the SSSD cache using the sss_cache utility:
  1. Make sure the sssd-tools is installed.
  2. To clear the SSSD cache for a specific user and leave the rest of the cache records intact:
    # sss_cache --user user_name
    To clear the cache for an entire domain:
    # sss_cache --domain domain_name
The utility invalidates records in the SSSD cache for a user, group, or domain. After this, SSSD retrieves the records from the identity provider to refresh the cache.
For details on sss_cache, see the sss_cache(8) man page.

A.1.5. SSSD Control and Status Utility

SSSD provides the sssctl utility to obtain status information, manage data files during troubleshooting, and other SSSD-related tasks.
  1. To use sssctl, install the sssd-tools package:
    [root@server ~]# yum install sssd-tools
  2. Some options of sssctl use the SSSD InfoPipe responder. To enable it, add ifp to the services option of your /etc/sssd/sssd.conf:
    [sssd]
    services = nss, sudo, pam, ssh, ifp
  3. Restart SSSD:
    [root@server ~]# systemctl restart sssd.service

A.1.5.1. SSSD Configuration Validation

The sssctl config-check command performs a static analysis of the SSSD configuration files. This enables you to validate the /etc/sssd/sssd.conf file and /etc/sssd/conf.d/* files to receive a report before restarting SSSD.
The command performs the following checks on SSSD configuration files:
Permissions
The owner and group owner must be set to root:root and the permissions to 600.
File names
File names in /etc/sssd/conf.d/ must use the suffix .conf and not start with a period (hidden files).
Typographical errors
Typographical errors are checked in section and option names. Note that values are not checked.
Options
Options must be placed in the correct sections.
To verify the configuration, run:
# sssctl config-check
Issues identified by validators: 3
[rule/allowed_sections]: Section [paM] is not allowed. Check for typos.
[rule/allowed_domain_options]: Attribute 'offline_timeoutX' is not allowed in section 'domain/idm.example.com'. Check for typos.
[rule/allowed_sudo_options]: Attribute 'homedir_substring' is not allowed in section 'sudo'. Check for typos.

Messages generated during configuration merging: 2
File /etc/sssd/conf.d/wrong-file-permissions.conf did not pass access check. Skipping.
File configuration.conf.disabled did not match provided patterns. Skipping.

Used configuration snippet files: 1
/etc/sssd/conf.d/sample.conf

A.1.5.2. Domain Information

The domain status displays a list of domains SSSD accesses, and enables you to retrieve their status.
  1. List all domains available within SSSD, including trusted domains:
    [root@server ~]# sssctl domain-list
    idm.example.com
    ad.example.com
    
  2. Retrieve the status of the domain idm.example.com:
    [root@server ~]# sssctl domain-status idm.example.com
    Online status: Online

A.1.5.3. Cached Entries Information

The sssctl command enables you to retrieve information about a cached entry, to investigate and debug cache-related authentication problems.
To query cache information for the user account admin, run:
[root@server ~]# sssctl user-show admin
Name: admin
Cache entry creation date: 07/10/16 16:09:18
Cache entry last update time: 07/14/16 10:13:32
Cache entry expiration time: 07/14/16 11:43:32
Initgroups expiration time: Expired
Cached in InfoPipe: No
To query the cache information for a group or netgroup, use:
[root@server ~]# sssctl group-show groupname
[root@server ~]# sssctl netgroup-show netgroupname

A.1.5.4. Truncating the Log Files

When you debug a problem, you can use sssctl logs-remove to truncate all SSSD log files in the /var/log/sssd/ directory to capture only newly created entries.
[root@server ~]# sssctl logs-remove
Truncating log files...

A.1.5.5. Removing the SSSD Cache

To remove the SSSD cache database files, the sssctl command provides the remove-cache option. Before the databases are removed, the command creates automatically a backup.
Use the following command to back up all local data and remove the SSSD cache:
[root@server ~]# sssctl cache-remove
SSSD must not be running. Stop SSSD now? (yes/no) [yes]
Creating backup of local data...
Removing cache files...
SSSD needs to be running. Start SSSD now? (yes/no) [yes]

Note

The backup stores only local data, such as local overrides, in the /var/lib/sss/backup/ directory.
To automatically import the backed-up content, run sssctl restore-local-data.

A.1.5.6. Obtaining Information about an LDAP Group Takes Long

Operations that involve looking up information about an LDAP group take very long, especially in case of large groups or nested groups.

What this means:

By default, LDAP group information lookups return all members for the group. For operations that involve large groups or nested groups, returning all members makes the process longer.

To fix the problem:

The membership lists returned in group lookups are not used when evaluating whether a user belongs to a group. To improve performance, especially for identity lookups, disable the group membership lookup:
  1. Open the /etc/sssd/sssd.conf file.
  2. In the [domain] section, set the ignore_group_members option to true.
    [domain/domain_name]
    [... file truncated ...]
    ignore_group_members = true