AD integration with SSSD

Latest response

We have several domain-joined servers running RHEL7 and configured (as per the Red Hat docs) to use SSSD for identity management and authentication. Initially, everything seemed fine but we started to notice problems on the hosts acting as Samba servers for Windows clients.

After a while the Samba shares were prompting for credentials but rejected them anyway. The problem was identified as the machine account passwords expiring every 30 days which is default AD behaviour. It seems SSSD was not renewing them automatically. Disjoining and re-joining the servers to AD was one workaround but not really practical.

The other solution found in the comments here https://access.redhat.com/solutions/2221561 suggest using a cron job with the command “net ads changetrustpw". This would reset the machine password periodically before it expired after 30 days. This seemed to be a workable solution but in fact leads to SSH authentication problems due to broken trust. The error in the logs:

[sssd[krb5_child[38028]]][38028]: Cannot find key for host/hostname@domain.com kvno 79 in keytab

Has anybody else running RHEL7 servers joined to AD with realmd/SSSD encountered this?

Other links I’ve looked at so far…

Potential conflict between Samba and realmd-based setup, and resolution
https://www.linux.ncsu.edu/blog/2018/03/30/potential-conflict-between-samba-and-realmd-based-setup-and-resolution/

msktutil to reset machine password
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/message/3ISKY3DROEPLH7YKKCTT5LQ5G3ZH6ABF/

Using adcli instead of realmd to join the domain
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/X7R52WLKNOAZKX3HKFUAKRHF5FZS3XKI/

Thanks

Responses

We have exact same issue, the first workaround

Just opened a ticket to RH :

We are hit by this exact problem described here : https://bugzilla.samba.org/show_bug.cgi?id=13429

Little story : we use SSSD and Samba without Winbind. Currently, SSSD is not able to update "secrets.tdb" by triggering "adcli update". The first workaround was to use "net ads changetrustpw" with "secrets and keytab" config of Samba to update keytab and secrets.

Unfortunately, looks like that workaround need a fix as once you use "net ads changetrustpw", it will suddenly stop updating the keytab and then, will stay at KVNO-1 (not the last). Sometime, could be KVNO-2" (probably too fast tests).

Looks like the net ads changetrustpw is doing ldap on another DC to update keytab and that DC didnt' yet receive the update (of course ..). If I try to fix directly with a "net ads keytab create", I hit the exact same issue as could connect to a DC which is not yet up-to-date and will simply provide previous KVNO. It is possible that it works well sometimes (chance) but we always ends up hitting that issue.

Found that issue on RHEL 7.5 but even with 7.6, the problem is still there.

Second workaround will be to "net ads keytab create" after a while once we are sure all DC's are updated. But this require admin password and so, not secure if to be used with cronjob ...

Hope that will help.

Hi Laurent,

Thank you for your reply. I tested your workaround by manually using "net ads changetrustpw", confirming that SSH auth was broken and then running "net ads keytab create -U Administrator". This does appear to resolve the auth problems which is great although it seems less than ideal having to run a command with Domain Admin credentials.

I'm still not sure I fully understand the cause of this problem in terms of your DC explanation. As far as I can tell replication is functioning well between our 3 DCs. The bug post you linked mentions something about the LDAP DC being different to the KDC. I don't believe this is the case but I'm not 100% sure how to confirm.

For now, I think recreating the keytab following a machine password reset is a solution we can live with.

Kind regards,

Just to add to this. It seems you can just run the following command without any AD credentials to fix the keytab.

net ads keytab create -P

We've suffering from this bug too. What I've found so far, systems experiencing this issue: - both latest RHEL 7.6 and RHEL 6.10 are affected - there are more entries in keytab than host/$hostname and HOSTNAME$ (in our case, the cifs/ prefix) - winbind is running (from 4.8, it's a must) - you're using NTLM authentication (besides kerberos)

The losing the membership after 30 days from sssd, specifically it calls adcli to renew the keytab and forgets to inform samba about the change. Similar talks: https://bugzilla.samba.org/show_bug.cgi?id=13429 https://bugzilla.redhat.com/show_bug.cgi?id=1665794 https://pagure.io/SSSD/sssd/issue/3920 <- we believe this is the root cause and a potential fix https://bugs.freedesktop.org/show_bug.cgi?id=100118 & https://gitlab.freedesktop.org/realmd/adcli/issues/6

With log_level=7 sssd logs the adcli invocation: /usr/sbin/adcli update --verbose --domain=acme.local --host-fqdn=testmachine.acme.local --computer-password-lifetime=0 --domain-controller=dc.acme.local

Which clearly WRONG because it does not call adcli with "add-samba-data" options. After that invocation: - sssd works - samba works when using kerberos - samba does not work with NTLM

The quick workaround is disabling sssd's auto-renew feature: ad_maximum_machine_account_password_age=0

When calling with "add-samba-data" parameters: /usr/sbin/adcli update --verbose --domain=acme.local --host-fqdn=testmachine.acme.local --computer-password-lifetime=0 --domain-controller=dc.acme.local --add-samba-data - sssd & samba works both with kerberos & ntlm

I just opened an official Red Hat ticket on this since the issue is clearly reproducable in our testing environment.

Basically the answer was from RH support to use 'ad_maximum_machine_account_password_age=0' as a workaround, until upstream responds the issue tracked at https://pagure.io/SSSD/sssd/issue/3920.

Thanks for the details Szabolcs. I have tried implementing 'ad_maximum_machine_account_password_age=0', but to know for sure if this worked I will need to check the server in 30 days. Can you confirm if this worked for you?

Hi, add this line in /etc/sssd/sssd.conf

ad_maximum_machine_account_password_age=0 
service sssd stop ; rm -f /var/lib/sss/db/* /var/log/sssd/* ; service sssd start

will do the job, I have done the same and samba works for 6 months without any problems.

We were faced with this issue, and because we only use samba (running daemon) on selected systems, we cannot relay completely on samba to refresh the trust, nor can we rely on sssd to ensure samba trusts are updated.

In the end, we have told sssd to not update via the suggested directive "ad_maximum_machine_account_password_age=0 "

and rely on a cron still. However, a static cron for this has proven (at least lately, AD upgrades recently too) to be a pain as "MANY" systems refresh there trust at the exactly same time and has become overwhelming for the AD servers to serve out.

ads_connect: No logon servers are currently available to service the logon request.

So, what I have done is created a cron.d file that will run every Sun Night (before Mon morning) to generate a completely random cron string to trigger the trust update once a week "/usr/bin/net ads changetrustpw" (This could be anywheres from 2-13 days between refreshes)

# cat /etc/cron.d/weekly_cron_setups 
SHELL=/bin/bash
55 23 * * Sun root  hour=$(echo -n 2; echo "$(( 1$(date +\%N) \% 4 ))"); min=$(( 1$(date +\%N) \% 60 )); days=(Mon Tue Wed Thu Fri Sat Sun); rd="$(( RANDOM \% 7 ))"; day="${days[$rd]}"; echo "$min $hour * * $day root /usr/bin/net ads changetrustpw | logger -t changetrustpw 2>&1" > /etc/cron.d/ads_changetrustpw

this will generate a random min (0-60) and a random hour between 8pm and 11pm 2(1-3) and select a random weekday name

# cat /etc/cron.d/ads_changetrustpw 
11 20 * * Sat root /usr/bin/net ads changetrustpw | logger -t changetrustpw 2>&1

None of the discussed solutions above have solved my problem.

We are facing the same issue here. Both packages sssd and adcli were installed as suggested by redhat. But after 30 days the machine password gets invalid (/etc/krb5.keytab) and sssd does "not" update the password as mentioned.

A rejoin to the domain fix the problem temporarily but this is awful.

Today we set the following settings in our /etc/sssd/sssd.conf to enable the machine password update. "ad_maximum_machine_account_password_age = 20" AND "ad_machine_account_password_renewal_opts = 86400:750"

Source: https://access.redhat.com/solutions/2420951 + https://sssd.io/docs/users/relnotes/notes_1_13_4.html + https://github.com/SSSD/sssd/issues/2083 + https://github.com/SSSD/sssd/commit/20ed1a2063e0463c9e97870ea4e5e607467b041e

I'll let you know if this leads to success or another failure within the next 20 days.

We hope this goes well for you Steven,

Post back when you can,

Regards
RJ

i have been facing this issue as well since May. except for some reason our systems are becoming problematic after 26 days. we've been having to reboot to get sssd to work properly and then realm rejoin. Did adjusting your max machine account password do anything?

We mistakenly believed that we had to strip off the domain name, which resulted in accidentally using the lowercase hostname to join, which puts the lowercase hostname in krb5.keytab. The problem is that during the automatic renewal, adcli asks AD for its own hostname, and AD always responds in UPPERCASE, which doesn't match the entries in krb5.keytab because krb5 on Linux is case-sensitive. If instead you specify the AD domain and don't specify the hostname, the join process should automatically trim off the domain and use the UPPERCASE hostname, and the UPPERCASE hostname will be used in krb5.keytab, which will match during the automatic update process. This is still not 100% reliable for some reason and I have a script that checks the age of krb5.keytab and runs the update when that gets too old.

Adding ad_maximum_machine_account_password_age=0 does not seem to work. Still receive "Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Preauthentication failed. Unable to create GSSAPI-encrypted LDAP connection" error every month. The only way to fix this is rejoining the machine to Domain. Any suggestion to fix this is much appreciated.

Hey guys, we faced the same issue 3 months ago.

Our setup looks like the following:

  • every linux node is running "sssd" and is fully joined to the AD (no problems with that)
  • we have a few nodes were we need to run samba / winbind to access SMB shares (this is were the trouble starts ...)

Problem:

  • What we saw was, that samba (winbind) tries to modify the system-wide krb5.keytab (/etc/krb5.keytab), and sssd modifies this one too.

The problem is, that both tools update this file at different times (sometimes days or weeks). If either one of the tools overwrite the settings inside the krb5.keytab your domain-join will be broken.

Solution (what we did -- and it is working fine):

Changes to "sssd":

  • added "ad_maximum_machine_account_password_age = 0" to /etc/sssd/sssd.conf

Changes to "samba/winbind":

  • added "machine password timeout = 0" to /etc/samba/smb.conf

Changes to "adcli":

  • create a new crontab file: /etc/cron.d/adcli-update
  • place the following content (without the quotes inside it):

"1 1 * * * root /usr/sbin/adcli update -v --add-samba-data --domain=YOUR_DOMAIN.LCL --domain-controller YOUR_DC.YOUR_DOMAIN.LCL 2>&1 | /usr/bin/logger -t adcli"

Restart all services:

  • systemctl stop sssd; systemctl stop smb; systemctl stop winbind; rm -fr /var/lib/sss/{mc,db}/* ;
  • systemctl start sssd; systemctl start winbind; systemctl start smb;

We simply disabled all "machine_password" modification inside of sssd, samba (winbind) and use adcli as a cronjob to watch the machine_password inside of the AD. If the password is going to expire (> 30 days), it will automatically update the password field inside of your AD, but only for your computer-account (machine account).

Well, we added the fix at June 2021 and we never had any issues with the tools again. All is working fine for us.

Feel free to reach out to me, I will answer the question as much as I can.

Cheers, Steven

@Steven, Thanks for the quick response. Will give it a try and see if that resolves the issues.