Potential conflict between Samba and realmd-based setup, and resolution

Potential conflict between Samba and realmd-based setup, and resolution
By daniel
Update 2018-04-05: I highly suggest you do not do this yet. We discovered that, for some currently unexplained reason, that after the machine trust password is updated, net ads changetrustpw ends up pulling older principles into the machine keytab than the one that was just created. This seems to be related to some form of syncing lag though that would imply that the command updates on one dc and pulls from another. This method is the “correct” way to do this when samba/winbind is involved, but know that it’s not working properly right now.

So I ran into an interesting issue today where some of my realmd/Samba joined (–membership-software=samba) servers lost their domain trust. Rejoining fixed the issue, but clearly it’s not good if my Samba servers lose their minds periodically. So what happened?
You may already know that sssd handles refreshing your machine password / domain trust periodically (about every 30 days by default) — this is something that anything joined to the domain does. (Windows, OS X, whatever) When sssd performs this task, it does so via adcli (you can see this in the debug logs). Now, when you join the domain using the samba membership software, it uses net ads join. With a more or less unconfigured Samba server, these practically do the same thing. They store the trust in /etc/krb5.keytab and go on their merry way.
So why did mine break? Well, if you take a look at my smb.conf, you’ll see I have:
kerberos method = secrets and keytab
One thing adcli does -not- know how to do, is update secrets.tdb. So what happened to my machines here is that sssd called adcli to update the trust, only updated krb5.keytab, and neglected to touch secrets.tdb.
Why do I need secrets.tdb? Well, just using keytab functionality in Samba works -great- if you are doing Kerberos auth. (smbclient -k or connecting from a Windows host already in the same domain, as a couple of examples) What it does -not- do, is allow you to connect directly to the Samba server using a username and password that is verified against AD’s Kerberos. Samba basically stores a copy of said machine password in secrets.tdb so that it can use it later. Unfortunately, we need to be able to allow people to connect using a username and password for the sake of non-domain joined machines and other assorted scenarios.
So what can you do to fix this? Basically, you tell sssd to stop updating the password, and do it yourself via cron. The sssd setting is:
[domain/wolftech.ad.ncsu.edu]
ad_maximum_machine_account_password_age = 0
And then you need to run (I’d recommend monthly): /usr/bin/net ads changetrustpw
I can’t claim credit for this, I grabbed it from this issue tracker: https://bugs.freedesktop.org/show_bug.cgi?id=100118
Hopefully, someday, Samba will be able to handle the username/password auth without having to even use secrets.tdb, or if it’s already in there I sure haven’t found it. Hope this helps someone!