sso w/ rhel 9 against active directory, (gssapi) has anyone gotten it to work?

Latest response

I've always joined my linux systems to a windows domain and used putty and now powershell 7 as well to connect without having to specify a password using GSSAPI authentication.

This worked with Centos 7, and Centos 8, but with RedHat 9 I have not for the life of me been able to get it to work has anyone? Do you have a write up how to make it work?

In the past it was pretty much as simple as:
- join the linux system to a domain
- verify you can login with an ad user
Prep to connect via GSSAPI:
- locate the linux computer object in AD
- edit properties, find 'Delegation' tab, select 'Trust this computer for delegation to any service (Kerberos only)'
- in putty specify the auto-login username to be 'username@domain'
- make sure the linux system hostname.domain resolves in dns
- ensure putty will attempt gss (it does by default)
- connect with putty without having to specify a password

Note:
Actually centos 8 required adding a line to /etc/krb5.conf, without it the SSO signin wouldn't work by default:

includedir /var/lib/sss/pubconf/krb5.include.d/

The steps above yield the following log when attempting to connect to ku.HOME.NET:

Nov 19 12:05:24 ku sshd[1810]: debug1: Forked child 2116.
Nov 19 12:05:24 ku sshd[2116]: debug1: Set /proc/self/oom_score_adj to 0
Nov 19 12:05:24 ku sshd[2116]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Nov 19 12:05:24 ku sshd[2116]: debug1: inetd sockets after dupping: 4, 4
Nov 19 12:05:24 ku sshd[2116]: Connection from 10.0.0.19 port 51633 on 10.0.0.18 port 22 rdomain ""
Nov 19 12:05:24 ku sshd[2116]: debug1: Local version string SSH-2.0-OpenSSH_8.7
Nov 19 12:05:24 ku sshd[2116]: debug1: Remote protocol version 2.0, remote software version OpenSSH_for_Windows_8.6
Nov 19 12:05:24 ku sshd[2116]: debug1: compat_banner: match: OpenSSH_for_Windows_8.6 pat OpenSSH* compat 0x04000000
Nov 19 12:05:24 ku sshd[2116]: debug1: SELinux support enabled [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: ssh_selinux_change_context: setting context from 'system_u:system_r:sshd_t:s0-s0:c0.c1023' to 'system_u:system_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: permanently_set_uid: 74/74 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: SSH2_MSG_KEXINIT received [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: kex: curve25519-sha256 need=64 dh_need=64 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: kex: curve25519-sha256 need=64 dh_need=64 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: rekey out after 134217728 blocks [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: Sending SSH2_MSG_EXT_INFO [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: rekey in after 134217728 blocks [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: KEX done [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: userauth-request for user travis@HOME.NET service ssh-connection method none [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: attempt 0 failures 0 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: PAM: initializing for "travis@HOME.NET"
Nov 19 12:05:24 ku sshd[2116]: debug1: PAM: setting PAM_RHOST to "10.0.0.19"
Nov 19 12:05:24 ku sshd[2116]: debug1: PAM: setting PAM_TTY to "ssh"
Nov 19 12:05:24 ku sshd[2116]: debug1: userauth-request for user travis@HOME.NET service ssh-connection method gssapi-with-mic [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: attempt 1 failures 0 [preauth]
Nov 19 12:05:24 ku sshd[2116]: debug1: No credentials were supplied, or the credentials were unavailable or inaccessible\nNo key table entry found matching host/ku@\n\n

Additional:

$ hostname -f
ku.HOME.NET

$ sudo klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 KU$@HOME.NET
   5 KU$@HOME.NET
   5 host/KU@HOME.NET
   5 host/KU@HOME.NET
   5 host/ku.HOME.NET@HOME.NET
   5 host/ku.HOME.NET@HOME.NET
   5 RestrictedKrbHost/KU@HOME.NET
   5 RestrictedKrbHost/KU@HOME.NET
   5 RestrictedKrbHost/ku.HOME.NET@HOME.NET
   5 RestrictedKrbHost/ku.HOME.NET@HOME.NET
Travis Loyd's picture
Log in to join the conversation

Responses

Dusan Baljevic's picture Guru 2929 points
19 November 2023 11:37 PM

Hi Travis,

I wish I had RHEL 9 test environment to verify your problem. I have been begging my support teams to build some for me, but they are too slow...

It appears you already confirmed that your hostname is a FQDN. "hostnamectl --static" should return FQDN.

Couple of suggestions:

  1. Are you running systemd-resolved and if so, can you disable it temporarily?

systemd-resolved is an unsupported Technology Preview on RHEL 9.

  1. I do not use upper-case domains in hostnames, but at the places where I worked, I observed another unusual problem with GSSAPI, which might be affecting you:

Lower-case KVNO for host/ku.home.net@HOME.NET does not exist

  1. Did you enable rDNS in /etc/krb5.conf, and if so, could you disable it temporarily:
[libdefaults]
 rdns = true
...

Actually, it would be good to see full contents of /etckrb5.conf and /etc/sssd/sssd.conf.

  1. Can you run this piece of Shell code on your server:
echo -e "\n\nComparing Kerberos key version numbers (KVNO) between Domain Controller and local keytab"
MYHOST="$(uname -n)"
MYDOM="HOME.NET”
KV1=$(su myusername -c "kvno host/$MYHOST 2>/dev/null" | awk -F= '{print $2}' | xargs)
KV2=$(klist -k $KRB5KEY 2>/dev/null | egrep -i " host/${MYHOST}@HOME.NET" | sort -k1 -n | uniq | tail -1 | awk '{print $1}' | xargs)
if [ "$KV1" != "" ] && [ "$KV2" != "" ]
then
      if [ "$KV1" != "$KV2" ]
      then
         echo -e "KVNO used by the Domain Controller and the locally acquired service principal keys differ"
      else
         echo -e "KVNO used by the Domain Controller (value $KV1) and the locally acquired service principal key (value $KV2) are same"
      fi
fi

Regards,

Dusan Baljevic (amateur radio VK2COT)

Travis Loyd's picture Community Member 65 points
20 November 2023 3:19 AM
  1. sudo systemctl status systemd-resolved [sudo] password for travis: Unit systemd-resolved.service could not be found.
  2. I'd prefer it to all be lowercase, but it is what it is, dns resolves both but it could be an issue maybe.
  3. rdns has been set to false, haven't tried it with true
  4. The script returned no output
Travis Loyd's picture Community Member 65 points
20 November 2023 3:20 AM 
$ kvno host/ku
host/ku@HOME.NET: kvno = 6
$ kvno host/ku.home.net
host/ku.home.net@HOME.NET: kvno = 6
$ kvno host/ku.HOME.NET
host/ku.HOME.NET@HOME.NET: kvno = 6

# kvno host/ku
kvno: Credentials cache 'KCM:0' not found while getting client principal name
Dusan Baljevic's picture Guru 2929 points
20 November 2023 3:54 AM

Uhh, I forgot to add variable for KRB5KEY:

echo -e "\n\nComparing Kerberos key version numbers (KVNO) between Domain Controller and local keytab"
MYHOST="$(uname -n)"
MYDOM="HOME.NET"
KRB5KEY="/etc/krb5.keytab"
KV1=$(su myusername -c "kvno host/$MYHOST 2>/dev/null" | awk -F= '{print $2}' | xargs)
KV2=$(klist -k $KRB5KEY 2>/dev/null | egrep -i " host/${MYHOST}@HOME.NET" | sort -k1 -n | uniq | tail -1 | awk '{print $1}' | xargs)
if [ "$KV1" != "" ] && [ "$KV2" != "" ]
then
      if [ "$KV1" != "$KV2" ]
      then
         echo -e "KVNO used by the Domain Controller and the locally acquired service principal keys differ"
      else
         echo -e "KVNO used by the Domain Controller (value $KV1) and the locally acquired service principal key (value $KV2) are same"
      fi
fi

Results from kvno command shows that KVNO was 6.

What is the result of command "klist -k..." as above?

What is the full content of /etc/nsswitch.conf, /etc/krb5.conf and /etc/ssh/sshd_config?

Since you are not using systemd-resolved, next question is if you use any other type of caching (dnsmasq, nscd, and so on)?

UseDNS, when enabled in /etc/ssh/sshd_config, does a reverse lookup on the IP connecting, and can play into Kerberos auth issues because Kerberos is heavily dependent on DNS.

Regards,

Dusan Baljevic (amateur radio VK2COT)

Travis Loyd's picture Community Member 65 points
22 November 2023 3:05 AM

I won't be able to troubleshoot this more until end of this week.

In the meantime I will say that I'm redeploying an environment with rhel 9 and at this point, I could just deploy out a new server and configure it correctly the first time, if there were instructions... rather than try to fix something not working.

At this point though, anything to get this going would be nice, so I will perform the next troubleshooting steps soon as I can, thank you for your help ahead of time.

Travis Loyd's picture Community Member 65 points
3 December 2023 3:07 AM

I can wait until your team puts together an environment for you to experiment in. I think we are in different time zones and I think figuring this out here would be like trying to play a long game of chess through the mail. Once you have the environment I bet you'll figure it out in a day or two. Look forward to hearing how things go for you.