root user can no longer ssh into any server, but other users , yes
Hello all...
Using Version Red Hat 8.3
I have a newly installed server setup, named Locutus. This server is to replace an existing server, named Scotty(Centos 7)
This server as well as all the others are located on the same subnet on the same lan.
I had a script running on Locutus to rsync over all needed files from Scotty to Locutus, nothing from /root or /etc on Scotty.
As an example, excert of the script is:
nice rsync -varp --delete root@scotty:/data/tools /data
It was working fine until a few days ago, possibly stopped after an yum update, I cant remember if it happened at same time.
From Locutus I can ssh into any server as any user EXCEPT root, lets say we use the user rob in this case. So the user rob can ssh from Locutus to every server on my network, no problem via password or public key.
What I have done so far:
Verified that the sshd_config allows root user to ssh in, on all servers.
Restarted sshd on all servers, including Locutus.
Made sure /root perms are all correct on Locutus and all servers.
Sshing to Scotty as the user root just hangs and then eventually gives this error :
[root@locutus ~]# ssh scotty date
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Broken pipe
I tried as the user rob to ssh to scotty as root, and it worked:
[rob@locutus ~]$ ssh root@scotty date
Password:
I tried to ssh as the root user to Scotty as user rob, it DID NOT work, just hangs, gives same error:
[root@locutus ~]# ssh rob@scotty date
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Broken pipe
On Locutus I renamed /root/.ssh dir to .ssh.orig and created new ,ssh dir, tried to ssh again and get same result as above.
On any destination server in /var/log/secure i get nothing in the log when trying to ssh in as the user root, its almost like the user root does not know what ip Scotty is at.
So I enabled ssh debug on a server named apoc, and tried to ssh as the user root again, i get the below in the /var/log/secure/log when trying to ssh to apoc as the user root:
Feb 24 10:21:42 apoc sshd[1194]: debug1: Forked child 32633.
Feb 24 10:21:42 apoc sshd[32633]: debug1: Set /proc/self/oom_score_adj to 0
Feb 24 10:21:42 apoc sshd[32633]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Feb 24 10:21:42 apoc sshd[32633]: debug1: inetd sockets after dupping: 3, 3
Feb 24 10:21:42 apoc sshd[32633]: Connection from 192.168.2.24 port 46620 on 192.168.2.234 port 22
Feb 24 10:21:42 apoc sshd[32633]: debug1: Client protocol version 2.0; client software version OpenSSH_8.0
Feb 24 10:21:42 apoc sshd[32633]: debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
Feb 24 10:21:42 apoc sshd[32633]: debug1: Local version string SSH-2.0-OpenSSH_7.4
Feb 24 10:21:42 apoc sshd[32633]: debug1: Enabling compatibility mode for protocol 2.0
Feb 24 10:21:42 apoc sshd[32633]: debug1: SELinux support enabled [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: permanently_set_uid: 74/74 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: SSH2_MSG_KEXINIT received [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: compression: none [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: compression: none [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: kex: curve25519-sha256 need=32 dh_need=32 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: kex: curve25519-sha256 need=32 dh_need=32 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: rekey after 4294967296 blocks [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: rekey after 4294967296 blocks [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: KEX done [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: userauth-request for user root service ssh-connection method none [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: attempt 0 failures 0 [preauth]
Feb 24 10:21:42 apoc sshd[32633]: debug1: user root matched 'User root' at line 148
Feb 24 10:21:42 apoc sshd[32633]: debug1: PAM: initializing for "root"
Feb 24 10:21:42 apoc sshd[32633]: debug1: PAM: setting PAM_RHOST to "locutus.domain.com"
Feb 24 10:21:42 apoc sshd[32633]: debug1: PAM: setting PAM_TTY to "ssh"
Firewalld is disabled on Locutus and on all other servers
Locutus is on a domain with FreeIPA as well as all other servers.
When sshing with -vvv from Locutus as the user root i get this(truncated):
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
Ticket expired
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Broken pipe
I'm pretty much out of things to check, unless i forgot something very obvious to check, lol
Oh ya, I CAN ssh to Locutus from any server as the user root.
Any suggestions appreciated.
Thanks and have a great day and stay safe!
Rob Morin
Montreal, Canada
