[semi-resolved] Red Hat 7.x ssh fails with error "System is booting up. See pam_nologin(8) \ Authentication Failed." (but system is booted)

This is not a Red Hat solution, this is the Red Hat discussion area. I'm documenting this here because I've had at minimum 3 systems have this issue and want to document it for others who might be experiencing this (or for myself or team members when I google it).

Important: Please submit a case with Red Hat

• And please cite this discussion in the case.

Environment

• Red Hat Enterprise Linux 7.x Server

Note: make a comment in this discussion if you encounter this on a system that is not RHEL 7

Issue

• ssh to a RHEL 7 server fails with the following erroneous error in the below block:

# your ip address or hostname will obviously differ
[you@yoursystem] # ssh 192.168.100.100
System is booting up. See pam_nologin(8)
Authentication failed.

• The system you are attempting to ssh to is actually not in the boot process.

• The user you are sshing to is not a nologin account

Perhaps Resolution?

• Attempt some of the steps below. The discussion after this shows some "resolved" this through reloading their system (kinda not a real true fix).

• Recommend making an sosreport and submitting it with a case to Red Hat.

• IMPORTANT Really know that this system is not in the process of booting. If it is not then manually log into the system, become root and remove /run/nologin file

• Log into the system using the console of the computer (physical or virtual, or web interface for a console). Then remove the /run/nologin file.

• Virtual system example, if it is a VMware system, use the VMware interface to log in and perform this.

• Visit the system and log into the console if it is a physical server.

• If you have a remote management tool (Dell for example has "iDrac"), use it to gain access to your system

• Some systems might have Red Hat Cockpit installed and running, use it if possible to attain terminal access.

• Again, once you are logged in, switch to the root account and remove the /run/nologin file

• Additionally check for the following files and remove if they exist:

ls  /{var/run,etc,run}/nologin && rm  /{var/run,etc,run}/nologin

Root Cause

• I personally am still examining this issue to find a root cause.

• A problem close to this exists in Fedora https://bugzilla.redhat.com/show_bug.cgi?id=1043212

Request any Red Hatters or anyone else post anything relevant on this topic (especially if there is a source solution or article from redhat).

Regards

RJ