Unable to SSH to new Virtual Machine after upgrading the template to RHEL 8.7 or 9

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 9
    • cloud-init-22.1-5 or higher (9.1)
    • cloud-init-21.1-10 or higher (9.0)
  • Red Hat Enterprise Linux (RHEL) 8
    • cloud-init-22.1-5 or higher (8.7)
  • Configuration file cloud.cfg originally created on RHEL 8.4 with cloud-init-20.3-10 or lower
  • Red Hat Virtualization 4
  • Red Hat OpenStack Platform 16
  • Red Hat OpenShift Container Platform 4
  • Amazon AWS
  • Microsoft Azure
  • Google Cloud Platform

Issue

  • Unable to SSH to a Virtual Machine after upgrading it to RHEL 8.7, 9 or higher versions.
  • During first boot, sshd fails to start on new Virtual Machines:
Nov 29 08:49:18 rhel8 systemd[1]: Starting OpenSSH server daemon...
Nov 29 08:49:18 rhel8 sshd[2946]: Unable to load host key: /etc/ssh/ssh_host_rsa_key
Nov 29 08:49:18 rhel8 sshd[2946]: Unable to load host key: /etc/ssh/ssh_host_ecdsa_key
Nov 29 08:49:18 rhel8 sshd[2946]: Unable to load host key: /etc/ssh/ssh_host_ed25519_key
Nov 29 08:49:18 rhel8 sshd[2946]: sshd: no hostkeys available -- exiting.
Nov 29 08:49:18 rhel8 systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 08:49:18 rhel8 systemd[1]: sshd.service: Failed with result 'exit-code'.
Nov 29 08:49:18 rhel8 systemd[1]: Failed to start OpenSSH server daemon.

Resolution

  • After upgrading the cloud-init package the cloud.cfg.rpmnew file may be created. If so please check for the differences and merge or update the cloud.cfg file.

    diff --suppress-common-lines -y /etc/cloud/cloud.cfg.rpmnew /etc/cloud/cloud.cfg

  • Ensure the configuration for ssh_genkeytypes is set as follows in /etc/cloud/cloud.cfg

    ssh_genkeytypes:  ['rsa', 'ecdsa', 'ed25519']

  • For Virtual Machine that have already being provisioned, you can use the below to manually generate host ssh keys:

    # /usr/libexec/openssh/sshd-keygen rsa
# /usr/libexec/openssh/sshd-keygen ecdsa
# /usr/libexec/openssh/sshd-keygen ed25519

Root Cause

Diagnostic Steps

  • While upgrading the cloud-init package, a new cloud.cfg.rpmnew was created:

    Preparing        :                                                      1/1 
Running scriptlet: cloud-init-22.1-5.el8.noarch                         1/1 
Upgrading        : cloud-init-22.1-5.el8.noarch                         1/2 
warning: /etc/cloud/cloud.cfg created as /etc/cloud/cloud.cfg.rpmnew
Running scriptlet: cloud-init-22.1-5.el8.noarch

  • The configuration file still contains the old ssh_genkeytypes: ~ syntax, which does not work with cloud-init version 22 or higher.

    # grep ssh_genkeytypes /etc/cloud/cloud.cfg /etc/cloud/cloud.cfg.rpmnew 
/etc/cloud/cloud.cfg:ssh_genkeytypes:  ~
/etc/cloud/cloud.cfg.rpmnew:ssh_genkeytypes:  ['rsa', 'ecdsa', 'ed25519']

  • There are no keys under /etc/ssh:

    # ls -l /etc/ssh/
total 576
-rw-r--r--. 1 root root 577388 Oct 26  2021 moduli
-rw-r--r--. 1 root root   1770 Oct 26  2021 ssh_config
drwxr-xr-x. 2 root root     28 Nov 22 16:31 ssh_config.d
-rw-------. 1 root root   4269 Oct 26  2021 sshd_config

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments