Chapter 1. Using secure communications between two systems with OpenSSH

SSH (Secure Shell) is a protocol which provides secure communications between two systems using a client-server architecture and allows users to log in to server host systems remotely. Unlike other remote communication protocols, such as FTP or Telnet, SSH encrypts the login session, which prevents intruders to collect unencrypted passwords from the connection.

Red Hat Enterprise Linux includes the basic OpenSSH packages: the general openssh package, the openssh-server package and the openssh-clients package. Note that the OpenSSH packages require the OpenSSL package openssl-libs, which installs several important cryptographic libraries that enable OpenSSH to provide encrypted communications.

1.1. SSH and OpenSSH

SSH (Secure Shell) is a program for logging into a remote machine and executing commands on that machine. The SSH protocol provides secure encrypted communications between two untrusted hosts over an insecure network. You can also forward X11 connections and arbitrary TCP/IP ports over the secure channel.

The SSH protocol mitigates security threats, such as interception of communication between two systems and impersonation of a particular host, when you use it for remote shell login or file copying. This is because the SSH client and server use digital signatures to verify their identities. Additionally, all communication between the client and server systems is encrypted.

A host key authenticates hosts in the SSH protocol. Host keys are cryptographic keys that are generated automatically when OpenSSH is first installed, or when the host boots for the first time.

OpenSSH is an implementation of the SSH protocol supported by Linux, UNIX, and similar operating systems. It includes the core files necessary for both the OpenSSH client and server. The OpenSSH suite consists of the following user-space tools:

  • ssh is a remote login program (SSH client).
  • sshd is an OpenSSH SSH daemon.
  • scp is a secure remote file copy program.
  • sftp is a secure file transfer program.
  • ssh-agent is an authentication agent for caching private keys.
  • ssh-add adds private key identities to ssh-agent.
  • ssh-keygen generates, manages, and converts authentication keys for ssh.
  • ssh-copy-id is a script that adds local public keys to the authorized_keys file on a remote SSH server.
  • ssh-keyscan gathers SSH public host keys.
Note

In RHEL 9, the Secure copy protocol (SCP) is replaced with the SSH File Transfer Protocol (SFTP) by default. This is because SCP has already caused security issues, for example CVE-2020-15778.

If SFTP is unavailable or incompatible in your scenario, you can use the -O option to force use of the original SCP/RCP protocol.

For additional information, see the OpenSSH SCP protocol deprecation in Red Hat Enterprise Linux 9 article.

Two versions of SSH currently exist: version 1, and the newer version 2. The OpenSSH suite in RHEL supports only SSH version 2. It has an enhanced key-exchange algorithm that is not vulnerable to exploits known in version 1.

OpenSSH, as one of core cryptographic subsystems of RHEL, uses system-wide crypto policies. This ensures that weak cipher suites and cryptographic algorithms are disabled in the default configuration. To modify the policy, the administrator must either use the update-crypto-policies command to adjust the settings or manually opt out of the system-wide crypto policies.

The OpenSSH suite uses two sets of configuration files: one for client programs (that is, ssh, scp, and sftp), and another for the server (the sshd daemon).

System-wide SSH configuration information is stored in the /etc/ssh/ directory. User-specific SSH configuration information is stored in ~/.ssh/ in the user’s home directory. For a detailed list of OpenSSH configuration files, see the FILES section in the sshd(8) man page.

Additional resources

1.2. Configuring and starting an OpenSSH server

Use the following procedure for a basic configuration that might be required for your environment and for starting an OpenSSH server. Note that after the default RHEL installation, the sshd daemon is already started and server host keys are automatically created.

Prerequisites

  • The openssh-server package is installed.

Procedure

  1. Start the sshd daemon in the current session and set it to start automatically at boot time:

    # systemctl start sshd
    # systemctl enable sshd
  2. To specify different addresses than the default 0.0.0.0 (IPv4) or :: (IPv6) for the ListenAddress directive in the /etc/ssh/sshd_config configuration file and to use a slower dynamic network configuration, add the dependency on the network-online.target target unit to the sshd.service unit file. To achieve this, create the /etc/systemd/system/sshd.service.d/local.conf file with the following content:

    [Unit]
    Wants=network-online.target
    After=network-online.target
  3. Review if OpenSSH server settings in the /etc/ssh/sshd_config configuration file meet the requirements of your scenario.
  4. Optionally, change the welcome message that your OpenSSH server displays before a client authenticates by editing the /etc/issue file, for example:

    Welcome to ssh-server.example.com
    Warning: By accessing this server, you agree to the referenced terms and conditions.

    Ensure that the Banner option is not commented out in /etc/ssh/sshd_config and its value contains /etc/issue:

    # less /etc/ssh/sshd_config | grep Banner
    Banner /etc/issue

    Note that to change the message displayed after a successful login you have to edit the /etc/motd file on the server. See the pam_motd man page for more information.

  5. Reload the systemd configuration and restart sshd to apply the changes:

    # systemctl daemon-reload
    # systemctl restart sshd

Verification

  1. Check that the sshd daemon is running:

    # systemctl status sshd
    ● sshd.service - OpenSSH server daemon
       Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
       Active: active (running) since Mon 2019-11-18 14:59:58 CET; 6min ago
         Docs: man:sshd(8)
               man:sshd_config(5)
     Main PID: 1149 (sshd)
        Tasks: 1 (limit: 11491)
       Memory: 1.9M
       CGroup: /system.slice/sshd.service
               └─1149 /usr/sbin/sshd -D -oCiphers=aes128-ctr,aes256-ctr,aes128-cbc,aes256-cbc -oMACs=hmac-sha2-256,>
    
    Nov 18 14:59:58 ssh-server-example.com systemd[1]: Starting OpenSSH server daemon...
    Nov 18 14:59:58 ssh-server-example.com sshd[1149]: Server listening on 0.0.0.0 port 22.
    Nov 18 14:59:58 ssh-server-example.com sshd[1149]: Server listening on :: port 22.
    Nov 18 14:59:58 ssh-server-example.com systemd[1]: Started OpenSSH server daemon.
  2. Connect to the SSH server with an SSH client.

    # ssh user@ssh-server-example.com
    ECDSA key fingerprint is SHA256:dXbaS0RG/UzlTTku8GtXSz0S1++lPegSy31v3L/FAEc.
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added 'ssh-server-example.com' (ECDSA) to the list of known hosts.
    
    user@ssh-server-example.com's password:

Additional resources

  • sshd(8) and sshd_config(5) man pages.

1.3. Setting an OpenSSH server for key-based authentication

To improve system security, enforce key-based authentication by disabling password authentication on your OpenSSH server.

Prerequisites

  • The openssh-server package is installed.
  • The sshd daemon is running on the server.

Procedure

  1. Open the /etc/ssh/sshd_config configuration in a text editor, for example:

    # vi /etc/ssh/sshd_config
  2. Change the PasswordAuthentication option to no:

    PasswordAuthentication no

    On a system other than a new default installation, check that PubkeyAuthentication no has not been set and the KbdInteractiveAuthentication directive is set to no. If you are connected remotely, not using console or out-of-band access, test the key-based login process before disabling password authentication.

  3. To use key-based authentication with NFS-mounted home directories, enable the use_nfs_home_dirs SELinux boolean:

    # setsebool -P use_nfs_home_dirs 1
  4. Reload the sshd daemon to apply the changes:

    # systemctl reload sshd

Additional resources

  • sshd(8), sshd_config(5), and setsebool(8) man pages.

1.4. Generating SSH key pairs

Use this procedure to generate an SSH key pair on a local system and to copy the generated public key to an OpenSSH server. If the server is configured accordingly, you can log in to the OpenSSH server without providing any password.

Important

If you complete the following steps as root, only root is able to use the keys.

Procedure

  1. To generate an ECDSA key pair for version 2 of the SSH protocol:

    $ ssh-keygen -t ecdsa
    Generating public/private ecdsa key pair.
    Enter file in which to save the key (/home/joesec/.ssh/id_ecdsa):
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/joesec/.ssh/id_ecdsa.
    Your public key has been saved in /home/joesec/.ssh/id_ecdsa.pub.
    The key fingerprint is:
    SHA256:Q/x+qms4j7PCQ0qFd09iZEFHA+SqwBKRNaU72oZfaCI joesec@localhost.example.com
    The key's randomart image is:
    +---[ECDSA 256]---+
    |.oo..o=++        |
    |.. o .oo .       |
    |. .. o. o        |
    |....o.+...       |
    |o.oo.o +S .      |
    |.=.+.   .o       |
    |E.*+.  .  . .    |
    |.=..+ +..  o     |
    |  .  oo*+o.      |
    +----[SHA256]-----+

    You can also generate an RSA key pair by using the -t rsa option with the ssh-keygen command or an Ed25519 key pair by entering the ssh-keygen -t ed25519 command.

  2. To copy the public key to a remote machine:

    $ ssh-copy-id joesec@ssh-server-example.com
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    joesec@ssh-server-example.com's password:
    ...
    Number of key(s) added: 1
    
    Now try logging into the machine, with: "ssh 'joesec@ssh-server-example.com'" and check to make sure that only the key(s) you wanted were added.

    If you do not use the ssh-agent program in your session, the previous command copies the most recently modified ~/.ssh/id*.pub public key if it is not yet installed. To specify another public-key file or to prioritize keys in files over keys cached in memory by ssh-agent, use the ssh-copy-id command with the -i option.

Note

If you reinstall your system and want to keep previously generated key pairs, back up the ~/.ssh/ directory. After reinstalling, copy it back to your home directory. You can do this for all users on your system, including root.

Verification

  1. Log in to the OpenSSH server without providing any password:

    $ ssh joesec@ssh-server-example.com
    Welcome message.
    ...
    Last login: Mon Nov 18 18:28:42 2019 from ::1

Additional resources

  • ssh-keygen(1) and ssh-copy-id(1) man pages.

1.5. Using SSH keys stored on a smart card

Red Hat Enterprise Linux enables you to use RSA and ECDSA keys stored on a smart card on OpenSSH clients. Use this procedure to enable authentication using a smart card instead of using a password.

Prerequisites

  • On the client side, the opensc package is installed and the pcscd service is running.

Procedure

  1. List all keys provided by the OpenSC PKCS #11 module including their PKCS #11 URIs and save the output to the keys.pub file:

    $ ssh-keygen -D pkcs11: > keys.pub
    $ ssh-keygen -D pkcs11:
    ssh-rsa AAAAB3NzaC1yc2E...KKZMzcQZzx pkcs11:id=%02;object=SIGN%20pubkey;token=SSH%20key;manufacturer=piv_II?module-path=/usr/lib64/pkcs11/opensc-pkcs11.so
    ecdsa-sha2-nistp256 AAA...J0hkYnnsM= pkcs11:id=%01;object=PIV%20AUTH%20pubkey;token=SSH%20key;manufacturer=piv_II?module-path=/usr/lib64/pkcs11/opensc-pkcs11.so
  2. To enable authentication using a smart card on a remote server (example.com), transfer the public key to the remote server. Use the ssh-copy-id command with keys.pub created in the previous step:

    $ ssh-copy-id -f -i keys.pub username@example.com
  3. To connect to example.com using the ECDSA key from the output of the ssh-keygen -D command in step 1, you can use just a subset of the URI, which uniquely references your key, for example:

    $ ssh -i "pkcs11:id=%01?module-path=/usr/lib64/pkcs11/opensc-pkcs11.so" example.com
    Enter PIN for 'SSH key':
    [example.com] $
  4. You can use the same URI string in the ~/.ssh/config file to make the configuration permanent:

    $ cat ~/.ssh/config
    IdentityFile "pkcs11:id=%01?module-path=/usr/lib64/pkcs11/opensc-pkcs11.so"
    $ ssh example.com
    Enter PIN for 'SSH key':
    [example.com] $

    Because OpenSSH uses the p11-kit-proxy wrapper and the OpenSC PKCS #11 module is registered to PKCS#11 Kit, you can simplify the previous commands:

    $ ssh -i "pkcs11:id=%01" example.com
    Enter PIN for 'SSH key':
    [example.com] $

If you skip the id= part of a PKCS #11 URI, OpenSSH loads all keys that are available in the proxy module. This can reduce the amount of typing required:

$ ssh -i pkcs11: example.com
Enter PIN for 'SSH key':
[example.com] $

Additional resources

1.6. Making OpenSSH more secure

The following tips help you to increase security when using OpenSSH. Note that changes in the /etc/ssh/sshd_config OpenSSH configuration file require reloading the sshd daemon to take effect:

# systemctl reload sshd
Important

The majority of security hardening configuration changes reduce compatibility with clients that do not support up-to-date algorithms or cipher suites.

Disabling insecure connection protocols

  • To make SSH truly effective, prevent the use of insecure connection protocols that are replaced by the OpenSSH suite. Otherwise, a user’s password might be protected using SSH for one session only to be captured later when logging in using Telnet. For this reason, consider disabling insecure protocols, such as telnet, rsh, rlogin, and ftp.

Enabling key-based authentication and disabling password-based authentication

  • Disabling passwords for authentication and allowing only key pairs reduces the attack surface and it also might save users’ time. On clients, generate key pairs using the ssh-keygen tool and use the ssh-copy-id utility to copy public keys from clients on the OpenSSH server. To disable password-based authentication on your OpenSSH server, edit /etc/ssh/sshd_config and change the PasswordAuthentication option to no:

    PasswordAuthentication no

Key types

  • Although the ssh-keygen command generates a pair of RSA keys by default, you can instruct it to generate ECDSA or Ed25519 keys by using the -t option. The ECDSA (Elliptic Curve Digital Signature Algorithm) offers better performance than RSA at the equivalent symmetric key strength. It also generates shorter keys. The Ed25519 public-key algorithm is an implementation of twisted Edwards curves that is more secure and also faster than RSA, DSA, and ECDSA.

    OpenSSH creates RSA, ECDSA, and Ed25519 server host keys automatically if they are missing. To configure the host key creation in RHEL, use the sshd-keygen@.service instantiated service. For example, to disable the automatic creation of the RSA key type:

    # systemctl mask sshd-keygen@rsa.service
    Note

    In images with cloud-init enabled, the ssh-keygen units are automatically disabled. This is because the ssh-keygen template service can interfere with the cloud-init tool and cause problems with host key generation. To prevent these problems the etc/systemd/system/sshd-keygen@.service.d/disable-sshd-keygen-if-cloud-init-active.conf drop-in configuration file disables the ssh-keygen units if cloud-init is running.

  • To exclude particular key types for SSH connections, comment out the relevant lines in /etc/ssh/sshd_config, and reload the sshd service. For example, to allow only Ed25519 host keys:

    # HostKey /etc/ssh/ssh_host_rsa_key
    # HostKey /etc/ssh/ssh_host_ecdsa_key
    HostKey /etc/ssh/ssh_host_ed25519_key
    Important

    The Ed25519 algorithm is not FIPS-140-compliant, and OpenSSH does not work with Ed25519 keys in FIPS mode.

Non-default port

  • By default, the sshd daemon listens on TCP port 22. Changing the port reduces the exposure of the system to attacks based on automated network scanning and therefore increase security through obscurity. You can specify the port using the Port directive in the /etc/ssh/sshd_config configuration file.

    You also have to update the default SELinux policy to allow the use of a non-default port. To do so, use the semanage tool from the policycoreutils-python-utils package:

    # semanage port -a -t ssh_port_t -p tcp <port_number>

    Furthermore, update firewalld configuration:

    # firewall-cmd --add-port <port_number>/tcp
    # firewall-cmd --remove-port=22/tcp
    # firewall-cmd --runtime-to-permanent

    In the previous commands, replace <port_number> with the new port number specified using the Port directive.

Root login

  • PermitRootLogin is set to prohibit-password by default. This enforces the use of key-based authentication instead of the use of passwords for logging in as root and reduces risks by preventing brute-force attacks.

    Warning

    Enabling logging in as the root user is not a secure practice because the administrator cannot audit which users run which privileged commands. For using administrative commands, log in and use sudo instead.

Using the X Security extension

  • The X server in Red Hat Enterprise Linux clients does not provide the X Security extension. Therefore, clients cannot request another security layer when connecting to untrusted SSH servers with X11 forwarding. Most applications are not able to run with this extension enabled anyway.

    By default, the ForwardX11Trusted option in the /etc/ssh/ssh_config.d/50-redhat.conf file is set to yes, and there is no difference between the ssh -X remote_machine (untrusted host) and ssh -Y remote_machine (trusted host) command.

    If your scenario does not require the X11 forwarding feature at all, set the X11Forwarding directive in the /etc/ssh/sshd_config configuration file to no.

Restricting access to specific users, groups, or domains

  • The AllowUsers and AllowGroups directives in the /etc/ssh/sshd_config configuration file server enable you to permit only certain users, domains, or groups to connect to your OpenSSH server. You can combine AllowUsers and AllowGroups to restrict access more precisely, for example:

    AllowUsers *@192.168.1.* *@10.0.0.* !*@192.168.1.2
    AllowGroups example-group

    The previous configuration lines accept connections from all users from systems in 192.168.1.* and 10.0.0.* subnets except from the system with the 192.168.1.2 address. All users must be in the example-group group. The OpenSSH server denies all other connections.

    The OpenSSH server permits only connections that pass all Allow and Deny directives in /etc/ssh/sshd_config. For example, if the AllowUsers directive lists a user that is not part of a group listed in the AllowGroups directive, then the user cannot log in.

    Note that using allowlists (directives starting with Allow) is more secure than using blocklists (options starting with Deny) because allowlists block also new unauthorized users or groups.

Changing system-wide cryptographic policies

  • OpenSSH uses RHEL system-wide cryptographic policies, and the default system-wide cryptographic policy level offers secure settings for current threat models. To make your cryptographic settings more strict, change the current policy level:

    # update-crypto-policies --set FUTURE
    Setting system policy to FUTURE
    Warning

    If your system communicates on the internet, you might face interoperability problems due to the strict setting of the FUTURE policy.

You can also disable only specific ciphers for the SSH protocol through the system-wide cryptographic policies. See the Customizing system-wide cryptographic policies with subpolicies section in the Security hardening document for more information.

To opt out of the system-wide cryptographic policies for your OpenSSH server, specify the cryptographic policy in a drop-in configuration file located in the /etc/ssh/sshd_config.d/ directory, with a two-digit number prefix smaller than 50, so that it lexicographically precedes the 50-redhat.conf file, and with a .conf suffix, for example, 49-crypto-policy-override.conf.

See the sshd_config(5) man page for more information.

To opt out of system-wide cryptographic policies for your OpenSSH client, perform one of the following tasks:

  • For a given user, override the global ssh_config with a user-specific configuration in the ~/.ssh/config file.
  • For the entire system, specify the cryptographic policy in a drop-in configuration file located in the /etc/ssh/ssh_config.d/ directory, with a two-digit number prefix smaller than 50, so that it lexicographically precedes the 50-redhat.conf file, and with a .conf suffix, for example, 49-crypto-policy-override.conf.

Additional resources

1.7. Connecting to a remote server using an SSH jump host

Use this procedure for connecting your local system to a remote server through an intermediary server, also called jump host.

Prerequisites

  • A jump host accepts SSH connections from your local system.
  • A remote server accepts SSH connections only from the jump host.

Procedure

  1. Define the jump host by editing the ~/.ssh/config file on your local system, for example:

    Host jump-server1
      HostName jump1.example.com
    • The Host parameter defines a name or alias for the host you can use in ssh commands. The value can match the real host name, but can also be any string.
    • The HostName parameter sets the actual host name or IP address of the jump host.
  2. Add the remote server jump configuration with the ProxyJump directive to ~/.ssh/config file on your local system, for example:

    Host remote-server
      HostName remote1.example.com
      ProxyJump jump-server1
  3. Use your local system to connect to the remote server through the jump server:

    $ ssh remote-server

    The previous command is equivalent to the ssh -J jump-server1 remote-server command if you omit the configuration steps 1 and 2.

Note

You can specify more jump servers and you can also skip adding host definitions to the configurations file when you provide their complete host names, for example:

$ ssh -J jump1.example.com,jump2.example.com,jump3.example.com remote1.example.com

Change the host name-only notation in the previous command if the user names or SSH ports on the jump servers differ from the names and ports on the remote server, for example:

$ ssh -J johndoe@jump1.example.com:75,johndoe@jump2.example.com:75,johndoe@jump3.example.com:75 joesec@remote1.example.com:220

Additional resources

  • ssh_config(5) and ssh(1) man pages.

1.8. Connecting to remote machines with SSH keys using ssh-agent

To avoid entering a passphrase each time you initiate an SSH connection, you can use the ssh-agent utility to cache the private SSH key. The private key and the passphrase remain secure.

Prerequisites

  • You have a remote host with SSH daemon running and reachable through the network.
  • You know the IP address or hostname and credentials to log in to the remote host.
  • You have generated an SSH key pair with a passphrase and transferred the public key to the remote machine.

Procedure

  1. Optional: Verify you can use the key to authenticate to the remote host:

    1. Connect to the remote host using SSH:

      $ ssh example.user1@198.51.100.1 hostname
    2. Enter the passphrase you set while creating the key to grant access to the private key.

      $ ssh example.user1@198.51.100.1 hostname
       host.example.com
  2. Start the ssh-agent.

    $ eval $(ssh-agent)
    Agent pid 20062
  3. Add the key to ssh-agent.

    $ ssh-add ~/.ssh/id_rsa
    Enter passphrase for ~/.ssh/id_rsa:
    Identity added: ~/.ssh/id_rsa (example.user0@198.51.100.12)

Verification

  • Optional: Log in to the host machine using SSH.

    $ ssh example.user1@198.51.100.1
    
    Last login: Mon Sep 14 12:56:37 2020

    Note that you did not have to enter the passphrase.

1.9. Additional resources