SSH Weirdness When FIPS Mode Enabled

Latest response

A customer recently asked me to help them sort out getting FIPS mode enabled on some of their systems. As per normal, before sending a procedure over, I took a test system and walked through the procedures. I also ensured that our standard MACs and Ciphers directives were commented-out so that the SSHD would allow connections at all. Everything appeared to be functional, so I left my system configured for FIPS.

Today, I was trying to work with one of our GIT repos that only allows SSH-based repo-access. GIT attempted to connect via SSH and the SSH client asked me for my key's password. I entered it, and it told me my password was incorrect. I tried again, outside the context of GIT, and got the same error (even though I verified I wasn't fat-fingering anything). So, I rebooted my host out of FIPS mode. Once it came back from reboot plain and GIT-wrapped SSH key-unlocks worked as previous.

Anyone run into anything similar? I'm still Googling about, but haven't found the right combination of terms to dig up something relevant/useful. Running the SSH daemon and client in verbose/debug modes hasn't given me anything useful, either.


Hi Tom. I'm assuming you transferred this ssh key-pair from some other system? Perhaps you've had it a while? I'm guessing it's a DSA key.

If you generate a new key (using ssh-keygen with no options) on any modern system (even RHEL 5.11), the key should be usable in FIPS mode. A quick check shows that all of the following fail in FIPS mode:

  • ssh-keygen -b 768
  • ssh-keygen -t rsa1
  • ssh-keygen -t dsa

I use an RSA (not RSA1) key, but it was a key I generated somewhere in the RHEL 5.5 days. So, it may be a bit-depth issue. Was going to try explicitly generating a larger key to see if that made a difference, but haven't had a chance to test, yet.

Does FIPS want greater than 1024 key-length?

ssh-keygen -l -f ~/.ssh/id_rsa
1024 78:04:97:1d:d3:1d:96:a8:5b:d6:9b:2e:7e:6a:31:c0 ${HOME}/.ssh/ (RSA)

Or is it purely a matter that it's missing some FIPS extensions?

Ok, it was key-length:

  1. Fired up an AWS instance
  2. Logged in and generated password-protected, 768- 1024- and 2048-bit RSA keys
  3. Configured host for FIPS mode
  4. Rebooted
  5. Logged in and used each key to ssh 0

The 768- and 1024-bit keys failed to let me use my password, the 2048 let me go about my way.

Bleah: now I gotta go upgrade a bunch of keys.

Ahhhh it must be an encrypted then ... why didn't I think of that? Point being: it's probably aes cipher but with an md5 digest. After all, even the openssl enc -aes-256-cbc command in RHEL7 defaults to using md5 to hash the provided passphrase. When I get a min I'll go check, but that's almost certainly your answer.

EDIT: Oh I just saw your most recent key-length post. Hmmmm. I know that 1024-bit unencrypted keys are fine because I did that earlier. Now I'm all curious.

EDIT2: Okay I just tried an en encrypted 1024-bit rsa key on a RHEL 6.6 FIPS machine and that was no problem.

EDIT3: Ahhh there we go. Generated an encrypted key on RHEL 5 and finally -- that fails to decrypt in FIPS mode when I type the right password. So yeah, it's like I originally said: md5 digest.

On further pondering and a little more inspection: it's might not be about md5; it could just be about the encryption cipher used instead. Either way, it aint gonna happen.

EDIT: Finally got a minute to go back to my RHEL 6 machine and it's DEFINITELY due to md5. Here we can see what the cipher was for my RHEL 5 encrypted key:

[r66]# head -3 id_rsa_rhel5 
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,A32820AB8412E6BF

And here we can see that that particular cipher IS available in FIPS mode.

[r66]# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
[r66]# openssl enc -des-ede3-cbc -a -k mypass -md sha256 <<<dummydata

Also, to be clear, you could re-encrypt such a key (change the password) to clear this up.

@Tom I did a little research and here's my executive summary:

  • Encrypted SSH keys generated by a RHEL 6 system in FIPS mode (and presumably other versions of RHEL/Fedora) automatically use an alternate PKCS8 format that doesn't make use of MD5. This isn't mentioned anywhere in the RHEL 6 ssh-keygen man page; however, newer versions (e.g., RHEL 7) have a -m option that let's you explicitly set the format ... but only works when you're doing import/export and I think it doesn't work with encrypted keys so I didn't even go down that hole.

  • To be clear, encrypted SSH keys generated with default options by every version of ssh-keygen currently available in RHEL & Fedora will be unusable in FIPS mode.

  • You can read in a standard key (on a non-FIPS system) and convert it to FIPS-compatible PKCS8 format by using the following command:

    openssl pkcs8 -topk8 -v2 aes128 -in id_rsa -out new-id_rsa

    Note that if you pass it an encrypted key, it will prompt for the current password as well as giving you the option to type a password for encryption of the new key. (Of course there's also -passin and -passout if you don't mind leaking your secrets to the rest of the system.)

I wrote FIPS mode can't decrypt existing passphrase-protected ssh keys to document all this. :)

Nice writeup.

Of course, now I need to go back and rerun the tests I did (unless, by selecting 2048-bits it ssh-keygen also selected different default-algorithms?) - since I'd generated my three test keys prior to enabling FIPS mode.

I just ran into this issue and I'm trying to understand what is going on. I'm just a part time Linux admin here...

So I'm able to use ssh keys from RHEL6 VM, which doesn't have FIPS enabled, to a RHEL7 VM which does have FIPS enabled. No issues.

Trying to use SSH keys from a RHEL7 VM that has FIPS to another RHEL7 VM that also has FIPS and I get the error above.

This is what I did to make it work.

Backup any private/public keys first before running the command below.

ssh-keygen -t rsa -b 2048

Would anyone advise larger bit size?

From there copies the .pub key to the other RHEL7 VM and placed in under /home//.ssh

Then output the .pub key to the authorized_keys file on the host that you are trying to log into

cat | ssh <user_name_here>@<ip_address_here> 'cat >> .ssh/authorized_keys' 

I don't think this will cause any long term issues...I found some of this info on a thread @ stackexchange