OpenJDK "InvalidKeyException: RSA key must be at least 1023 bytes" running in FIPS mode

Solution In Progress - Updated -

Issue

  • The attached Test.java application runs fine on JDK11 u16.1:
$ java Test
HTTP Response: 200
Successfully connected

But throws the following exception on JDK8 u345:

$ java Test
Connect HTTPS: Unsupported signature algorithm: rsa_pss_rsae_sha256
javax.net.ssl.SSLException: Unsupported signature algorithm: rsa_pss_rsae_sha256
    at sun.security.ssl.Alert.createSSLException(Alert.java:133)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
    at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:298)
    at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeConsumer.consume(ECDHServerKeyExchange.java:527)
    at sun.security.ssl.ServerKeyExchange$ServerKeyExchangeConsumer.consume(ServerKeyExchange.java:122)
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1572)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
    at Test.connectHTTPS(Test.java:52)
    at Test.main(Test.java:25)
Caused by: java.security.InvalidKeyException: RSA key must be at least 1023 bytes

Environment

  • Red Hat Enterprise Linux in FIPS mode
  • Red Hat build of OpenJDK

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content