java.security.InvalidAlgorithmParameterException for SSL when using jdk 1.8.0_162+ and FIPS mode in NSS database

Solution Verified - Updated -

Issue

  • We are getting the following exception when using jdk 1.8.0_162+ and employing SSL with FIPS mode in NSS database
ERROR [org.xnio.nio] (default I/O-6) XNIO000011: Task io.undertow.protocols.ssl.SslConduit$5$1@76e3aa10 failed with an exception: java.lang.RuntimeException: java.security.InvalidAlgorithmParameterException: Key format must be RAW
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1527) [jsse.jar:1.8.0_162]
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) [jsse.jar:1.8.0_162]
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) [jsse.jar:1.8.0_162]
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) [jsse.jar:1.8.0_162]
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) [rt.jar:1.8.0_162]
    at io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:751)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:648)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$5$1.run(SslConduit.java:1048)
    at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) [xnio-nio-3.5.4.Final-redhat-1.jar:3.5.4.Final-redhat-1]
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) [xnio-nio-3.5.4.Final-redhat-1.jar:3.5.4.Final-redhat-1]
Caused by: java.security.ProviderException: java.security.InvalidAlgorithmParameterException: Key format must be RAW
    at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1273) [jsse.jar:1.8.0_162]
    at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183) [jsse.jar:1.8.0_162]
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301) [jsse.jar:1.8.0_162]
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) [jsse.jar:1.8.0_162]
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) [jsse.jar:1.8.0_162]
  • This is occurring regardless of TLS version employed

Environment

  • Red Hat JBoss Enterprise Application Platform 7
  • Using an SSL connector
  • using FIPS mode in NSS database

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.