java.security.InvalidAlgorithmParameterException for SSL when using jdk 1.8.0_162+ and FIPS mode in NSS database
Issue
- We are getting the following exception when using jdk 1.8.0_162+ and employing SSL with FIPS mode in NSS database
ERROR [org.xnio.nio] (default I/O-6) XNIO000011: Task io.undertow.protocols.ssl.SslConduit$5$1@76e3aa10 failed with an exception: java.lang.RuntimeException: java.security.InvalidAlgorithmParameterException: Key format must be RAW
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1527) [jsse.jar:1.8.0_162]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) [jsse.jar:1.8.0_162]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) [jsse.jar:1.8.0_162]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) [jsse.jar:1.8.0_162]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) [rt.jar:1.8.0_162]
at io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:751)
at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:648)
at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
at io.undertow.protocols.ssl.SslConduit$5$1.run(SslConduit.java:1048)
at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) [xnio-nio-3.5.4.Final-redhat-1.jar:3.5.4.Final-redhat-1]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) [xnio-nio-3.5.4.Final-redhat-1.jar:3.5.4.Final-redhat-1]
Caused by: java.security.ProviderException: java.security.InvalidAlgorithmParameterException: Key format must be RAW
at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1273) [jsse.jar:1.8.0_162]
at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183) [jsse.jar:1.8.0_162]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301) [jsse.jar:1.8.0_162]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) [jsse.jar:1.8.0_162]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) [jsse.jar:1.8.0_162]
- This is occurring regardless of TLS version employed
Environment
- Red Hat JBoss Enterprise Application Platform 7
- Using an SSL connector
- using FIPS mode in NSS database
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.