SSL reverse proxy to CXF endpoint on JBoss Fuse 6.1 causes high cpu
Environment
JBoss Fuse 6.1
Apache HTTPD 2.x
Issue
When a long running request to a CXF Web Service running on Fuse takes longer than the mod_ssl SSLCacheTimeout we get high CPU on the Fuse Java process.
Steps to reproduce:
1) Start with a fresh install of JBoss Fuse 6.1
2) Unzip jetty-issue-demo.zip
3) Set the JKS path in jetty-issue-demo.xml (JKS file provided in zip)
4) build jetty-issue-demo with the following command:
mvn clean install
5) cd $FUSE_HOME
6) uncomment admin user in etc/user.properties
7) install feature camel-jackson
8) install feature camel-mvel
9) install wrap:mvn:org.jasypt/jasypt-spring31/1.9.0
10) install jetty-issue-demo
11) verify the CXF endpoint is deployed by running:
cxf:list-endpoints
12) Install Apache HTTPD (tested with version 2.4.10)
13) Install Apache mod_ssl module
14) Generate a key using the instructions provided here
15) Copy the following configuration to a new file in conf.d:
LoadModule ssl_module modules/mod_ssl.so
Listen 8443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 300
#----------------------------------
# 443
#----------------------------------
NameVirtualHost *:8443
<VirtualHost *:8443>
ServerName sslproxy.usersys.redhat.com
ServerAlias sslproxy.usersys.redhat.com
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/sslproxy.usersys.redhat.com.crt
SSLCertificateKeyFile /etc/pki/tls/private/sslproxy.usersys.redhat.com.key
ErrorLog logs/8443_error_log
# Host Time "Request String" Bytes Status ServerActioningRequest ThisServerName TimeTakenMicroseconds(/1000000)
CustomLog logs/8443_balancer_log "%h %t \"%r\" %b %s %{BALANCER_WORKER_NAME}e %v %D"
LogLevel warn
SSLProxyEngine On
# CXF REST endpoint
<Proxy balancer://cxfrs>
BalancerMember https://localhost:9292/demo retry=10
</Proxy>
ProxyPass /demo balancer://cxfrs
</VirtualHost>
16) Start Apache HTTPD, and verify you can get high cpu by accessing the application on the following url:
Resolution
There is a bug long running request to a CXF Web Service may lead to high CPU opened.
It is fixed in Fuse 6.1 R2P4.
Diagnostic Steps
When checking the OS High CPU thread against the corresponding Java thread, the threads causing high CPU look like this:
"qtp680330057-225" prio=10 tid=0x00007f42dc732000 nid=0x33f3 runnable [0x00007f42a8e40000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:811)
at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1123)
at org.eclipse.jetty.server.AbstractHttpConnection$Output.sendContent(AbstractHttpConnection.java:1237)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination$JettyOutputStream.copyFrom(JettyHTTPDestination.java:414)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:105)
at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.copyFrom(AbstractHTTPDestination.java:706)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:105)
at org.apache.cxf.helpers.IOUtils.copyAndCloseInput(IOUtils.java:113)
at org.apache.cxf.jaxrs.provider.BinaryDataProvider.copyInputToOutput(BinaryDataProvider.java:136)
at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:105)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1317)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:284)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:157)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:86)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
- locked <0x00000000e2783df8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:81)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
- locked <0x00000000e27424c0> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.phase.PhaseInterceptorChain.resume(PhaseInterceptorChain.java:242)
- locked <0x00000000e27424c0> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:78)
- locked <0x00000000e27424c0> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:355)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:319)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:72)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:193)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handleAsync(Server.java:410)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:519)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
"qtp680330057-223" prio=10 tid=0x00007f42dc72d800 nid=0x33f1 runnable [0x00007f42a9041000]
java.lang.Thread.State: RUNNABLE
at sun.security.ssl.SSLEngineImpl.getHandshakeStatus(SSLEngineImpl.java:548)
at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:351)
- locked <0x00000000e317d4b0> (a org.eclipse.jetty.io.nio.SslConnection)
at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:48)
at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:690)
at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:697)
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)
at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1123)
at org.eclipse.jetty.server.AbstractHttpConnection$Output.sendContent(AbstractHttpConnection.java:1237)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination$JettyOutputStream.copyFrom(JettyHTTPDestination.java:414)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:105)
at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.copyFrom(AbstractHTTPDestination.java:706)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:105)
at org.apache.cxf.helpers.IOUtils.copyAndCloseInput(IOUtils.java:113)
at org.apache.cxf.jaxrs.provider.BinaryDataProvider.copyInputToOutput(BinaryDataProvider.java:136)
at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:105)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1317)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:284)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:157)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:86)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
- locked <0x00000000e354cda8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:81)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
- locked <0x00000000e31921c8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.phase.PhaseInterceptorChain.resume(PhaseInterceptorChain.java:242)
- locked <0x00000000e31921c8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:78)
- locked <0x00000000e31921c8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:355)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:319)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:72)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:193)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handleAsync(Server.java:410)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:519)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Close
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.