mod_proxy error End of file found: proxy: error reading status line from remote server

Solution Verified - Updated -

Environment

  • Red Hat JBoss Enterprise Web Server(EWS)
    • Apache httpd
    • mod_proxy
    • mod_cluster
  • Red Hat JBoss Enterprise Application Platform (EAP)
  • Red Hat JBoss Enterprise Portal Platform (EPP)

Issue

  • We are getting this error sporadically in one of our applications. Do you have any suggestions as to what is causing this or how to troubleshoot the error?

    [error] [client 10.20.30.40] (70014)End of file found: proxy: error reading status line from remote server host.example.com, referer: https://host.example.com/MainPortal/#UserBlog
[error] [client 10.20.30.40] proxy: Error reading from remote server returned by /MainPortal/blogs/userblog, referer: https://host.example.com/MainPortal/#UserBlog

    and then a 502 response code is given for the response

Resolution

  • These can happen either because of network problems or problems on the backend server or because of a race condition in the proxy code. If the back end is a java based server (Tomcat or JBoss), see Java application unresponsive for troubleshooting backend issues that are inducing such proxy connection errors and timeouts.
  • The bug that caused this was a race condition that was fixed in 2.2.10 so that shouldn't be causing any issues in the EWS 1.0.1 (which ships with httpd 2.2.14). 
  • Try increasing the Timeout or ProxyTimeout to see if that mitigates the issue (say to 600 then 900).   If using mod_cluster, nodeTimeout can be set in mod-cluster-jboss-beans.xml. It is possible that backend/application code is misbehaving and delaying responses.
  • Try adding SetEnv proxy-nokeepalive 1 to your httpd config to disable keepalive to avoid any issues caused by failed keepalive connections.

Root Cause

  • https://issues.apache.org/bugzilla/show_bug.cgi?id=37770

  • The following implies End of file found -- TCP FIN was sent from server without providing a response:

    [error] [client 10.20.30.40] (70014)End of file found: proxy: error reading status line from remote server mydomain.test.com, referer: https://mydomain.test.com/

  • The following implies the timeout specified has expired -- indicates network, backend, or keepalive issues:

    [error] [client 10.20.30.40] (7007)The timeout specified has expired: proxy: error reading status line from remote server  mydomain.test.com, referer: https://mydomain.test.com/

    And such a timeout can cause corresponding ClientAbortExceptions on JBoss:

    java.io.IOException: ClientAbortException:  java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:401)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:424)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:413)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

Diagnostic Steps

  • Verify the configuration has been optimized using the Red Hat Load Balancer Configuration Tool https://access.redhat.com/labs/lbconfig if you have not done so already.
  • If after optimization the problem still persists, provide the following:
  1. Apache error_log and the corresponding access_log (check what type of request was it ? how many bytes transferred)
  2. Version of Apache httpd  ( you can do this by executing the command httpd -V . note upper case V)
  3. server.xml from the JBoss EAP instance
  4. httpd.conf
  • Verify EAP shutdown has not been initiated. This can be verified by looking in the server log for entry
JBoss Shutdown Hook) Runtime shutdown hook called, forceHalt: true

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments