In mod_jk why am I seeing "[warn] awaited reply cpong received [3|6] instead" ?

Solution Verified - Updated -

Environment

  • Windows Internet Information Services (IIS) 6.0
  • mod_jk 1.2.28
  • isapi redirector
  • iPhone client

Issue

  • The following warning in the mod_jk isapi redirector logging when an iPhone makes an http POST request

    [warn] jk_ajp_common.c (883): awaited reply cpong, received 6 instead
[debug] jk_connect.c (681): About to shutdown socket 668
[debug] jk_connect.c (732): Shutdown socket 668 and read 306 lingering bytes
[info] jk_ajp_common.c (1442): (worker1) failed sending request, socket -1 prepost cping/cpong failure (errno=0)

  • Socket read errors in the JBoss logging.

  • Under what circumstances would the following warn message happen?

    [warn] jk_ajp_common.c (883): awaited reply cpong, received 3 instead

Resolution

This warning is known to present itself under certain conditions highlighted in the Root cause.  To resolve this warning upgrade to the latest mod_jk 1.2.31 found in the EAP 5.1.1 native which you can download here: https://access.redhat.com/jbossnetwork/restricted/listSoftware.html?downloadType=distributions&product=appplatform&version=5.1.1 and includes changes so cping can do a dummy read to consume the extra data in the network buffer that the client does not consume due to the previously broken response. This was already being done for other data, but not for cping.

Root Cause

  • The " received 3" is a JK_AJP13_SEND_BODY_CHUNK packet. mod_proxy_ajp/mod_jk is waiting for a 9 (AJP13_CPONG_REPLY). So this is a protocol error.
  • The "received  6" refers to a "Get Body Chunk" AJP protocol packet from the backend server to the web server. The backend server is trying to get further data from the request.
  • Most likely the upstream client initiated a connection close, yet it wasn't done cleanly (e.g. a hard socket close), so the connection close was not propagated to the backend.
  • Data from the previous write is in the TCP send buffer and not yet delivered to mod_jk across the wire when a new cping request comes.
  • The socket read errors in the backend logging are due to the upstream connection being closed when the backend is trying to do further reading.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments