NullPointerException in InternalOutputBuffer.realWriteBytes

Solution Unverified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x

Issue

  • We see a NullPointerException in InternalOutputBuffer.realWriteBytes, for example:
Caused by: java.lang.NullPointerException
    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124)
    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)
    at org.apache.coyote.Response.doWrite(Response.java:567)
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:397)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:425)
    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:414)
    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
  • Migration Issue from JBoss AS 7.1.1-Final to EAP 6.3. In server.log we found below exception:
Caused by: java.net.SocketException: Connection reset by peer: socket write error
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:706) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:730) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:543) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.Response.doWrite(Response.java:594) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:421) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
.......
16:33:53,355 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (http-127.0.0.1/127.0.0.1:8443-2) : ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:426) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
......
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:706) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:730) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:543) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.coyote.Response.doWrite(Response.java:594) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:421) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
    ... 29 more

Resolution

  • Ensure servlet output streams are not cached and reused beyond the scope of a single request.
  • The issue can be resolved by degrading the jsf version of JBoss 7.1.1 final.

Root Cause

  • InternalOutputBuffer.realWriteBytes would throw a NullPointerException there due to a null outputstream. This outputstream would be nulled out when the response and its buffers are recycled after completion of a request. So this is a likely indication that the application has grabbed some servlet stream from a response object and has cached and reused it across the scope of the request that the stream applies to. This could have even more devastating effects besides the NPE since you're trying to write to some stream that may now be recycled and reused by some other connection/request.

Diagnostic Steps

  • A way we could test for further confirmation is by setting the following in your java options:

    -Dorg.apache.catalina.connector.RECYCLE_FACADES=true

  • This would recycle the servlet output stream itself, so if the stream is reused after being recycled upon request completion, this will mean the exception we'd get with the above property is:

        Caused by: java.lang.NullPointerException
            at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

Migration Issue from AS 7.1.1-Final to EAP 6.3

  • Please note that JBoss AS 7.4.0 Final is the right version to migrate to EAP 6.3.0 not AS 7.1.1-Final as per KB article.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.