JVM crash triggered by libtcnative

Solution Verified - Updated -

Issue

  • The JVM appears to be crashing when using the native connectors. The fatal error log shows the following:
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode linux-amd64 )
    # Problematic frame:
    # C  [libtcnative-1.so+0x12a9a]  Java_org_apache_tomcat_jni_Socket_sendbb+0x5a
    
    ---------------  T H R E A D  ---------------
    
    Current thread (0x000000004de56000):  JavaThread "Finalizer" daemon [_thread_in_native, id=29043, stack(0x000000004157e000,0x000000004159f000)]
    
    siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000040
    
    Registers:
    RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0x000000004159cfe8
    RSP=0x000000004159cfe0, RBP=0x00002aab5851cf68, RSI=0x0000000000000000, RDI=0x0000000000000000
    R8 =0x000000000000000e, R9 =0x00002b7fd92f2c38, R10=0x00002aaaaea28959, R11=0x00002b7fd9014220
    R12=0x000000000000000e, R13=0x0000000000000000, R14=0x000000004159cfe8, R15=0x000000004de56000
    RIP=0x00002aab4fd50a9a, EFL=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
      TRAPNO=0x000000000000000e
    
    Top of Stack: (sp=0x000000004159cfe0)
    0x000000004159cfe0:   00002aaadaf1fb48 000000000000000e
    0x000000004159cff0:   00002aaab51e3e48 000000004159d088
    Stack: [0x000000004157e000,0x000000004159f000],  sp=0x000000004159cfe0,  free space=7b0000000000000018k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    C  [libtcnative-1.so+0x12a9a]  Java_org_apache_tomcat_jni_Socket_sendbb+0x5a
    
    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    j  org.apache.tomcat.jni.Socket.sendbb(JII)I+0
    J  org.apache.coyote.ajp.AjpAprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
    J  org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
    j  org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+86
    j  org.apache.catalina.connector.OutputBuffer.flush()V+2
    j  org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
    j  javax.imageio.stream.MemoryCacheImageOutputStream.flushBefore(J)V+47
    j  javax.imageio.stream.MemoryCacheImageOutputStream.close()V+15
    j  javax.imageio.stream.ImageInputStreamImpl.finalize()V+8
    v  ~StubRoutines::call_stub
    J  java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V
    J  java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V
    J  java.lang.ref.Finalizer$FinalizerThread.run()V
    v  ~StubRoutines::call_stub
    
  • Our application is the google implementation for the Comet protocol. This is just a technique for allowing a web server to push data to a browser (also known as Ajax Push). JVM crashed on startup with the following when using tcnative

    >Stack: [0x000000004d485000,0x000000004d586000],  sp=0x000000004d582200,  free space=3f40000000000000018k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    C  [libtcnative-1.so+0x1459a]  Java_org_apache_tomcat_jni_Socket_sendbb+0x5a
    
    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    j  org.apache.tomcat.jni.Socket.sendbb(JII)I+0
    j  org.apache.coyote.ajp.AjpAprProcessor.flush()V+22
    j  org.apache.coyote.ajp.AjpAprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+66
    j  org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
    j  org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+94
    j  org.apache.catalina.connector.OutputBuffer.flush()V+2
    j  org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
    j  net.myapp.gwt.comet.server.deflate.DeflaterOutputStream.flush()V+15
    j  net.myapp.gwt.comet.server.impl.CountOutputStream.flush()V+11
    j  sun.nio.cs.StreamEncoder.implFlush()V+15
    j  sun.nio.cs.StreamEncoder.flush()V+12
    j  java.io.OutputStreamWriter.flush()V+4
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.flush()V+25
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.terminate()V+12
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.tryTerminate()V+1
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.initiate()V+143
    j  net.myapp.gwt.comet.server.impl.IEHTMLFileCometServletResponse.initiate()V+1
    j  net.myapp.gwt.comet.server.CometServlet.doCometImpl(Lnet/zschech/gwt/comet/server/impl/CometServletResponseImpl;)V+1
    j  net.myapp.gwt.comet.server.CometServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+90
    j  javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+35
    
  • Tomcat native crashes destroying an APR connection when using java imaging:
Stack: [0x00007f5261f6d000,0x00007f526206e000],  sp=0x00007f526206c580,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libapr-1.so.0+0x188d0]  signed char+0x20

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.tomcat.jni.Pool.destroy(J)V+0
j  org.apache.tomcat.util.net.AprEndpoint$Sendfile.add(Lorg/apache/tomcat/util/net/AprEndpoint$SendfileData;)Z+92
J  org.apache.coyote.http11.Http11AprProcessor.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;
j  org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;+39
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+155
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
  • Or
Stack: [0x00007f239c2a7000,0x00007f239c3a8000],  sp=0x00007f239c3a65d0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libapr-1.so.0.3.9+0x188bd]  signed char+0xd

[error occurred during error reporting (printing native stack), id 0xb]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.tomcat.jni.Socket.destroy(J)V+0
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+128
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
  • Tomcat native crashes setting timeouts on a socket when used in conjunection with AWT:
Stack: [0x00007fd9ff9fa000,0x00007fd9ffafb000],  sp=0x00007fd9ffaf9750,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtcnative-1.so+0x12a5b]  Java_org_apache_tomcat_jni_Socket_timeoutSet+0x2b
j  org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(J)Z+63
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+153
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x4e14a0]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1e0
V  [libjvm.so+0x711a39]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19
V  [libjvm.so+0x4e0de6]  JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*)+0x126
V  [libjvm.so+0x4e0e97]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Thread*)+0x47
V  [libjvm.so+0x574c00]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0x81c491]  JavaThread::run()+0x121
V  [libjvm.so+0x712eef]  java_start(Thread*)+0x13f

Environment

  • JBoss Enterprise Application Platform (EAP) 5
  • JBoss Enterprise Web Server (EWS)
    • Tomcat
  • JBoss Native

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In