Issue

We have found that ActiveMQ broker stopped functioning, after reviewing the logs on the server we found that the broker instance had a java core and a heap space dump. The JON agent JVM also crashed on the same machine. The AMQ instance and JON agent were recycled but we are trying to find the cause of this problem as it may happens again.

Here is an error we found in Fuse MQ log file:

 - 5.7.0.fuse-71-047 | Transport Connection to: tcp://xxx.x.xxx.xxx:51568 failed: java.io.EOFException
2015-03-22 01:54:32,856 | WARN  | TCP Accept-44431 | tcp                              | sun.rmi.runtime.Log$LoggerLog     232 |  -  -  | RMI TCP Accept-44431: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=44431] throws
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
        at java.lang.Thread.startImpl(Native Method)[:1.6.0]
        at java.lang.Thread.start(Thread.java:914)[:1.6.0]
        at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)[:1.6.0]
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)[:1.6.0]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)[:1.6.0]
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:396)[:1.6.0]
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:353)[:1.6.0]
        at java.lang.Thread.run(Thread.java:761)[:1.6.0]
2015-03-22 01:54:32,866 | WARN  | 89)-10.251.3.168 | misc                             | .sun.jmx.remote.util.ClassLogger  209 |  -  -  | The server has decided to close this client connection

We are using IBM JDK on RHEL 6.5.