Skip to navigation

Warning message

log in to add comments or rate this document

Taskomatic and tomcat produces heapdumps on RHN Satellite for "out of memory" exceptions

Updated 2014-01-24T15:08:29+00:00

Issue

  • Our satellite server is producing java heap dumps and it is causing us to receive 503 errors on the Satellite web interface in addition to clogging our inboxes
  • Heapdumps are being put into /usr and thus filling up the root filesystem
  • The messages in the /var/log/message file are:
/var/log/messages.1:Jan  9 01:52:15  java[27378]: JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20130109.015204.27378.0002.txt' in response to an event 
/var/log/messages.1:Jan  9 01:52:15  java[27378]: JVMDUMP032I JVM requested Snap dump using '/usr/share/tomcat5/Snap.20130109.015204.27378.0003.trc' in response to an event 
/var/log/messages.1:Jan  9 01:52:16  java[27378]: JVMDUMP032I JVM requested Heap dump using '/tmp/heapdump.20130109.015216.27378.0004.phd' in response to an event  
  • java.lang.OutOfMemoryError exception in /var/log/tomcat5/catalina.out file:
JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20130109.121341.10405.0011.txt' in response to an event
JVMDUMP010I Java dump written to /tmp/javacore.20130109.121341.10405.0011.txt
JVMDUMP032I JVM requested Snap dump using '/usr/share/tomcat5/Snap.20130109.121341.10405.0012.trc' in response to an event
JVMDUMP030W Cannot write dump to file /usr/share/tomcat5/Snap.20130109.121341.10405.0012.trc: Permission denied
JVMDUMP010I Snap dump written to /tmp/Snap.20130109.121341.10405.0012.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1572)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Exception in thread "Thread-7" ContainerBase.java:1559)
    at java.lang.Thread.run(Thread.java:736)
java.lang.OutOfMemoryError
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:777)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:771)
    at java.lang.Thread.uncaughtException(Thread.java:1213)
2013-01-09 12:13:44,158 [RHN Message Dispatcher] ERROR com.redhat.rhn.common.messaging.ActionExecutor - java.lang.OutOfMemoryError
  • Exception in /var/log/rhn/rhn_taskomatic_daemon.log file
INFO   | jvm 28   | 2013/10/03 14:05:15 | JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
INFO   | jvm 28   | 2013/10/03 14:05:15 | JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
INFO   | jvm 28   | 2013/10/03 14:05:15 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20131003.140515.9571.0010.phd' in response to an event
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20131003.140515.9571.0010.phd
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20131003.140515.9571.0012.trc' in response to an event
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20131003.140515.9571.0012.trc
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20131003.140515.9571.0011.txt' in response to an event
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20131003.140515.9571.0011.txt
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO   | jvm 28   | 2013/10/03 14:05:20 | Exception in thread "Thread-49" java.lang.OutOfMemoryError

Environment

  • Red Hat Network Satellite 5.3
  • Red Hat Network Satellite 5.4
  • Red Hat Network Satellite 5.5
  • Red Hat Network Satellite 5.6

Resolution

  • Until the version spacewalk-taskomatic-2.0.2-50 and spacewalk-java-config-2.0.2-50 it is not possible to change the Taskomatic memory at /etc/rhn/rhn.conf. This implies that each RHN Satellite update, the new minimum and maximum memory parameters for Taskomatic must to be reconfigured at /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf. The bugzilla#1057655 is tracking this functionality.

  • Increase the taskomatic JVM maximum heap size from 512Mb to 1.3Gb to stop the heap dumps from occurring

  • Find the following values in /etc/rhn/default/rhn_taskomatic_daemon.conf (Note: In Satellite 5.5 and 5.6 the location of this conf file is /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf):

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=512
  • Change the value of wrapper.java.maxmemory to 1.3Gb, like so:
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1331  
  • Restart the taskomatic daemon:
 #  service taskomatic restart
  • Increase the tomcat maximum heap size from 256Mb to 512Mb.

  • For Satellite server running on Red Hat Enterprise Linux 5 edit /etc/tomcat5/tomcat5.conf file and for Satellite server running on Red Hat Enterprise Linux 6 edit /etc/tomcat6/tomcat6.conf and increase the -Xmx to 512Mb:

JAVA_OPTS="$JAVA_OPTS -ea -Xms256m -Xmx512m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC"
  • Restart the tomcat daemon:

    • Satellite server running on Red Hat Enterprise Linux 5:
    #  service tomcat5 restart
    
    • Satellite server running on Red Hat Enterprise Linux 6:
    #  service tomcat6 restart
    
  • To have future heapdumps placed in a different directory to /usr, follow the instructions in this article: How do I change the directory where java heapdumps are written?

Root Cause

  • Currently, the minimum size is set to 256Mb and the maximum size is set to 512Mb in /etc/rhn/default/rhn_taskomatic_daemon.conf . Java typically uses 500Mb of the heap dump alone. Running out of memory on the heap in a java process (in this case taskomatic) can cause this error.

  • Currently, the minimum and the maximum size is set to 256Mb in /etc/tomcat5/tomcat5.conf

Diagnostic Steps

  • Collecting the heap dumps and looking at the javacore-*.txt, we see the following:
1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received
...
1CICMDLINE     /usr/bin/java -Dibm.dst.compatibility=true -Xms256m -Xmx512m -Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/10.2.0.4/client64/lib:/usr ... -Dwrapper.jvmid=5 com.redhat.rhn.taskomatic.core.TaskomaticDaemon
...
2CIUSERARG               -Xms256m
2CIUSERARG               -Xmx512m

Here we can see that the JVM for 'com.redhat.rhn.taskomatic.core.TaskomaticDaemon' is encountering an OutOfMemory error and that the current memory limits for the JVM heap are 256MB for the startup size and 512MB for the maximum allowed size. Increasing the -Xmx setting will hopefully resolve the issue.
...

  • for tomcat we can see that java core file contains:
1CICMDLINE     /usr/lib/jvm/java/bin/java -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -ea -Xms256m -Xmx256m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar:/usr/share/java/mx4j/mx4j-impl.jar:/usr/share/java/mx4j/mx4j-jmx.jar -Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start

2CIUSERARG               -Xms256m
2CIUSERARG               -Xmx256m

Here we can see that the min and max JVM heap limit are set to 256Mb. To resolve the issue increase the max heap limit -Xmx to 512Mb.