Warning message

log in to add comments or rate this document

Satellite server not sending latest Errata patches or package updates to clients: Taskomatic and tomcat java heap dumps "OutOfMemoryError"

Updated 2014-12-10T12:51:32+00:00

Issue

  • Red Hat Satellite 5.6 server is producing java heap dumps and it is causing to receive 503 errors on the Satellite web interface in addition to clogging our inboxes
  • Heap dumps are being put into system's /usr directory and thus filling up the root filesystem
  • Clients are failing to sync, update through yum, or receive latest errata patches from Satellite server due to taskomatic or tomcat out of memory errors on server
  • Satellite server slow or throwing out of memory exceptions after upgrade to 5.5 or 5.6
  • Trying to update client system, receiving "Error while executing packages action: empty transaction" messages from Satellite
  • Satellite server is not syncing yum repositories due to OOM exceptions
  • Taskomatic logs reflects errror:- java.lang.OutOfMemoryError
  • Catalina.out reflects error:- ERROR com.redhat.rhn.common.messaging.ActionExecutor - java.lang.OutOfMemoryError

Environment

  • Red Hat Satellite 5.x

Resolution

  • There problem was addressed by the errata RHBA-2014-1651. Make sure you have applied it.

  • If you are running a Satellite 5.5 or minor, you can apply the work around below:

Additional memory workarounds:
* The new minimum and maximum memory parameters for Taskomatic must to be reconfigured at /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf. Bugzilla#1037708 is currently open and is tracking a functionality of adding these data in /etc/rhn/rhn.conf.

  • Increase the taskomatic JVM maximum heap size from 512Mb to 1.3Gb to stop the heap dumps from occurring

  • Find the following values in /etc/rhn/default/rhn_taskomatic_daemon.conf (Note: In Satellite 5.5 and 5.6 the location of this conf file is /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf):

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=512
  • Change the value of wrapper.java.maxmemory to 1.3Gb. If needed, you can add more memory depending on your work-load. Just make sure to have enough memory available on the system.
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1331  
  • Restart the taskomatic daemon:
 #  service taskomatic restart

NOTE : it is better to restart the whole satellite if you also make changes to taskomatic :

# rhn-satellite restart
  • Increase the tomcat maximum heap size from 256Mb to 512Mb.

  • For Satellite server up to 5.5 edit /etc/tomcat5/tomcat5.conf or /etc/tomcat6/tomcat6.conf depending on your OS version ; if using Satellite servers 5.6 the file will instead be /etc/sysconfig/tomcat5 or /etc/sysconfig/tomcat6. Increase the -Xmx to a higher value such as 512Mb here:

JAVA_OPTS="$JAVA_OPTS -ea -Xms256m -Xmx512m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC"
  • Restart the tomcat daemon:

    • Satellite server running on Red Hat Enterprise Linux 5:
    #  service tomcat5 restart
    
    • Satellite server running on Red Hat Enterprise Linux 6:
    #  service tomcat6 restart
    
  • To have future heapdumps placed in a different directory to /usr, follow the instructions in this article: How do I change the directory where java heapdumps are written?

Root Cause

  • Currently, the minimum size is set to 256Mb and the maximum size is set to 512Mb in /etc/rhn/default/rhn_taskomatic_daemon.conf . Java typically uses 500Mb of the heap dump alone. Running out of memory on the heap in a java process (in this case taskomatic) can cause this error.

  • Currently, the minimum and the maximum size is set to 256Mb in /etc/tomcat5/tomcat5.conf

  • This error is documented fully under Java application "java.lang.OutOfMemoryError: GC overhead limit exceeded".

  • This issue was reported on bugzilla #1132398.

Diagnostic Steps

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received
...
1CICMDLINE     /usr/bin/java -Dibm.dst.compatibility=true -Xms256m -Xmx512m -Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/10.2.0.4/client64/lib:/usr ... -Dwrapper.jvmid=5 com.redhat.rhn.taskomatic.core.TaskomaticDaemon
...
2CIUSERARG               -Xms256m
2CIUSERARG               -Xmx512m

Here we can see that the JVM for 'com.redhat.rhn.taskomatic.core.TaskomaticDaemon' is encountering an OutOfMemory error and that the current memory limits for the JVM heap are 256MB for the startup size and 512MB for the maximum allowed size. Increasing the -Xmx setting will hopefully resolve the issue.
...

  • for tomcat we can see that java core file contains:
1CICMDLINE     /usr/lib/jvm/java/bin/java -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -ea -Xms256m -Xmx256m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar:/usr/share/java/mx4j/mx4j-impl.jar:/usr/share/java/mx4j/mx4j-jmx.jar -Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start

2CIUSERARG               -Xms256m
2CIUSERARG               -Xmx256m

Here we can see that the min and max JVM heap limit are set to 256Mb. To resolve the issue increase the max heap limit -Xmx to 512Mb.
* The messages in the /var/log/message file are:

/var/log/messages.1:Jan  9 01:52:15  java[27378]: JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20130109.015204.27378.0002.txt' in response to an event 
/var/log/messages.1:Jan  9 01:52:15  java[27378]: JVMDUMP032I JVM requested Snap dump using '/usr/share/tomcat5/Snap.20130109.015204.27378.0003.trc' in response to an event 
/var/log/messages.1:Jan  9 01:52:16  java[27378]: JVMDUMP032I JVM requested Heap dump using '/tmp/heapdump.20130109.015216.27378.0004.phd' in response to an event  
  • java.lang.OutOfMemoryError exception in /var/log/tomcat5/catalina.out file:
JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20130109.121341.10405.0011.txt' in response to an event
JVMDUMP010I Java dump written to /tmp/javacore.20130109.121341.10405.0011.txt
JVMDUMP032I JVM requested Snap dump using '/usr/share/tomcat5/Snap.20130109.121341.10405.0012.trc' in response to an event
JVMDUMP030W Cannot write dump to file /usr/share/tomcat5/Snap.20130109.121341.10405.0012.trc: Permission denied
JVMDUMP010I Snap dump written to /tmp/Snap.20130109.121341.10405.0012.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1572)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Exception in thread "Thread-7" ContainerBase.java:1559)
    at java.lang.Thread.run(Thread.java:736)
java.lang.OutOfMemoryError
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:777)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:771)
    at java.lang.Thread.uncaughtException(Thread.java:1213)
2013-01-09 12:13:44,158 [RHN Message Dispatcher] ERROR com.redhat.rhn.common.messaging.ActionExecutor - java.lang.OutOfMemoryError
  • Exception in /var/log/rhn/rhn_taskomatic_daemon.log file
INFO   | jvm 28   | 2013/10/03 14:05:15 | JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
INFO   | jvm 28   | 2013/10/03 14:05:15 | JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
INFO   | jvm 28   | 2013/10/03 14:05:15 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20131003.140515.9571.0010.phd' in response to an event
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20131003.140515.9571.0010.phd
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20131003.140515.9571.0012.trc' in response to an event
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20131003.140515.9571.0012.trc
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20131003.140515.9571.0011.txt' in response to an event
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20131003.140515.9571.0011.txt
INFO   | jvm 28   | 2013/10/03 14:05:20 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO   | jvm 28   | 2013/10/03 14:05:20 | Exception in thread "Thread-49" java.lang.OutOfMemoryError
  • Taskomatic reports java.lang.OutOfMemoryError: GC overhead limit exceeded
  • After downloading a new channel the following error is reported in taskomatic logs:
INFO   | jvm 1    | 2013/11/22 11:00:17 | 2013-11-22 11:00:17,134 [Thread-53] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating new repository metadata for channel 'rhel-x86_64-server-5'(sha1) 15661 packages, 2856 errata
INFO   | jvm 1    | 2013/11/22 11:00:50 | Exception in thread "Thread-53" java.lang.OutOfMemoryError: GC overhead limit exceeded