The satellite services of Red Hat Satellite 5 fail to start with java error messages

Solution Verified - Updated -

Environment

  • Red Hat Satellite Server 5.x
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Multiple java packages installed on system

Issue

  • After running yum on Red Hat Satellite 5 various satellite services fail to start with "unrecognized option" errors

  • How to restore the JVM of a satellite 5 instance to IBM

  • The satellite services of Red Hat Satellite 5 fail to start with java error messages

  • The satellite services of Red Hat Satellite 5 fail to start because they try to start the wrong java virtual machine (jvm)

  • Taskomatic jobs not completing ( kickstart trees unavailable or other items not run)

STATUS | wrapper  | 2012/10/10 17:56:15 | Launching a JVM...
INFO   | jvm 5    | 2012/10/10 17:56:15 | Unrecognized option: -Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
INFO   | jvm 5    | 2012/10/10 17:56:15 | Could not create the Java virtual machine.
ERROR  | wrapper  | 2012/10/10 17:56:15 | JVM exited while loading the application.
FATAL  | wrapper  | 2012/10/10 17:56:15 | There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
FATAL  | wrapper  | 2012/10/10 17:56:15 |   There may be a configuration problem: please check the logs.
STATUS | wrapper  | 2012/10/10 17:56:16 | <-- Wrapper Stopped
  • Taskomatic service is nor running :
# service taskomatic status
RHN Taskomatic is not running.
  • Tomcat throws the error message on catalina.out:
Aug 26, 2013 6:45:52 a.m. org.apache.catalina.loader.WebappClassLoader clearReferencesRmiTargets
INFO: Failed to find class sun.rmi.transport.Target to clear context class loader. This is expected on non-Sun JVMs.
java.lang.ClassNotFoundException: sun.rmi.transport.Target not found in org.apache.catalina.loader.StandardClassLoader{urls=[file:/usr/share/java/tomcat6/,file:/usr/share/java/tomcat6/tomcat-i18n-ja-6.0.24.jar,file:/usr/share/java/tomcat6/catalina-6.0.24.jar,file:/usr/share/java/tomcat6/annotations-api-6.0.24.jar,file:/usr/share/java/tomcat6/tomcat-i18n-es-6.0.24.jar,file:/usr/share/java/tomcat6/catalina-ha-6.0.24.jar,file:/usr/share/java/ecj-3.4.2.jar,file:/usr/share/java/tomcat6/catalina-ant-6.0.24.jar,file:/usr/share/java/tomcat6/jasper-6.0.24.jar,file:/usr/share/tomcat6/bin/tomcat-juli-6.0.24.jar,file:/usr/share/java/tomcat6/jasper-el-6.0.24.jar,file:/usr/share/java/tomcat6-servlet-2.5-api-6.0.24.jar,file:/usr/share/java/tomcat6/tomcat-i18n-fr-6.0.24.jar,file:/usr/share/java/log4j-1.2.14.jar,file:/usr/share/java/jakarta-commons-pool-1.3.jar,file:/usr/share/java/mx4j/mx4j-jmx-3.0.1.jar,file:/usr/share/java/tomcat6-el-2.1-api-6.0.24.jar,file:/usr/share/java/jakarta-commons-collections-3.2.1.jar,file:/usr/share/java/jakarta-commons-dbcp-1.2.1.jar,file:/usr/share/java/tomcat6/tomcat-coyote-6.0.24.jar,file:/usr/share/java/tomcat6-jsp-2.1-api-6.0.24.jar,file:/usr/share/java/tomcat6/catalina-tribes-6.0.24.jar,file:/usr/share/java/mx4j/mx4j-remote-3.0.1.jar], parent=gnu.gcj.runtime.SystemClassLoader{urls=[file:./,file:/usr/share/tomcat6/bin/bootstrap.jar,file:/usr/share/tomcat6/bin/tomcat-juli.jar,file:/usr/share/java/commons-daemon.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}}

Resolution

Installing other Red Hat supplied Java's on Red Hat Satellite is unsupported. Satellite server should use IBM JVM provided by the satellite channel only.

  • Re-configure the system to use the IBM JVM provided by the satellite channel:
# alternatives --config java

There are 3 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
*+ 2           /usr/lib/jvm/jre-1.6.0-ibm.x86_64/bin/java
   3           /usr/lib/jvm/jre-1.5.0-gcj/bin/java

Enter to keep the current selection[+], or type selection number: 2
  • Restart the Satellite services:
# rhn-satellite restart
  • Make sure the JVM used on your satellite is the 1.6.0 IBM one provided by the satellite channel remove the java-1.7.0-openjdk package if found to be installed on the satellite server and restart the Satellite services by running below mentioned command:
# yum remove java-1.7.0-openjdk
# rhn-satellite restart

Root Cause

  • The configuration file is intended to be used with the IBM 1.6.0 jvm, not with openjdk. As such some options may not be compatible with openjdk.

  • This may result from an incorrect installation via the RHEL 6 GUI, in which "Basic Server" is mistaken for "@Base" as specified in the Satellite Installation Documentation. The "Basic Server" selection, will install additional groups include the "@java-platform", which installs an incorrect version of java, then set to the default before the java-ibm package is installed during the Satellite script installation.

To avoid this:
1. Incorrect selection
basic_server_selection.png
2. Note all the additional groups installed, including java-platform
customize_now.png
3. The better way to ensure success, and avoid multiple Java packages installed is to select 'Minimal'
minimal.png
4. Then, choose just the 'Base' group:
select_base_group.png

Diagnostic Steps

  • Check the /var/log/rhn/rhn_taskomatic_daemon.log
STATUS | wrapper  | 2013/07/25 19:00:12 | Launching a JVM...
INFO   | jvm 4    | 2013/07/25 19:00:12 | Unrecognized option: -Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
INFO   | jvm 4    | 2013/07/25 19:00:12 | Error: Could not create the Java Virtual Machine.
INFO   | jvm 4    | 2013/07/25 19:00:12 | Error: A fatal exception has occurred. Program will exit.
ERROR  | wrapper  | 2013/07/25 19:00:12 | JVM exited while loading the application.
  • List the installed java machines :
  # yum list installed java-*
  • Check alternatives :
# alternatives --display java
  • Check rpm -qa *java* output:
java-1.6.0-ibm-devel-1.6.0.10.1-1jpp.5.el6_2.x86_64 Tue 23 Apr 2013 
01:48:51 PM CDT
java-1.6.0-ibm-1.6.0.10.1-1jpp.5.el6_2.x86_64 Tue 23 Apr 2013 01:48:47 PM 
CDT
java-1.6.0-openjdk-1.6.0.0-1.57.1.11.9.el6_4.x86_64 Fri 15 Mar 2013 
05:06:35 PM CDT
  • Check /var/log/tomcat*/catalina.out

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

4 Comments

This does not resolve my issue. I am set to use IBM jvm but still rhn_search and taskomatic fail with same error above after 5.4 to 5.5 upgrade.

Running 5.5 and settings alternatives to 1.6.0-ibm has resolved the errors I was seeing in the logs. However the root cause of being unable to retrieve updates from end clients still exists.

Solution was to make sure ibm java was set. I did a alternatives --config java and my ibm was selected. I changed it and exited then ran again and reset it back to ibm java. After this satellite would start perfectly. I then ran a cobbler-update, satellite-sync, and tested installs and config files and other items in Satellite. Everything was now working

I have done "alternatives --config java", set to ibm, restart taskomatic. It still didn't work. I then ran satellite-sync which fixed the issue. why satellite-sync did fix the issue because generates new repository metadata, and that was the main issue.