What information do I supply when opening a Middleware support case?

Updated -

When opening a case with JBoss Middleware Support, it is not known what information the support engineers will need to diagnose the issue. As a result, the first response from JBoss Support is often a request for additional information. This results in an initial delay in resolving the issue.

Assumptions

You are opening a JBoss Middleware support case in the Red Hat Customer Portal.

Solution

To be best prepared for typical data captures required for production outage RCAs, please see Recommended preparation for typical JBoss troubleshooting data points.

Technology-specific information to supply when opening a JBoss support case:

Apache HTTPD Server

Files:

  1. /etc/httpd/conf/*.conf
  2. /etc/httpd/conf.d/*.conf
  3. httpd access and error logging from relevant hosts
  4. /etc/sysconfig/httpd

Information:

  1. The output from running httpd -V on Apache. For example: httpd -V > httpd.out. Note the capital "V". A small "v" will not produce the desired output.

JBoss HTTP Connector (mod_cluster)

  1. Files from Apache HTTPD Server section (Include conf.d/JBoss_HTTP.conf or similar)
  2. Files from JBoss EAP section

Apache Tomcat Connectors (mod_jk)

Files:

  1. boot.log
  2. httpd.conf
  3. workers.properties
  4. mod_jk.conf
  5. mod_jk.log
  6. JBOSS_HOME/server/SERVERCONF/deploy/JBOSSWEB/server.xml
  7. JBOSS_HOME/server/SERVERCONF/deploy/JBOSSWEB/META-INF/jboss-service.xml
  8. /etc/sysconfig/httpd (RHEL)

Information:

  1. The output from running httpd -V on Apache. For example: httpd -V > httpd.out. Note the capital "V". A small "v" will not produce the desired output.

  2. The output from running the ldd command on the mod_jk binary. For example: ldd mod_jk.so.

  3. How many CPUs and RAM does the JBoss box have?

  4. Are the JBoss instances collocated or on dedicated boxes?

  5. Is there a firewall between Apache and JBoss?

  6. Where was the mod_jk binary obtained (e.g. downloaded from apache.org, compiled in house, from OS vendor, etc.)?

  7. What version of mod_jk are you using? If not sure, run the following command: strings mod_jk.so | grep mod_jk/

  8. Exact version of JBoss.

Apache Tomcat

Files:

  1. server.conf
  2. catalina.out
  3. application web.xml

Information:

  1. Exact version of Tomcat.
  2. Exact OS and version.

Java Virtual Machine (JVM)

Files:

  1. boot.log
  2. GC logging
  3. run.conf, run.bat, or wherever JVM options are defined.

Information:

  1. Exact OS and version.
  2. Exact JVM and version.

JBoss Enterprise Application Platform (EAP)

Files:

  1. boot.log (only for EAP 4 and EAP 5)
  2. server.log
  3. console.log, proccess-controller.log and host-controller.log (only for EAP 6+ running in a domain configuration)
  4. GC logging
  5. Configuration files:
    • $JBOSS_HOME/bin/standalone.conf and $JBOSS_HOME/standalone/configuration/standalone*.xml (for EAP 6+ running in a standalone configuration)
    • $JBOSS_HOME/bin/domain.conf, $JBOSS_HOME/standalone/configuration/domain.xml and $JBOSS_HOME/standalone/configuration/host*.xml (for EAP 6+ running in a domain configuration)

Information:

  1. Exact version of JBoss.
  2. Exact OS and version.
  3. Exact JDK and version.
  4. Is it running in standalone or domain configuration?
  5. Is it a single node, or multiple clustered nodes?
  6. List application deployable (jar/war/ear/sar) and where they are deployed.

On JBoss EAP 6.x run the 'JBoss Diagnostics Reporter' JDR

See https://access.redhat.com/solutions/221103

JBoss Fuse 6.*

Files:

  1. $data/log/fuse.log*

Where $data is the default value of the KARAF_DATA environment variable which is your ./data directory in your installation.

Information:

  1. Exact version of JBoss Fuse with patches
  2. Are the containers deployed in a Fabric?

  3. Exact version of Java, as reported by java -version

  4. Application spring configuration, or Blueprint configuration file (located in META-INF/spring/.xmllocated or in OSGI-INF/blueprint/.xml)

Camel

  1. debug logging - enable debug logging by changing the etc/org.ops4j.pax.logging file, log4j.rootLogger property to DEBUG

  2. It can be helpful to enable the tracer on camel routes (trace="true" on the camel context). You may also do this dynamically through the route MBean operation using JConsole or Hawtio.

  3. Full camel route definitions

  4. Any relevant camel processor code

  5. If the project is Maven based, the complete project if it is small or at least pom file.

  6. If the route had been previously working, a thorough description of when the problem arose, system related activities and what changes may have been made around that time.

Bundle related issues
  1. Increase the logging level according, set DEBUG in the Karaf console will be suitable
  2. Identify the bundle ID using the osgi:list command and attach the output.
  3. Execute the command "headers " and capture the output. Note items in red as these are missing dependencies.
Security related issues
  1. Capture the output of jaas:realms
  2. If you have configured an alternative security system as LDAP, please provide details of that system and the corresponding configuration file.
  3. Increase the logging level to DEBUG from karaf console
  4. If LDAP is deployed, use the debug=true in the module definition
Fabric

  1. Provide the output of the fabric:container-list command.

  2. Provide the output of fabric:ensemble-list

  3. Identify which container is problematic and when the issue started.

  4. Maximize the logging to DEBUG via etc/org.ops4j.pax.logging file.

  5. If the issue is specific to one profile, provide the out of the fabric:profile-display command

  6. If a specific application is at hand, please provide the corresponding application.

  7. Please provide the contents of the etc/org.ops4j.pax.url.mvn.cfg if a maven related issue occurs.

  8. Provide the output of config:list from the affected container.

  9. $data/log/karaf.log* from each container in the fabric.

Where $data is the default value of the KARAF_DATA environment variable which is your ./data directory in your installation.

JBoss Fuse 7.*

Fuse 7 is able to run on Apache Karaf, Red Hat JBoss EAP, or Spring Boot. The information collected will differ depending upon which platform is used.

General Information:
  1. Exact version of Red Hat Fuse with patches
  2. Exact version of Java, as reported by java -version
Running On Karaf:
  1. $data/log/fuse.log*, Where $data is the default value of the KARAF_DATA environment variable (in a default installation, this is the ./data directory)
Bundle related issues
  1. Increase the logging level according, setting DEBUG in the Karaf console will be suitable
  2. Identify the bundle ID using the osgi:list command and attach the output
  3. Execute the command headers <bundle id> and capture the output. Note items in red as these are missing dependencies
Security related issues
  1. Capture the output of jaas:realms
  2. If you have configured an alternative security system as LDAP, please provide details of that system and the corresponding configuration file
  3. Increase the logging level to DEBUG from the Karaf console
  4. If LDAP is deployed, use debug=true in the module definition
Running On EAP:
  1. Exact version of EAP with patches
  2. Is EAP running as a single node or multiple clustered nodes?
  3. server.log file
Running On Spring Boot:
  1. Spring Boot doesn't enforce any specific logging. If you have included a way to generate application logs in your product, please include the generated logs
  2. Logging levels in Spring Boot can be changed in the application.properties file. Changing logging.level.root will affect all applications, while changing logging.level.<package.name> will affect the specific application(s) using that package.name
Camel:
  1. Debug logging - enable debug logging by changing log4j.rootLogger property in the etc/org.ops4j.pax.logging file to DEBUG
  2. It can be helpful to enable the tracer on Camel routes (trace="true" on the camel context). You may also do this dynamically through the route MBean operation using JConsole or Hawtio
  3. Full Camel route definitions
  4. Any relevant Camel processor code
  5. If the project is Maven based, the complete project if it is small or at least its POM file
  6. If the route had been previously working, a thorough description of when the problem arose, system related activities, and what changes may have been made around that time

JBoss A-MQ 6.*

Files:

  1. data/log/amq.log*
  2. etc/activemq.xml

Where data is the default value of the KARAF_DATA environment variable. By default this is the /data directory.

Information:

  1. Exact version of JBoss A-MQ

  2. Exact version of Java, as reported by java -version

  3. For stuck or missing messages attach the output of the activemq:bstat command

  4. For message store issues, a backup of the message store (e.g. kahadb)

  5. For out of memory issues - a heap dump

  6. For performance issues - capture multiple (e.g. 3) thread dumps spaced every 20 seconds in combination with GC logging.

  7. Screenshots of broker stats

  8. Screenshot of the JMX stats for the destination in question

  9. For failover issues turn on org.apache.activemq.transport logging

JBoss AMQ Broker 7.*

Files:
1. configuration file: artemis.xml
2. logs: broker.xml

Information:
1. Exact version of AMQ Broker
2. Exact version of Java, as reported by java -version
3. Architecture information

Depending on scenario:
3a. For slowness or stuck messages,
- heap dump
- thread dumps
- gc logs
- output of queue stat

3b. Unable to consume messages,
- screen shot of queue stat

3c. For stuck or missing messages attach the output of the activemq:bstat command

Red Hat MRG Messaging (MRG-M)

Follow this solution (basically, provide sosreport and qpidd broker logs).

To run sosreport the sos package must be installed. If for any reason the package is not present, please follow the steps in this article to install it: What is a sosreport and how do I create one

BRMS 6.0.*

Information:

Issues about runtime rules not firing or wrong order:

  1. Exact version of BRMS

  2. Is the latest rollup patch applied?

  3. Rule files (DRL, Guided Decision Tables, JCR export (version 5 only), git repository (version 6 only))

  4. Knowledge Runtime logger: How to enable runtime logging in BRMS and BPM Suite?

  5. Fact model used (model jars)

  6. kmodule.xml

  7. Reproducer (when possible)

Issues about Business Central editors:

  1. Application Server log (e.g. server.log for EAP)

  2. Screenshots

  3. Browser used

  4. Application server where it is deployed

  5. Rule files (DRL, Guided Decision Tables, JCR export (version 5 only), git repository (version 6 only))

RDHM 7.*

Information:

Issues about runtime rules not firing or wrong order:

  1. Exact version of RHDM

  2. Is the latest rollup patch applied?

  3. Rule files (DRL, Guided Decision Tables, git repository)

  4. Knowledge Runtime logger: How to enable runtime logging in BRMS and BPM Suite?

  5. Fact model used (model jars)

  6. kmodule.xml

  7. Reproducer (when possible)

Issues about Business Central editors:

  1. Application Server log (e.g. server.log for Red Hat EAP)

  2. Application Server configuration files (e.g. standalone-full.xml or domain.xml for EAP)

  3. Screenshots

  4. Browser used

  5. Application server where it is deployed

  6. Rule files (DRL, Guided Decision Tables, git repository)

BPM Suite 6.0.*

Information:

Issues about runtime process execution:

  1. Exact version of BPM Suite

  2. Is the latest rollup patch applied?

  3. Database used

  4. Application Server log (e.g. server.log for EAP)

  5. Is the process executed by Kie Remote APIs (Rest, JMS) or wrapping the jBPM API directly?

  6. Is Spring being used? If yes, provide Spring beans file

  7. Process Files (.bpmn2, custom domain workitems if used, JCR export (version 5 only), git repository (version 6 only))

  8. Fact model used (model jars)

  9. kmodule.xml

  10. Reproducer (when possible)

Issues about Business Central editors:

  1. Application Server log (e.g. server.log for EAP)
  2. Screenshots
  3. Browser used
  4. Application server where it is deployed
  5. Rule and Process files (.bpmn2, DRLs, repository exports, etc)

RHPAM 7.*

Information:

Issues about runtime process execution:

  1. Exact version of RHPAM

  2. Is the latest rollup patch applied?

  3. Database used

  4. Application Server log (e.g. server.log for EAP)

  5. Is the process executed by Kie Remote APIs (Rest, JMS) or wrapping the Kie API directly?

  6. Process or Case Management files (.bpmn2, custom domain workitems if used, git repository)

  7. Fact models used (model jars)

  8. kmodule.xml and kie-deployment-descriptor.xml

  9. Reproducer (when possible)

Issues about Business Central editors:

  1. Application Server log (e.g. server.log for EAP)
  2. Application Server configuration files (e.g. standalone-full.xml or domain.xml for EAP)
  3. Screenshots
  4. Browser used
  5. Application server where it is deployed
  6. Rule and Process files (.bpmn2, DRLs, repository exports, etc)

Issues about database deadlocks:

  1. Application Server log (e.g. server.log for EAP)

  2. Application Server configuration files (e.g. standalone-full.xml or domain.xml for EAP)

  3. Database being used

  4. Database SQL log

JBoss Operations Network (JBoss ON)

Files:

  1. server.log
  2. agent.log
  3. rhq-server.properties

Information:

  1. Exact version of JBoss ON.
  2. Exact OS and version.
  3. Screenshot of the issue, if relevant.

Red Hat Developer Studio 12.* (RHDS)

Files:

  1. $RHDS_HOME/studio/devstudio.ini file
  2. From inside RHDS go to the Window > Show View > Other > Error Log and attach the Export Log resulting file to the case
  3. Possible either screencast or a screenshot showing the problem if it's UI based issue
  4. $WORKSPACE/.metadata/ directory

Information:
1. Exact version of Red Hat Developer Studio
2. What tooling was the issue seen on?
3. What sort of project was being worked on?

Red Hat Data Grid (RHDG)

Files:

  1. Configuration files:
    • RHDG server until version 7
      • $RHDG_HOME/standalone/configuration/standalone.xml and $JDG_HOME/standalone/configuration/clustered.xml (for JDG 7+ running in a standalone configuration)
      • RHDG_HOME/domain/configuration/domain.xml and JDG_HOME/domain/configuration/host*.xml (for JDG 7+ running in a domain configuration)
    • RHDG server 8.x onwards
      • $RHDG_HOME/server/conf/infinispan.xml or other configuration directory or file <infinispan.xml> used
      • $RHDG_HOME/server/data/caches.xml cache configurations added by administration interfaces at runtime
    • RHDG in embedded mode
      • custom configuration and/or initialization code
  2. log files
    Necessary from all nodes of the cluster, if it is a cluster and the issue is related to it. If it is related to Cross Site Replication the log files of both sides are necessary.
    For functional issues it might be enough to have the log from one node.
    • RHDG server until version 7
      • $RHDG_HOME/standalone/log/server.log
    • RHDG server 8.x onwards
      • $RHDG_HOME/server/log/server.log
  3. GC logging could be helpful for cluster partitioning and timeouts if the cause are memory issues
  4. Thread dumps or a series of it, could be helpful for any locking or timeout
  5. Heap dumps in case of memory issues and leaking

Information:

  1. Exact version of RHDG.
  2. DG mode embedded or client server, with client server the type of client in use
  3. Exact OS and version.
  4. Exact JDK and version.
  5. Is it a single node, or multiple clustered nodes?
  6. If running client/server and version is not 8 or better, is it running in standalone or domain configuration?
  7. List application deployable (jar/war/ear/sar) and where they are deployed.
    • RHDG server until version 7
      • `$RHDG_HOME/standalone/deployments (for JDG 7+ running in a standalone configuration)
      • If running in domain mode share the RHDG_HOME/domain/configuration/domain.xml and the deployed artifacts
    • RHDG server 8.x onwards
      Show the $RHDG_HOME/lib content, which should only contain the files distributed by Red Hat
      and `$RHDG_HOME//lib for the custom jar files
    • RHDG in embedded mode, deployed as EAP application
      • deployments which include additional libraries

On RHDG 7.x run the 'JBoss Diagnostics Reporter' JDR see https://access.redhat.com/solutions/221103

** On RHDG 8.x on OCP 4, https://access.redhat.com/solutions/6452611

JBoss Red Hat Single Sign-On

  • In addition to files for JBoss-EAP, please export the realm and share with us to move the investigation
    • You can do it from the admin console from Manage->Export.

Comments