What information do I supply when opening a Middleware support case?
When opening a case with JBoss Middleware Support, it is not known what information the support engineers will need to diagnose the issue. As a result, the first response from JBoss Support is often a request for additional information. This results in an initial delay in resolving the issue.
Assumptions
You are opening a JBoss Middleware support case in the Red Hat Customer Portal.
Solution
To be best prepared for typical data captures required for production outage RCAs, please see Recommended preparation for typical JBoss troubleshooting data points.
Technology-specific information to supply when opening a JBoss support case:
- Apache HTTPD Server
- JBoss HTTP Connector
- Apache Tomcat Connectors
- Apache Tomcat
- Java Virtual Machine (JVM)
- JBoss Enterprise Application Platform (EAP)
- JBoss Fuse 6.*
- Red Hat Fuse 7.*
- JBoss A-MQ 6.*
- JBoss AMQ Broker 7.*
- Red Hat MRG Messaging (MRG-M)
- Red Hat JBoss BRMS 6.0.*
- Red Hat JBoss BPM Suite 6.0.*
- Red Hat Process Automation Manager 7.*
- Red Hat Decision Manager 7.*
- JBoss Operations Network (JON)
- Red Hat Developer Studio 12.*
- Red Hat Data Grid (RHDG)
- Red Hat Single Sign-On (SSO)
Apache HTTPD Server
Files:
/etc/httpd/conf/*.conf
/etc/httpd/conf.d/*.conf
- httpd access and error logging from relevant hosts
/etc/sysconfig/httpd
Information:
- The output from running
httpd -V
on Apache. For example:httpd -V > httpd.out
. Note the capital "V". A small "v" will not produce the desired output.
JBoss HTTP Connector (mod_cluster)
- Files from Apache HTTPD Server section (Include conf.d/JBoss_HTTP.conf or similar)
- Files from JBoss EAP section
Apache Tomcat Connectors (mod_jk)
Files:
boot.log
httpd.conf
workers.properties
mod_jk.conf
mod_jk.log
JBOSS_HOME/server/SERVERCONF/deploy/JBOSSWEB/server.xml
JBOSS_HOME/server/SERVERCONF/deploy/JBOSSWEB/META-INF/jboss-service.xml
/etc/sysconfig/httpd
(RHEL)
Information:
-
The output from running
httpd -V
on Apache. For example:httpd -V > httpd.out
. Note the capital "V". A small "v" will not produce the desired output. -
The output from running the ldd command on the mod_jk binary. For example:
ldd mod_jk.so
. -
How many CPUs and RAM does the JBoss box have?
-
Are the JBoss instances collocated or on dedicated boxes?
-
Is there a firewall between Apache and JBoss?
-
Where was the mod_jk binary obtained (e.g. downloaded from apache.org, compiled in house, from OS vendor, etc.)?
-
What version of mod_jk are you using? If not sure, run the following command:
strings mod_jk.so | grep mod_jk/
-
Exact version of JBoss.
Apache Tomcat
Files:
server.conf
catalina.out
- application
web.xml
Information:
- Exact version of Tomcat.
- Exact OS and version.
Java Virtual Machine (JVM)
Files:
boot.log
- GC logging
run.conf
,run.bat
, or wherever JVM options are defined.
Information:
- Exact OS and version.
- Exact JVM and version.
JBoss Enterprise Application Platform (EAP)
Files:
boot.log
(only for EAP 4 and EAP 5)server.log
console.log
,proccess-controller.log
andhost-controller.log
(only for EAP 6+ running in a domain configuration)- GC logging
- Configuration files:
$JBOSS_HOME/bin/standalone.conf
and$JBOSS_HOME/standalone/configuration/standalone*.xml
(for EAP 6+ running in a standalone configuration)$JBOSS_HOME/bin/domain.conf
,$JBOSS_HOME/standalone/configuration/domain.xml
and$JBOSS_HOME/standalone/configuration/host*.xml
(for EAP 6+ running in a domain configuration)
Information:
- Exact version of JBoss.
- Exact OS and version.
- Exact JDK and version.
- Is it running in standalone or domain configuration?
- Is it a single node, or multiple clustered nodes?
- List application deployable (jar/war/ear/sar) and where they are deployed.
On JBoss EAP 6.x run the 'JBoss Diagnostics Reporter' JDR
See https://access.redhat.com/solutions/221103
JBoss Fuse 6.*
Files:
$data/log/fuse.log*
Where $data
is the default value of the KARAF_DATA
environment variable which is your ./data directory in your installation.
Information:
- Exact version of JBoss Fuse with patches
- Are the containers deployed in a Fabric?
-
Exact version of Java, as reported by
java -version
-
Application spring configuration, or Blueprint configuration file (located in META-INF/spring/.xmllocated or in OSGI-INF/blueprint/.xml)
Camel
-
debug logging - enable debug logging by changing the
etc/org.ops4j.pax.logging
file,log4j.rootLogger
property to DEBUG -
It can be helpful to enable the tracer on camel routes (trace="true" on the camel context). You may also do this dynamically through the route MBean operation using JConsole or Hawtio.
-
Full camel route definitions
-
Any relevant camel processor code
-
If the project is Maven based, the complete project if it is small or at least pom file.
-
If the route had been previously working, a thorough description of when the problem arose, system related activities and what changes may have been made around that time.
Bundle related issues
- Increase the logging level according, set DEBUG in the Karaf console will be suitable
- Identify the bundle ID using the osgi:list command and attach the output.
- Execute the command "headers
" and capture the output. Note items in red as these are missing dependencies.
Security related issues
- Capture the output of jaas:realms
- If you have configured an alternative security system as LDAP, please provide details of that system and the corresponding configuration file.
- Increase the logging level to DEBUG from karaf console
- If LDAP is deployed, use the debug=true in the module definition
Fabric
-
Provide the output of the fabric:container-list command.
-
Provide the output of fabric:ensemble-list
-
Identify which container is problematic and when the issue started.
-
Maximize the logging to DEBUG via etc/org.ops4j.pax.logging file.
-
If the issue is specific to one profile, provide the out of the fabric:profile-display command
-
If a specific application is at hand, please provide the corresponding application.
-
Please provide the contents of the etc/org.ops4j.pax.url.mvn.cfg if a maven related issue occurs.
-
Provide the output of config:list from the affected container.
-
$data/log/karaf.log*
from each container in the fabric.
Where $data
is the default value of the KARAF_DATA
environment variable which is your ./data directory in your installation.
JBoss Fuse 7.*
Fuse 7 is able to run on Apache Karaf, Red Hat JBoss EAP, or Spring Boot. The information collected will differ depending upon which platform is used.
General Information:
- Exact version of Red Hat Fuse with patches
- Exact version of Java, as reported by
java -version
Running On Karaf:
$data/log/fuse.log*
, Where$data
is the default value of theKARAF_DATA
environment variable (in a default installation, this is the./data
directory)
Bundle related issues
- Increase the logging level according, setting
DEBUG
in the Karaf console will be suitable - Identify the bundle ID using the osgi:list command and attach the output
- Execute the command
headers <bundle id>
and capture the output. Note items in red as these are missing dependencies
Security related issues
- Capture the output of
jaas:realms
- If you have configured an alternative security system as LDAP, please provide details of that system and the corresponding configuration file
- Increase the logging level to
DEBUG
from the Karaf console - If LDAP is deployed, use
debug=true
in the module definition
Running On EAP:
- Exact version of EAP with patches
- Is EAP running as a single node or multiple clustered nodes?
server.log
file
Running On Spring Boot:
- Spring Boot doesn't enforce any specific logging. If you have included a way to generate application logs in your product, please include the generated logs
- Logging levels in Spring Boot can be changed in the
application.properties
file. Changinglogging.level.root
will affect all applications, while changinglogging.level.<package.name>
will affect the specific application(s) using thatpackage.name
Camel:
- Debug logging - enable debug logging by changing
log4j.rootLogger
property in theetc/org.ops4j.pax.logging
file toDEBUG
- It can be helpful to enable the tracer on Camel routes (
trace="true"
on the camel context). You may also do this dynamically through the route MBean operation using JConsole or Hawtio - Full Camel route definitions
- Any relevant Camel processor code
- If the project is Maven based, the complete project if it is small or at least its POM file
- If the route had been previously working, a thorough description of when the problem arose, system related activities, and what changes may have been made around that time
JBoss A-MQ 6.*
Files:
data/log/amq.log*
etc/activemq.xml
Where data
is the default value of the KARAF_DATA
environment variable. By default this is the
Information:
-
Exact version of JBoss A-MQ
-
Exact version of Java, as reported by
java -version
-
For stuck or missing messages attach the output of the
activemq:bstat
command -
For message store issues, a backup of the message store (e.g. kahadb)
-
For out of memory issues - a heap dump
-
For performance issues - capture multiple (e.g. 3) thread dumps spaced every 20 seconds in combination with GC logging.
-
Screenshots of broker stats
-
Screenshot of the JMX stats for the destination in question
-
For failover issues turn on
org.apache.activemq.transport
logging
JBoss AMQ Broker 7.*
Files:
1. configuration file: artemis.xml
2. logs: broker.xml
Information:
1. Exact version of AMQ Broker
2. Exact version of Java, as reported by java -version
3. Architecture information
Depending on scenario:
3a. For slowness or stuck messages,
- heap dump
- thread dumps
- gc logs
- output of queue stat
3b. Unable to consume messages,
- screen shot of queue stat
3c. For stuck or missing messages attach the output of the activemq:bstat
command
Red Hat MRG Messaging (MRG-M)
Follow this solution (basically, provide sosreport and qpidd
broker logs).
To run sosreport
the sos
package must be installed. If for any reason the package is not present, please follow the steps in this article to install it: What is a sosreport and how do I create one
BRMS 6.0.*
Information:
Issues about runtime rules not firing or wrong order:
-
Exact version of BRMS
-
Is the latest rollup patch applied?
-
Rule files (DRL, Guided Decision Tables, JCR export (version 5 only), git repository (version 6 only))
-
Knowledge Runtime logger: How to enable runtime logging in BRMS and BPM Suite?
-
Fact model used (model jars)
-
kmodule.xml
-
Reproducer (when possible)
Issues about Business Central editors:
-
Application Server log (e.g. server.log for EAP)
-
Screenshots
-
Browser used
-
Application server where it is deployed
-
Rule files (DRL, Guided Decision Tables, JCR export (version 5 only), git repository (version 6 only))
RDHM 7.*
Information:
Issues about runtime rules not firing or wrong order:
-
Exact version of RHDM
-
Is the latest rollup patch applied?
-
Rule files (DRL, Guided Decision Tables, git repository)
-
Knowledge Runtime logger: How to enable runtime logging in BRMS and BPM Suite?
-
Fact model used (model jars)
-
kmodule.xml
-
Reproducer (when possible)
Issues about Business Central editors:
-
Application Server log (e.g. server.log for Red Hat EAP)
-
Application Server configuration files (e.g. standalone-full.xml or domain.xml for EAP)
-
Screenshots
-
Browser used
-
Application server where it is deployed
-
Rule files (DRL, Guided Decision Tables, git repository)
BPM Suite 6.0.*
Information:
Issues about runtime process execution:
-
Exact version of BPM Suite
-
Is the latest rollup patch applied?
-
Database used
-
Application Server log (e.g. server.log for EAP)
-
Is the process executed by Kie Remote APIs (Rest, JMS) or wrapping the jBPM API directly?
-
Is Spring being used? If yes, provide Spring beans file
-
Process Files (.bpmn2, custom domain workitems if used, JCR export (version 5 only), git repository (version 6 only))
-
Fact model used (model jars)
-
kmodule.xml
-
Reproducer (when possible)
Issues about Business Central editors:
- Application Server log (e.g. server.log for EAP)
- Screenshots
- Browser used
- Application server where it is deployed
- Rule and Process files (.bpmn2, DRLs, repository exports, etc)
RHPAM 7.*
Information:
Issues about runtime process execution:
-
Exact version of RHPAM
-
Is the latest rollup patch applied?
-
Database used
-
Application Server log (e.g. server.log for EAP)
-
Is the process executed by Kie Remote APIs (Rest, JMS) or wrapping the Kie API directly?
-
Process or Case Management files (.bpmn2, custom domain workitems if used, git repository)
-
Fact models used (model jars)
-
kmodule.xml and kie-deployment-descriptor.xml
-
Reproducer (when possible)
Issues about Business Central editors:
- Application Server log (e.g. server.log for EAP)
- Application Server configuration files (e.g. standalone-full.xml or domain.xml for EAP)
- Screenshots
- Browser used
- Application server where it is deployed
- Rule and Process files (.bpmn2, DRLs, repository exports, etc)
Issues about database deadlocks:
-
Application Server log (e.g. server.log for EAP)
-
Application Server configuration files (e.g. standalone-full.xml or domain.xml for EAP)
-
Database being used
-
Database SQL log
JBoss Operations Network (JBoss ON)
Files:
server.log
agent.log
rhq-server.properties
Information:
- Exact version of JBoss ON.
- Exact OS and version.
- Screenshot of the issue, if relevant.
Red Hat Developer Studio 12.* (RHDS)
Files:
$RHDS_HOME/studio/devstudio.ini
file- From inside RHDS go to the
Window
>Show View
>Other
>Error Log
and attach theExport Log
resulting file to the case - Possible either screencast or a screenshot showing the problem if it's UI based issue
$WORKSPACE/.metadata/
directory
Information:
1. Exact version of Red Hat Developer Studio
2. What tooling was the issue seen on?
3. What sort of project was being worked on?
Red Hat Data Grid (RHDG)
Files:
- Configuration files:
- RHDG server until version 7
$RHDG_HOME/standalone/configuration/standalone.xml
and$JDG_HOME/standalone/configuration/clustered.xml
(for JDG 7+ running in a standalone configuration)RHDG_HOME/domain/configuration/domain.xml
andJDG_HOME/domain/configuration/host*.xml
(for JDG 7+ running in a domain configuration)
- RHDG server 8.x onwards
$RHDG_HOME/server/conf/infinispan.xml
or other configuration directoryor file <infinispan.xml> used $RHDG_HOME/server/data/caches.xml
cache configurations added by administration interfaces at runtime
- RHDG in embedded mode
- custom configuration and/or initialization code
- RHDG server until version 7
- log files
Necessary from all nodes of the cluster, if it is a cluster and the issue is related to it. If it is related to Cross Site Replication the log files of both sides are necessary.
For functional issues it might be enough to have the log from one node.- RHDG server until version 7
$RHDG_HOME/standalone/log/server.log
- RHDG server 8.x onwards
$RHDG_HOME/server/log/server.log
- RHDG server until version 7
- GC logging could be helpful for cluster partitioning and timeouts if the cause are memory issues
- Thread dumps or a series of it, could be helpful for any locking or timeout
- Heap dumps in case of memory issues and leaking
Information:
- Exact version of RHDG.
- DG mode embedded or client server, with client server the type of client in use
- Exact OS and version.
- Exact JDK and version.
- Is it a single node, or multiple clustered nodes?
- If running client/server and version is not 8 or better, is it running in standalone or domain configuration?
- List application deployable (jar/war/ear/sar) and where they are deployed.
- RHDG server until version 7
- `$RHDG_HOME/standalone/deployments (for JDG 7+ running in a standalone configuration)
- If running in domain mode share the
RHDG_HOME/domain/configuration/domain.xml
and the deployed artifacts
- RHDG server 8.x onwards
Show the$RHDG_HOME/lib
content, which should only contain the files distributed by Red Hat
and `$RHDG_HOME//lib for the custom jar files - RHDG in embedded mode, deployed as EAP application
- deployments which include additional libraries
- RHDG server until version 7
On RHDG 7.x run the 'JBoss Diagnostics Reporter' JDR see https://access.redhat.com/solutions/221103
** On RHDG 8.x on OCP 4, https://access.redhat.com/solutions/6452611
JBoss Red Hat Single Sign-On
- In addition to files for JBoss-EAP, please export the realm and share with us to move the investigation
- You can do it from the admin console from Manage->Export.
Comments