Resolution

Choose one of the following methods to generate a Java thread dump on Unix:

• Option 1: OpenJDK / Sun JDK
• Option 2: kill -3 Linux script (not-continuous)
• Option 3: kill -3 Linux script (continuous)
• Option 4: JBoss EAP Parameter
• Option 5: jstack Linux script(continuous)

Option 1: OpenJDK / Sun JDK

Use jps -lv to find the Java process ID for issuing kill -QUIT or kill -3.
Be sure the -Xrs JVM option is not being used, as it causes SIGQUIT and SIGWAITING signals to be ignored. Running kill -3 sends a SIGQUIT signal to the JVM, so using this option will cause kill -3 to be ignored. See Java Application launcher.

If using OpenJDK or Sun JDK 1.6 or later, using jstack is an option. This is useful when redirecting standard out to a file is problematic for some reason (e.g. it is not desirable to restart the JVM just to redirect standard out). Execute the following, passing in the Java process ID:

jstack -l JAVA_PID > jstack.out

Note the process ID number of the Java process (e.g. using top, a grep on ps -axw, etc.) and send a QUIT signal to the process with the kill -QUIT or kill -3 command. For example:

kill -3 JAVA_PID

Option 2: kill -3 Linux script (not-continuous)

Download either threaddump_linux.sh.tar.gz or threaddump_solaris.sh.tar.gz, and extract the script. Make the script executable with chmod 755.

It will capture a series of 6 thread dumps spaced 20 seconds apart (modify as needed), passing in the Java process ID as an argument. For example:

Linux: sh ./threaddump_linux.sh JAVA_PID
Solaris: bash ./threaddump_solaris.sh JAVA_PID

Be sure to test the script before the issue happens to make sure it runs properly in your environment.

Option 3: kill -3 Linux script (continuous)

Download either threaddump_linux-continuous.sh.tar.gz or threaddump_solaris-continuous.sh.tar.gz, and extract the script. Make the script executable with chmod 755.

It will capture thread dumps spaced 20 seconds apart (modify as needed), passing in the Java process ID as an argument. For example:

Linux: sh ./threaddump_linux-continuous.sh JAVA_PID
Solaris: bash ./threaddump_solaris-continuous.sh JAVA_PID

Be sure to test the script before the issue happens to make sure it runs properly in your environment.

Option 4: JBoss EAP Parameter

Use the below command to start JBoss instance and then use kill -3 to generate the thread dumps.

If the Java application is started with a service script that logs console output, the thread dumps will be in the console log. Otherwise, redirect stdout to a file on startup.

nohup $JBOSS_HOME/bin/run.sh -c  yourinstancename $JBOSS_OPTS >> console-$(date +%Y%m%d).out  2>&1 < /dev/null &
kill -3 JAVA_PID

This will redirect your output/dump to the file console specified in the above command.

Option 5: jstack linux script (continuous)

Download linux_jstack-continuous.tar.gz, and extract the script. Make the script executeable with chmod 755
It will use jstack to capture a series of 6 thread dumps spaced 20 seconds apart (modify as needed), passing in the Java process ID as an argument. Make sure you set JAVA_HOME in this script. It generates a file called jstack_threaddump.out in the directory where this script is executed. For example:

./threaddump_linux_jstack-continuous.sh JAVA_PID

Notes

• Verify that the Java process is still running with the ps aux command (R or S state in STAT column) . For example, jstack -F <pid> puts the target Java process in a "trace stop" (T) state. Threads in the (T) state will receive the signal for a thread dump; however, output will be delayed until the process continues.
• You need to make sure to execute jstack command from the same user as the java process. Please see Getting "Unable to open socket file" message when executing jstack / jmap / jcmd and unable to generate a thread dump / heap dump for more details.
• There are known bugs related to using other jstack options (e.g. -F, -m, etc.) and/or thread dump tools are not able to parse the output, so if any other options besides -l are proposed, be sure to test capturing and parsing the output.
• In some circumstances running jstack might cause a performance impact. It is reported that a slowness in response time can be observed when running jstack in some scenarios.
• If you want to take thread dumps to identify which Java threads consume high cpu, please see How do I identify high CPU utilization by Java threads on Linux/Solaris and use the attached sample script in the article.

JBoss EAP 5.x

To redirect stdout to a file on startup:

sh run.sh > console.log 2>&1

If having trouble redirecting stdout, modify $JBOSS_HOME/bin/run.sh and change the line:

org.jboss.Main "$@"

to read:

org.jboss.Main "$@" > console.log

Save and restart JBoss normally. Now kill -3 or kill -QUIT should create a thread dump in the file console.log.

JBoss EAP 6.x

The recommendation is to use jstack.

JBoss Fuse 6

If you are capturing thread dumps from a child container, note that the process is different to the root container. Be sure to check the process ID (PID) using 'ps' or similar before selecting the PID to run the script against. eg:

ps -ef | grep child1

testusr 27803     1  0 09:41 pts/1    00:00:12 /usr/java/jdk1.7.0_21/jre/bin/java -server ... -Dkaraf.home=/home/testusr/apps/product-distributions -Dkaraf.base=/home/testusr/apps/product-distributions/instances/child1  org.apache.karaf.main.Main

Related

How do I identify high CPU utilization by Java threads on Linux/Solaris