How do I generate a Java thread dump on Linux/Unix?
Environment
- Java
- Linux and most Unix-like environments
Issue
- How do I generate a thread dump in JBoss while running on Linux?
- How do I generate a JBoss stack trace on Linux?
- How do I redirect output of kill -3 to a file?
- JBoss has high cpu usage, freezes, hangs, or doesn't release idle threads, how can I get a thread dump to troubleshoot?
- JMS messages are getting lost, how can I generate a thread dump to investigate?
- Unable to take thread dump using Kill -3 in server.log or console.log
Resolution
Disclaimer: Links contained herein to an external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.
Choose one of the following methods to generate a Java thread dump on Unix:
- Option 1: OpenJDK / Sun JDK
- Option 2:
kill -3
Linux script (not-continuous) - Option 3:
kill -3
Linux script (continuous) - Option 4: JBoss EAP Parameter
- Option 5:
jstack
Linux script(continuous) - Option 6: IBM J9
Note: Typically gc logging is provided whenever thread dumps are captured. Excessive gc can cause threading issues, so gc and thread dump analysis go hand in hand.
Option 1: OpenJDK / Sun JDK
Use jps -lv
to find the Java process ID for issuing kill -QUIT
or kill -3
.
Be sure the -Xrs
JVM option is not being used, as it causes SIGQUIT
and SIGWAITING
signals to be ignored. Running kill -3
sends a SIGQUIT
signal to the JVM, so using this option will cause kill -3
to be ignored. See Java Application launcher.
If using OpenJDK or Sun JDK 1.6 or later, using jstack
is an option. This is useful when redirecting standard out to a file is problematic for some reason (e.g. it is not desirable to restart the JVM just to redirect standard out). Execute the following, passing in the Java process ID:
jstack -l JAVA_PID > jstack.out
Note the process ID number of the Java process (e.g. using top
, a grep
on ps -axw
, etc.) and send a QUIT
signal to the process with the kill -QUIT
or kill -3
command 1. For example:
kill -3 JAVA_PID
Option 2: kill -3
Linux script (not-continuous)
Download either threaddump_linux.sh.tar.gz
or threaddump_solaris.sh.tar.gz
, and extract the script. Make the script executable with chmod 755
.
The script will capture a series of 6 thread dumps spaced 20 seconds apart (this can be modified as needed), passing in the Java process ID as an argument. For example:
Linux: sh ./threaddump_linux.sh JAVA_PID
Solaris: bash ./threaddump_solaris.sh JAVA_PID
Be sure to test the script before the issue happens to make sure it runs properly in your environment.
Option 3: kill -3
Linux script (continuous)
Download either threaddump_linux-continuous.sh.tar.gz
or threaddump_solaris-continuous.sh.tar.gz
, and extract the script. Make the script executable with chmod 755
.
It will capture thread dumps spaced 20 seconds apart (modify as needed), passing in the Java process ID as an argument. For example:
Linux: sh ./threaddump_linux-continuous.sh JAVA_PID
Solaris: bash ./threaddump_solaris-continuous.sh JAVA_PID
Be sure to test the script before the issue happens to make sure it runs properly in your environment.
Option 4: JBoss EAP Parameter
Use the below command to start JBoss instance and then use kill -3
to generate the thread dumps.
If the Java application is started with a service script that logs console output, the thread dumps will be in the console log. Otherwise, redirect stdout
to a file on startup.
nohup $JBOSS_HOME/bin/run.sh -c yourinstancename $JBOSS_OPTS >> console-$(date +%Y%m%d).out 2>&1 < /dev/null &
kill -3 JAVA_PID
This will redirect your output/dump to the file console specified in the above command.
Option 5: jstack linux script (continuous)
Download linux_jstack-continuous.tar.gz
, and extract the script. Make the script executeable with chmod 755
It will use jstack to capture a series of 6 thread dumps spaced 20 seconds apart (modify as needed), passing in the Java process ID as an argument. Make sure you set JAVA_HOME
in this script. It generates a file called jstack_threaddump.out
in the directory where this script is executed. For example:
./threaddump_linux_jstack-continuous.sh JAVA_PID
Option 6: IBM J9
IBM J9 has it own monitoring tool, IBM Thread and Monitor Dump Analyzer for Java - TMDA. However, kill -3 PID
can still be used to generate a thread dump, as described in SeeOption 1: OpenJDK / Sun JDK kill -3
section.
More information can be found in OpenJ9 Docs - Java Dumps.
Notes
- Verify that the Java process is still running with the
ps aux
command (R
orS
state inSTAT
column) . For example,jstack -F <pid>
puts the target Java process in a "trace stop" (T) state. Threads in the (T) state will receive the signal for a thread dump; however, output will be delayed until the process continues. - You need to make sure to execute
jstack
command from the same user as the java process. Please see Getting "Unable to open socket file" message when executing jstack / jmap / jcmd and unable to generate a thread dump / heap dump for more details. - There are known bugs related to using other
jstack
options (e.g.-F
,-m
, etc.) and/or thread dump tools are not able to parse the output, so if any other options besides-l
are proposed, be sure to test capturing and parsing the output. - In some circumstances running jstack might cause a performance impact. It is reported that a slowness in response time can be observed when running jstack in some scenarios.
- If you want to take thread dumps to identify which Java threads consume high cpu, please see How do I identify high CPU utilization by Java threads on Linux/Solaris and use the attached sample script in the article.
JBoss EAP 5.x
To redirect stdout
to a file on startup:
sh run.sh > console.log 2>&1
If having trouble redirecting stdout
, modify $JBOSS_HOME/bin/run.sh
and change the line:
org.jboss.Main "$@"
to read:
org.jboss.Main "$@" > console.log
Save and restart JBoss normally. Now kill -3
or kill -QUIT
should create a thread dump in the file console.log
.
JBoss EAP 7.x/6.x
The recommendation is to use jstack
. See Option 5: jstack
Linux script(continuous).
JBoss Fuse 6
If you are capturing thread dumps from a child container, note that the process is different to the root container. Be sure to check the process ID (PID) using 'ps' or similar before selecting the PID to run the script against. eg:
ps -ef | grep child1
testusr 27803 1 0 09:41 pts/1 00:00:12 /usr/java/jdk1.7.0_21/jre/bin/java -server ... -Dkaraf.home=/home/testusr/apps/product-distributions -Dkaraf.base=/home/testusr/apps/product-distributions/instances/child1 org.apache.karaf.main.Main
Related
How do I identify high CPU utilization by Java threads on Linux/Solaris
-
Note:
kill -3
command does not produce output directly to a file from the command line. Instead, it produces output to theconsole output
of the process it was invoked on. For a JBoss EAP domain mode server, its console output should be captured in the host controller log by default. ↩︎
Attachments
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
12 Comments
Hello
Referred Linux script doesn't work:
[dariusz.panasiuk as root@jat421-pop1/home/dariusz.panasiuk]#./threaddump_linux.sh.tar_10 $(pgrep java -u jboss) -bash: ./threaddump_linux.sh.tar_10: cannot execute binary file
[dariusz.panasiuk as root@jat421-pop1/home/dariusz.panasiuk]#
[dariusz.panasiuk as root@jat421-pop1/home/dariusz.panasiuk]#wc -l threaddump_linux.sh.tar_10
24 threaddump_linux.sh.tar_10
cheers
Dariusz
It's tar-ed gzip
tar -xzvf threaddump_linux.sh.tar_10.gz
There should be a full explanation like which pid to take a dump of... for e.g. under jboss eap 6 managed domain. you will see 5 minimum java processes running.... how to find out which process pid to use whilte taking dump.
Thanks
J
Which process to take a thread dump of depends on the problem you are trying to solve. If it is a hang or bottleneck in a JBoss instance, take a dump of that. Some problems around management operations getting stuck may require taking dumps of all of them.
where is the output stored for this kill -3 command?
It is written to stdout. Depending on how your Java application is started and whether it is attaching stdout to its logger, will determine where the thread trace is written.
In most cases, it is written to the terminal from where the process was launched. This is why it is important to redirect the terminal's stdout and stderr file descriptors to a file.
Using jstack takes out the guesswork in where your output will be written and I recommend using it if at all possible.
Hope that helps,
Larry
Thanks Larry, however, Jstack is not available in RHEL 6.6 . I have installed openjdk 1.7 but couln't find jstack.
jstack is included in the openjdk development packages. For example:
java-1.7.0-openjdk-devel
Are you not able to install the devel package which includes the JDK components?
Hi Larry,
Yes, I managed to install. There are certail problems with JDK 1.7.0_79, which prevent me from getting output. Followed this https://access.redhat.com/solutions/9907, but I end up getting
Error communicating with server. The message was:
Error Class Code: 37
However, I am the one who registered rhn_classic on the server. This is very confusing.
You will not need the debuginfo packages. The devel package is in the rhel/update repository. Basically,
java-1.7.0-openjdk
is the JRE andjava-1.7.0-openjdk-devel
is the JDK.jstack
is included in the JDK.As for the Error Class Code: 37, the solution While subscribing channels to systems using "rhn-channel" utility from the client system, it fails with error "Error Class Code: 37"
seems relevant.
Shall not be one of prerequisites of successful usage of
jstack
presence of devel and debuginfo packages? Since e.g. on Satellite6 (wherecandlepin
is deployed as an Apache Tomcat application), I had to follow https://access.redhat.com/solutions/1441633, in particular:You shouldn't need the debug packages unless you are attempting to troubleshoot the native JVM components. This would not typically apply to Java thread dumps. However, I could see the use as it relates to a Java core dump.
You are correct about the openjdk-devel package though. But this solution applies to both Oracle JDK and OpenJDK as well as platforms that may not be RPM based. Basically, you need to make certain you have the jstack binary available which is part of the JDK and not typically included with the JRE.