Chapter 3. Debug symbols for OpenJDK 8

Debug symbols help in investigating a crash in OpenJDK applications.

3.1. Installing the debug symbols

This procedure describes how to install the debug symbols for OpenJDK.

Prerequisites

  • Installed the gdb package on your local sytem.

    • You can issue the sudo yum install gdb command on your CLI to install this package on your local system.

Procedure

  1. To install the debug symbols, enter the following command: 

    $ sudo yum debuginfo-install java-1.8.0-openjdk

$ sudo yum debuginfo-install java-1.8.0-openjdk-headless

    These commands install java-1.8.0-openjdk-debuginfo, java-1.8.0-openjdk-headless-debuginfo, and additional packages that provide debug symbols for OpenJDK 8 binaries. These packages are not self-sufficient and do not contain executable binaries.

    Note

    The debuginfo-install is provided by the yum-utils package.

  2. To verify that the debug symbols are installed, enter the following command: 

    $ gdb which java

Reading symbols from /usr/bin/java...Reading symbols from /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-2.el8_5/bin/java-1.8.0.322.b06-2.el8_5.x86_64.debug...done.
(gdb)

3.2. Checking the installation location of debug symbols

This procedure explains how to find the location of debug symbols.

Note

If the debuginfo package is installed, but you cannot get the installation location of the package, then check if the correct package and java versions are installed. After confirming the versions, check the location of debug symbols again.

Prerequisites

  • Installed the gdb package on your local sytem.

    • You can issue the sudo yum install gdb command on your CLI to install this package on your local system.
    • Installed the debug symbols package. See Installing the debug symbols.

Procedure

  1. To find the location of debug symbols, use gdb with which java commands: 

    $ gdb which java

Reading symbols from /usr/bin/java...Reading symbols from /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-2.el8_5/bin/java-1.8.0.322.b06-2.el8_5.x86_64.debug...done.
done.
(gdb)

  2. Use the following commands to explore the *-debug directory to see all the debug versions of the libraries, which include java, javac, and javah: 

    $ cd /usr/lib/debug/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-2.el8_5
    $ tree

OJDK 8 version:
└── java-1.8.0-openjdk-1.8.0.322.b06-2.el8_5
├── bin
│   ...
│ │──  java-java-1.8.0.322.b06-2.el8_5.x86_64.debug
│   ├── javac-java-1.8.0.322.b06-2.el8_5.x86_64.debug
│   ├── javadoc-java-1.8.0.322.b06-2.el8_5.x86_64.debug
│   ...
└── lib
├── jexec-java-1.8.0.322.b06-2.el8_5.x86_64.debug
├── jli
│   └── libjli.so-java-1.8.0.322.b06-2.el8_5.x86_64.debug
├── jspawnhelper-java-1.8.0.322.b06-2.el8_5.x86_64.debug
│   ...
Note

The javac and javah tools are provided by the java-1.8.0-openjdk-devel package. You can install the package using the command: $ sudo debuginfo-install java-1.8.0-openjdk-devel.

3.3. Checking the configuration of debug symbols

You can check and set configurations for debug symbols.

  • To get the list of the installed packages, enter the following command: 

    $ sudo yum list installed | grep 'java-1.8.0-openjdk-debuginfo'

  • If some debug information packages have not been installed, enter the following command to install the missing packages: 

    $ sudo yum debuginfo-install glibc-2.28-151.el8.x86_64 libgcc-8.4.1-1.el8.x86_64 libstdc++-8.4.1-1.el8.x86_64 sssd-client-2.4.0-9.el8.x86_64 zlib-1.2.11-17.el8.x86_64

  • Run the following command if you want to hit a specific breakpoint: 

    $ gdb -ex 'handle SIGSEGV noprint nostop pass' -ex 'set breakpoint pending on' -ex 'break JavaCalls::call' -ex 'run' --args java ./HelloWorld

    The above command completes the following tasks:

    • Handles the SIGSEGV error as the JVM uses SEGV for stack overflow check.
    • Sets pending breakpoints to yes.
    • Calls the break statement in JavaCalls::call function. The function to starts the application in HotSpot (libjvm.so).

3.4. Configuring the debug symbols in a fatal error log file

When a Java application is down due to a JVM crash, a fatal error log file is generated, for example: hs_error, java_error. These error log files are generated in current working directory of the application. The crash file contains information from the stack.

Procedure

  1. You can remove all the debug symbols by using the strip -g command.

    The following code shows an example of non-stripped hs_error file: 

    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xb83d2a]  Unsafe_SetLong+0xda
j  sun.misc.Unsafe.putLong(Ljava/lang/Object;JJ)V+0
j  Crash.main([Ljava/lang/String;)V+8
v  ~StubRoutines::call_stub
V  [libjvm.so+0x6c0e65]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xc85
V  [libjvm.so+0x73cc0d]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .constprop.1]+0x31d
V  [libjvm.so+0x73fd16]  jni_CallStaticVoidMethod+0x186
C  [libjli.so+0x48a2]  JavaMain+0x472
C  [libpthread.so.0+0x9432]  start_thread+0xe2

    The following code shows an example of stripped hs_error file: 

    Stack: [0x00007ff7e1a44000,0x00007ff7e1b44000],  sp=0x00007ff7e1b42850,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xa7ecab]
j  sun.misc.Unsafe.putAddress(JJ)V+0
j  Crash.crash()V+5
j  Crash.main([Ljava/lang/String;)V+0
v  ~StubRoutines::call_stub
V  [libjvm.so+0x67133a]
V  [libjvm.so+0x682bca]
V  [libjvm.so+0x6968b6]
C  [libjli.so+0x3989]
C  [libpthread.so.0+0x7dd5]  start_thread+0xc5

  2. Enter the following command to check that you have the same version of debug symbols and the fatal error log file: 

    $ java -version
    Note

    You can also use the sudo update-alternatives --config 'java' to complete this check.

  3. Use the nm command to ensure that libjvm.so has ELF data and text symbols: 

    $ nm /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-2.el8_5/lib/server/libjvm.so-1.8.0.322.b06-2.el8_5.x86_64.debug

Additional resources