Chapter 3. Diagnosing Performance Issues

3.1. Enabling Garbage Collection Logging

Examining garbage collection logs can be useful when attempting to troubleshoot Java performance issues, especially those related to memory usage.

Other than some additional disk I/O activity for writing the log files, enabling garbage collection logging does not significantly affect server performance.

Garbage collection logging is already enabled by default for a standalone JBoss EAP server running on OpenJDK or Oracle JDK. For a JBoss EAP managed domain, garbage collection logging can be enabled for the host controller, process controller, or individual JBoss EAP servers.

  1. Get the correct JVM options for enabling garbage collection logging for your JDK. Replace the path in the options below to where you want the log to be created.

    Note

    The Red Hat Customer Portal has a JVM Options Configuration Tool that can help you generate optimal JVM settings.

    • For OpenJDK 8 or Oracle JDK 8:

      -verbose:gc -Xloggc:<path_to_directory>/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading
    • For versions 9 or later of OpenJDK, Oracle JDK, or any JDK that supports JEP 271:

      -Xlog:gc*:file=<path_to_directory>/gc.log:time,uptimemillis:filecount=5,filesize=3M
    • For IBM JDK:

      -Xverbosegclog:<path_to_directory>/gc.log
  2. Apply the garbage collection JVM options to your JBoss EAP server.

    See the JBoss EAP Configuration Guide for instructions on how to apply JVM options to a standalone server or servers in a managed domain.

Additional resources

3.2. Java Heap Dumps

A Java heap dump is a snapshot of a JVM heap created at a certain point in time. Creating and analyzing heap dumps can be useful for diagnosing and troubleshooting issues with Java applications.

Depending on which JDK you are using, there are different ways of creating and analyzing a Java heap dump for a JBoss EAP process. This section covers common methods for Oracle JDK, OpenJDK, and IBM JDK.

3.2.1. Creating a Heap Dump

3.2.1.1. OpenJDK and Oracle JDK

Create an On-Demand Heap Dump

You can use the jcmd command to create an on-demand heap dump for JBoss EAP running on OpenJDK or Oracle JDK.

  1. Determine the process ID of the JVM that you want to create a heap dump from.
  2. Create the heap dump with the following command:

    $ jcmd JAVA_PID GC.heap_dump -all=true FILENAME.hprof

    This creates a heap dump file in the HPROF format, usually located in EAP_HOME or EAP_HOME/bin. Alternatively, you can specify a file path to another directory.

Create a Heap Dump Automatically on OutOfMemoryError

You can use the -XX:+HeapDumpOnOutOfMemoryError JVM option to automatically create a heap dump when an OutOfMemoryError exception is thrown.

This creates a heap dump file in the HPROF format, usually located in EAP_HOME or EAP_HOME/bin. Alternatively, you can set a custom path for the heap dump using -XX:HeapDumpPath=/path/. If you specify a file name using -XX:HeapDumpPath, for example, -XX:HeapDumpPath=/path/filename.hprof, the heap dumps will overwrite each other.

See the JBoss EAP Configuration Guide for instructions on how to apply JVM options to a standalone server or servers in a managed domain.

3.2.1.2. IBM JDK

When using the IBM JDK, heap dumps are automatically generated when an OutOfMemoryError is thrown.

Heap dumps from the IBM JDK are saved in the /tmp/ directory as a portable heap dump (PHD) formatted file.

3.2.2. Analyzing a Heap Dump

Heap Dump Analysis Tools

There are many tools that can analyze heap dump files and help identify issues. Red Hat Support recommends using the Eclipse Memory Analyzer tool (MAT), which can analyze heap dumps formatted in either HPROF or PHD formats.

For information on using Eclipse MAT, see the Eclipse MAT documentation.

Heap Dump Analysis Tips

Sometimes the cause of the heap performance issues are obvious, but other times you may need an understanding of your application’s code and the specific circumstances that cause issues like an OutOfMemoryError. This can help to identify whether an issue is a memory leak, or if the heap is just not large enough.

Some suggestions for identifying memory usage issues include:

  • If a single object is not found to be consuming too much memory, try grouping by class to see if many small objects are consuming a lot of memory.
  • Check if the biggest usage of memory is a thread. A good indicator of this is if the OutOfMemoryError-triggered heap dump is much smaller than the specified Xmx maximum heap size.
  • A technique to make memory leaks more detectable is to temporarily double the normal maximum heap size. When an OutOfMemoryError occurs, the size of the objects related to the memory leak will be about half the size of the heap.

When the source of a memory issue is identified, you can view the paths from garbage collection roots to see what is keeping the objects alive.

3.3. Identifying High CPU Utilization by Java Threads

Note

For customers using JBoss EAP on Red Hat Enterprise Linux or Solaris, the JVMPeg lab tool on the Red Hat Customer Portal helps collect and analyze Java thread information to identify high CPU utilization. Follow the instructions for using the JVMPeg lab tool instead of using the following procedure.

For OpenJDK and Oracle JDK environments, Java thread diagnostic information is available using the jstack utility.

  1. Identify the process ID of the Java process that is utilizing a high percentage of the CPU.

    It can also be useful to obtain per-thread CPU data on high-usage processes. This can be done using the top -H command on Red Hat Enterprise Linux systems.

  2. Using the jstack utility, create a stack dump of the Java process. For example, on Linux and Solaris:

    jstack -l JAVA_PROCESS_ID > high-cpu-tdump.out

    You might need to create multiple dumps at intervals to see any changes or trends over a period of time.

  3. Analyze the stack dumps. You can use a tool such as the Thread Dump Analyzer (TDA).

3.4. Runtime statistics for managed executor services and managed scheduled executor services

You can monitor the performance of managed executor services and managed scheduled executor services by viewing the runtime statistics generated with the management CLI attributes. You can view the runtime statistics for a standalone server or for an individual server mapped to a host.

Important

The domain.xml configuration does not include a resource for the runtime statistic management CLI attributes, so you cannot use the management CLI attributes to view the runtime statistics for a managed domain.

Table 3.1. Displays management CLI attributes for monitoring the performance of managed executor services and of managed scheduled executor services.

AttributeDescription

active-thread-count

The approximate number of threads that are actively executing tasks.

completed-task-count

The approximate total number of tasks that have completed execution.

hung-thread-count

The number of executor threads that are hung.

max-thread-count

The largest number of executor threads.

current-queue-size

The current size of the executor’s task queue.

task-count

The approximate total number of tasks that have been submitted for execution.

thread-count

The current number of executor threads.

Example of viewing the runtime statistics for a managed executor service running on a standalone server.

[standalone@localhost:9990 /] /subsystem=ee/managed-executor-service=default:read-resource(include-runtime=true,recursive=true)

Example of the runtime statistics for a managed scheduled executor service running on a standalone server.

[standalone@localhost:9990 /] /subsystem=ee/managed-scheduled-executor-service=default:read-resource(include-runtime=true,recursive=true)

Example of viewing the runtime statistics for a managed executor service running on a server mapped to a host.

[domain@localhost:9990 /] /host=<host_name>/server=<server_name>/subsystem=ee/managed-executor-service=default:read-resource(include-runtime=true,recursive=true)

Example of the runtime statistics for a managed scheduled executor service running on a server mapped to a host.

[domain@localhost:9990 /] /host=<host_name>/server=<server_name>/subsystem=ee/managed-scheduled-executor-service=default:read-resource(include-runtime=true,recursive=true)

Additional resources