Why is the real time significantly larger than the user and sys time for GC ?

Solution Unverified - Updated -

Environment

  • VMWare
  • Red Hat Enterprise Linux (RHEL)
  • Java
  • Red Hat JBoss Enterprise Application Platform

Issue

The real time appears to be orders of magnitude larger than the user and sys time for Full Garbage Collection (GC)

3214.214: [Full GC [PSYoungGen: 69218K->0K(593728K)] [PSOldGen: 1378444K->1263945K(1398144K)] 1447663K->1263945K(1991872K) [PSPermGen: 207134K->207134K(421504K)], 42.3680890 secs] [Times: user=4.58 sys=0.37, real=42.37 secs]
.. ..
51459.604: [Full GC (System) [PSYoungGen: 22081K->0K(571328K)] [PSOldGen: 1339894K->1226216K(1398144K)] 1361975K->1226216K(1969472K) [PSPermGen: 201287K->201287K(262400K)], 48.6652560 secs] [Times: user=3.98 sys=0.30, real=48.65 secs]

Reclaiming a slightly higher heap space later on did not show the same discrepancy between the user + sys and real time

11311.352: [Full GC [PSYoungGen: 93900K->0K(521664K)] [PSOldGen: 1368816K->1021761K(1398144K)] 1462717K->1021761K(1919808K) [PSPermGen: 202939K->202939K(386816K)], 3.4639640 secs] [Times: user=3.43 sys=0.00, real=3.46 secs]

Resolution

  • Paging has an impact on performance as it can lead to an increase in the garbage collection time.
    For a system which uses a significant chunk of swap space, the time taken to page a portion of the JVM heap into memory will increase the time scale of the garbage collector.
    • Look at increasing the memory allocation to the VM and JVM
    • Try using Large Pages

Root Cause

  • Swap space exhaustion
  • Excessive page swapping

Diagnostic Steps

Linux

  • Verify whether the system is swapping by analyzing the System Activity Report (SAR).
  • Running the command sar will show the latest metrics CPU utilization, swap space usage and paging metrics. By default, metrics spanning 28 days are stored. This can be found in the /var/log/sa/ directory.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments