How to diagnose Linux OS issues impacting JBoss performance?

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6
  • Linux with sysstat installed

Issue

JBoss becomes unresponsive though JBoss looks to run fine, with no visible memory leaks or known issues.

Resolution

Interpreting the outputs will show you how the system is behaving under normal operation times.

Root Cause

  • High swap utilization
  • High disk access activity
  • Faulty hard disk drives or storage

Diagnostic Steps

The attached script will generate, locally, the file output.tar.bz2 which will contain 100 samples over a little more than 1 minute.

Each directory will contain the following files:

  • top.out

    • List of processes running and their state and resource consumption.
    • Items to look for:
      • load average
      • Swap utilization
      • RES: Resident Memory Size (KiB), the non-swapped physical memory a task has used.
      • VIRT: Virtual Memory Size (KiB), the total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used
  • vm.out

    • System resource counters
    • Items to look for:
      • swpd: the amount of virtual memory used
      • si: Amount of memory swapped in from disk (/s)
      • so: Amount of memory swapped to disk (/s)
      • bi: Blocks received from a block device (blocks/s)
      • bo: Blocks sent to a block device (blocks/s)
  • io.out
    • Disk access statistics
    • Items to look for:
      • rkB/s: The number of kilobytes read from the device per second
      • wkB/s: The number of kilobytes written to the device per second
  • ps.out
    • Similar to top.out
    • Items to look for:
      • VSZ: virtual memory size of the process in KiB (1024-byte units)
      • RSS: resident set size, the non-swapped physical memory that a task has used (in kiloBytes)
      • CMD: includes the full command-line used to start the process
  • psh.out
    • Similar to ps.out
    • Items to look for:
      • PPID: parent process ID
      • NLWP: number of lwps (threads) in the process
  • sar.out
    • CPU statistics
    • Items to look for:
      • %iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request
      • %steal: Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor
  • free.out
    • Amount of free and used memory in the system, displayed in kiloBytes

Attach the resulting outputs.tar.bz2 to the case you are working on.

Attachments

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.