Red Hat Enterprise Linux diagnostic tool configuration and recommendations

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL)

Issue

  • Recommendations for diagnostic tools in Red Hat Enterprise Linux (RHEL)
  • What can be used to help diagnose server issues in RHEL?

Resolution

  • Kdump/Netdump

    • Used for capturing kernel vmcores in order to diagnose causes of kernel panics.

    • It is standard practice to install and configure kdump/netdump on all servers that are experiencing kernel panics or system hangs.

    • For kdump configuration in Red Hat Enterprise Linux 5 and above, see:
      How do I configure kexec/kdump on Red Hat Enterprise Linux?

      • In addition to dumping vmcores to disk, kdump can save vmcores over the network via ssh, nfs, and ftp.
    • For netdump server and netdump client configuration in Red Hat Enterprise Linux 4, see:
      How do I configure netdump on Red Hat Enterprise Linux 3 and 4?

      • Netdump can dump vmcores over the network only when a netdump server has been previously configured.

  • Sysstat

    • The sysstat package provides tools (e.g., iostat, mpstat, pidstat) and a cron job that collects information about the system utilization. NOTE: sysstat package under RHEL4 or RHEL5 does not have pidstat. systat package shipped with RHEL 6 and above has pidstat command.

    • By default, the sysstat package is not installed on Red Hat Enterprise Linux 4 and 5. Install it via running yum install sysstat in RHEL5 or up2date sysstat in RHEL4.

    • The sysstat package logs information about load, disk usage, memory/swap usage, network utilization, etc.

    • The default interval for logging is 10 minutes. To change this interval, see the following:
      How can I change the default logging interval for the "sar" monitoring utility?

    • Sysstat in RHEL5 does not capture data about disk activity by default. To change this, see the following:
      Why does sar -d result in "Requested activities not available in file" error?

    • Sysstat does not capture any data about specific processes. If data about specific processes is required please see the information on running top in batch mode

    • It is recommended that sysstat be run on all machines. The load placed on a machine by sysstat is very minimal, and the data is only captured once every 10 minutes (by default).


  • Top in batch mode

    • Running top in batch mode can capture process specific information. This is useful to detect which processes are consuming large amounts of RAM or cpu resources.

    • A simple method for executing top in batch mode would be:

      # top -b -n <number of iterations> -d <delay between updates>
      
    • A more advanced method of capturing batch mode top output would be to write out a script to capture regularly, for example:

      MYLOG="/tmp/top.$(date +%b%d-%T).log"
      while :
      do
        date >>$MYLOG
        top -b -n 1 >>$MYLOG
        echo -e "\n\n" >>$MYLOG
        sleep 60
      done
      
      • This command can be used to capture top output, with a date and time stamp (very useful when comparing with sar data).

      • Adjust the path that the log is written to by changing the MYLOG variable.

      • Adjust the frequency of the capture by changing the sleep value (e.g., sleep 5m).

    • NOTE: It is not recommended to continuously run and capture top in batch mode. Please use top batch mode captures on a case by case basis. While the data can be invaluable in certain situations, in other situations the data is useless. Please consult with Red Hat support for details for specific use cases.

    • Related: How to monitor CPU or memory usage on a per-user basis


  • Ksar

    • An open source graphical application that graphs sar data, ksar can be very helpful, as it allows a user to visually observe trends and spikes found in sar data.

    • Visit the ksar project page for further details and download.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments