How to use pstack to identify hanging threads or high CPU usage in Apache HTTPD

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL)
    • Apache (httpd)
  • Red Hat JBoss Enterprise Web Server (EWS)
    • Apache (httpd)

Issue

  • My Apache server has slow response times how can I determine why the response time are taking such a long time.
  • Apache is unresponsive.
  • Apache response time going slow intermittently.
  • We are noticing high cpu caused by JBoss Enterprise Web Server (EWS)2.0?
  • http service taking more cpu.
  • We are observing high CPU utilization by httpd process in RHEL.
  • We have a RHEL virtual machine, where httpd service taking more CPU and load on the server is also high.
  • Apache consumes too much cpu.
  • We are using apache 2.2.15-39 for web mail and the load on the server during peak hours hovers upto 400(15 minute load average).

Resolution

  1. Increase Apache httpd's logging by editing LogLevel (in /etc/httpd/conf/httpd.conf, unless already specified elsewhere)

    LogLevel debug

  2. To capture more friendly (and verbose) access_logs, modify the LogFormat directive for the relevant format nickname -- e.g., by default that would be the combined nickname in /etc/httpd/conf/httpd.conf

    • Change from the default of:

      LogFormat "%h %l %u %t \"%r\" %>s %b" combined

    • To:

      LogFormat "Time Taken: %T %h %l %u %t \"%r\" %>s %b PID: %P TID: %{tid}P" combined

  3. During slow request times, generate a series of thread dumps to capture what Apache httpd is doing

    • The following compound command will capture 20 thread dumps for all httpd processes (with 20 seconds in between each dump) stored in a pstack.cap file in the current directory

      rm pstack.cap; for y in {1..20}; do echo $y >&2; echo -e "\nDate: $(date)"; for pid in $(ps -C httpd,httpd.worker --noheader -o pid); do echo "    PID: $pid"; pstack $pid; done; sleep 20; done >>pstack.cap

    • If seeing high CPU usage from apache, use the following compound command instead which--in addition to the above--will store top output (CPU details) in a high-cpu.out file in the current directory

      rm pstack.cap high-cpu.out; for y in {1..20}; do echo $y >&2; echo -e "\nDate: $(date)"; top -b -n 1 -H >>high-cpu.out; for pid in $(ps -C httpd,httpd.worker --noheader -o pid); do echo "    PID: $pid"; pstack $pid; done; sleep 20; done >>pstack.cap

    • Note: both of the above loops will take 400 sec (6.6 min)
      To speed things up, lower the sleep value (sleep 20) or the number of loops (the 20 in for y in {1..20})

  4. Once the all the data is captured, correlate information from the access_logs to the pstack.cap file by time and PID to show exactly what code is causing the long response times (also potentially cross-reference PID from high-cpu.out file with the pstack.cap file to clarify the CPU consuming threads and what calls they are making)

Note: On RHEL7 use gstack command which is same like as pstack

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.