Apache HTTPD high CPU on RHEL
Environment
- Red Hat Enterprise Linux (RHEL)
- Red Hat Software Collections (RHSCL)
- Red Hat JBoss Core Services (JBCS)
- Apache Web Server (HTTPD)
Issue
- My Apache server has slow response times how can I determine why the response time are taking such a long time.
- Apache is unresponsive.
- Apache response time going slow intermittently.
- We are noticing high cpu caused by JBoss Enterprise Web Server (EWS)2.0?
- http service taking more cpu.
- We are observing high CPU utilization by httpd process in RHEL.
- We have a RHEL virtual machine, where httpd service taking more CPU and load on the server is also high.
- Apache consumes too much cpu.
- We are using apache 2.2.15-39 for web mail and the load on the server during peak hours hovers upto 400(15-minute load average).
- How to identify high CPU utilization by Apache httpd threads on Linux platform?
Resolution
-
Increase Apache httpd's logging by editing
LogLevel
(in/etc/httpd/conf/httpd.conf
, unless already specified elsewhere)LogLevel debug
-
To capture more friendly (and verbose) access_logs, modify the
LogFormat
directive for the relevant format nickname -- e.g., by default that would be thecombined
nickname in/etc/httpd/conf/httpd.conf
-
Change from the default of:
LogFormat "%h %l %u %t \"%r\" %>s %b" combined
-
To:
LogFormat "Time Taken: %T %h %l %u %t \"%r\" %>s %b PID: %P TID: %{tid}P" combined
-
-
The
pstack
command is provided through thegdb
package:sudo yum install gdb
-
During slow request times, generate a series of thread dumps to capture what Apache httpd is doing
-
The following compound command will capture 20 thread dumps for all httpd processes (with 20 seconds in between each dump) stored in a
pstack.cap
file in the current directoryrm pstack.cap; for y in {1..20}; do echo $y >&2; echo -e "\nDate: $(date)"; for pid in $(ps -C httpd,httpd.worker --noheader -o pid); do echo " PID: $pid"; pstack $pid; done; sleep 20; done >>pstack.cap
-
If seeing high CPU usage from apache, use the following compound command instead which--in addition to the above--will store
top
output (CPU details) in ahigh-cpu.out
file in the current directoryrm pstack.cap high-cpu.out; for y in {1..20}; do echo $y >&2; echo -e "\nDate: $(date)"; top -b -n 1 -H >>high-cpu.out; for pid in $(ps -C httpd,httpd.worker --noheader -o pid); do echo " PID: $pid"; pstack $pid; done; sleep 20; done >>pstack.cap
-
Note: both of the above loops will take 400 sec (6.6 min)
To speed things up, lower the sleep value (sleep 20
) or the number of loops (the20
infor y in {1..20}
)
-
-
Once the all the data is captured, correlate information from the access_logs to the
pstack.cap
file by time and PID to show exactly what code is causing the long response times (also potentially cross-reference PID fromhigh-cpu.out
file with thepstack.cap
file to clarify the CPU consuming threads and what calls they are making)
Note: On RHEL7+ use gstack
command which is same like as pstack
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments