high CPU and memory in httpd from mod_proxy_hcheck

Issue

• With mod_proxy_hcheck, httpd is hitting high CPU and memory. We have high CPU threads in calls like below:

#0  0x00007fd4491883ad in add_task () from /usr/lib64/libaprutil-1.so.0
#1  0x00007fd43dffdf99 in hc_watchdog_callback (state=<optimized out>, data=0xbd87c8, pool=<optimized out>) at mod_proxy_hcheck.c:975
#2  0x00007fd43ddf6198 in wd_worker (thread=<optimized out>, data=0xbac5a0) at mod_watchdog.c:209
#3  0x00007fd4480f3ea5 in start_thread () from /usr/lib64/libpthread.so.0
#4  0x00007fd447c18b0d in clone () from /usr/lib64/libc.so.6

• And many threads are also accumulating pending healthcheck attempts:

#0  0x00007ff39d5b0ddd in poll () from /usr/lib64/libc.so.6
#1  0x00007ff39e317ffb in apr_wait_for_io_or_timeout () from /usr/lib64/libapr-1.so.0
#2  0x00007ff39e313a1b in apr_socket_connect () from /usr/lib64/libapr-1.so.0
#3  0x00007ff39c673cf4 in ap_proxy_connect_backend (proxy_function=proxy_function@entry=0x7ff3939a19c0 "HCOH", conn=0x7ff31405f340, worker=worker@entry=0x7ff3140e1d10, s=0x231ebb8) at proxy_util.c:3066
#4  0x00007ff39399fc5c in hc_check_http (baton=0x7ff2d7e09020) at mod_proxy_hcheck.c:775
#5  hc_check (thread=0x7ff314000d58, b=0x7ff2d7e09020) at mod_proxy_hcheck.c:851
#6  0x00007ff39eb2aded in thread_pool_func () from /usr/lib64/libaprutil-1.so.0
#7  0x00007ff39da96ea5 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ff39d5bbb0d in clone () from /usr/lib64/libc.so.6

• When a balancer member has been down for a time, we see all healthcheck activity stop or slow. After that balancer member recovers, healthchecks then occur in excess for all balancer members much faster than the hcinterval.