httpd child process destruction stalls in wsgi_python_child_cleanup
Issue
- After some time, httpd becomes unresponsive and begins to continuously log the following:
 
[mpm_event:error] [pid 1577:tid 1577] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
- All child processes are created and they are all persisting in wsgi_python_child_cleanup:
 
    PID: 655889
#0  0x000003ffa649ab26 in __futex_abstimed_wait_cancelable64 () from target:/lib64/libc.so.6
#1  0x000003ffa64a63d8 in do_futex_wait.constprop () from target:/lib64/libc.so.6
#2  0x000003ffa64a6478 in __new_sem_wait_slow64.constprop.0 () from target:/lib64/libc.so.6
#3  0x000003ffa1baafba in PyThread_acquire_lock_timed () from target:/lib64/libpython3.9.so.1.0
#4  0x000003ffa1c65426 in acquire_timed () from target:/lib64/libpython3.9.so.1.0
#5  0x000003ffa1c651be in lock_PyThread_acquire_lock () from target:/lib64/libpython3.9.so.1.0
#6  0x000003ffa1bd3132 in method_vectorcall_VARARGS_KEYWORDS () from target:/lib64/libpython3.9.so.1.0
#7  0x000003ffa1bc0e8c in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.9.so.1.0
#8  0x000003ffa1bd1e24 in function_code_fastcall () from target:/lib64/libpython3.9.so.1.0
#9  0x000003ffa1bd19e8 in _PyFunction_Vectorcall () from target:/lib64/libpython3.9.so.1.0
#10 0x000003ffa2011f48 in ShutdownInterpreter_call () from target:/etc/httpd/modules/mod_wsgi_python3.so
#11 0x000003ffa1bcb832 in _PyObject_MakeTpCall () from target:/lib64/libpython3.9.so.1.0
#12 0x000003ffa1be8604 in PyObject_VectorcallMethod () from target:/lib64/libpython3.9.so.1.0
#13 0x000003ffa1ca13a4 in wait_for_thread_shutdown () from target:/lib64/libpython3.9.so.1.0
#14 0x000003ffa1b9e072 in Py_EndInterpreter () from target:/lib64/libpython3.9.so.1.0
#15 0x000003ffa2010a7e in Interpreter_dealloc () from target:/etc/httpd/modules/mod_wsgi_python3.so
#16 0x000003ffa1bbc1d8 in free_keys_object () from target:/lib64/libpython3.9.so.1.0
#17 0x000003ffa2019e4c in wsgi_python_child_cleanup () from target:/etc/httpd/modules/mod_wsgi_python3.so
#18 0x000003ffa66ad052 in apr_pool_destroy () from target:/lib64/libapr-1.so.0
#19 0x000003ffa3e863f8 in clean_child_exit () from target:/etc/httpd/modules/mod_mpm_event.so
#20 0x000003ffa3e89180 in child_main () from target:/etc/httpd/modules/mod_mpm_event.so
#21 0x000003ffa3e89744 in make_child () from target:/etc/httpd/modules/mod_mpm_event.so
#22 0x000003ffa3e89806 in startup_children () from target:/etc/httpd/modules/mod_mpm_event.so
#23 0x000003ffa3e8a8f4 in event_run () from target:/etc/httpd/modules/mod_mpm_event.so
#24 0x000002aa39cae3ca in ap_run_mpm ()
#25 0x000002aa39c996d8 in main ()
  Environment
- Red Hat Enterprise Linux (RHEL)
 - Apache httpd
 - python3-mod_wsgi
 
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.