[Satellite6] sporadic httpd segfault in libpython2.7.so.1.0 after httpd service reload

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6

Issue

  • service httpd reload (or force-reload) is sometimes executed by a monitoring tool
  • that is time to time followed by httpd segfault like:

Nov 25 14:20:34 my-satellite6 kernel: httpd[5758]: segfault at 8 ip 00007f3e0cfc5a90 sp 00007f3df1bbad70 error 4 in libpython2.7.so.1.0[7f3e0cec6000+17d000]

Resolution

The issue is described by a bugzilla
Resolved by errata and the release of Satellite 6.3.2.

For a workaround, do not (force-)reload httpd service.

Root Cause

(below description is so far just hypothesis) The service reload terminates some threads that might had started to use qpid.messaging library. If Python's Garbage Collector is executed just after that, it might wrongly free some objects from that library, despite they are used by another thread. This thread then raises the segfault when accessing the freed objects.

Diagnostic Steps

System logs having:

Oct 29 03:20:22 satellite.example.com systemd[1]: Reloaded The Apache HTTP Server.
..
Oct 29 03:20:24 satellite.example.com kernel: httpd[46771]: segfault at 8 ip 00007f3f6adb3a90 sp 00007f3f52f78d70 error 4 in libpython2.7.so.1.0[7f3f6acb4000+17d000]

Backtrace of underlying coredump:

#0  _PyTrash_thread_destroy_chain () at /usr/src/debug/Python-2.7.5/Objects/object.c:2510
#1  0x00007f0a15cf4ab8 in call_function (oparg=<optimized out>, pp_stack=0x7f09f8f8cbe0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4410
#2  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f09f40267b0, for file /usr/lib/python2.7/site-packages/django/dispatch/dispatcher.py, line 282, in _remove_receiver (), throwflag=throwflag@entry=0)
    at /usr/src/debug/Python-2.7.5/Python/ceval.c:3040
#3  0x00007f0a15cf6efd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=4, kws=0x557311e47be8, kwcount=0, defs=0x557311b4db68,
    defcount=1, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3640
#4  0x00007f0a15cf43fc in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7f09f8f8cdf0, func=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4504
#5  call_function (oparg=<optimized out>, pp_stack=0x7f09f8f8cdf0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4429
#6  PyEval_EvalFrameEx (
    f=f@entry=Frame 0x557311e47a10, for file /usr/lib/python2.7/site-packages/qpid/compat.py, line 127, in wait (self=<PipeWaiter(write_fd=33, read_fd=32) at remote 0x557311dfc2d8>, timeout=3, start=<float at remote 0x5573103b1458>, ready=False), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3040
#7  0x00007f0a15cf6efd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=2, kws=0x557311e326a8, kwcount=0, defs=0x557311b4dba8,
    defcount=1, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3640
#8  0x00007f0a15cf43fc in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7f09f8f8d000, func=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4504
#9  call_function (oparg=<optimized out>, pp_stack=0x7f09f8f8d000) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4429
..

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.