Apache httpd is frequently crashing when sending a client cert to backend server using mod_proxy with Two-way SSL

Solution Verified - Updated -

Issue

  • The httpd is configured as a reverse proxy server which connects with the backend server with 2-way ssl.When it tries to send a client certificate, it exits with abort:
#0  0x00007f2774430387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f2774431a78 in __GI_abort () at abort.c:90
#2  0x00007f2774472ed7 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f2774585350 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007f277447b299 in malloc_printerr (ar_ptr=0x7f27747c1760 <main_arena>, ptr=<optimized out>, 
    str=0x7f2774585410 "double free or corruption (fasttop)", action=3) at malloc.c:4967
#4  _int_free (av=0x7f27747c1760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5  0x00007f276942580d in CRYPTO_free (str=0x55e8a8924950) at mem.c:434
#6  0x00007f27694eb556 in EVP_PKEY_free (x=<optimized out>) at p_lib.c:406
#7  0x00007f276983f92d in ssl3_send_client_certificate (s=s@entry=0x7f2754024050) at s3_clnt.c:3469
#8  0x00007f276983fecf in ssl3_connect (s=0x7f2754024050) at s3_clnt.c:421
#9  0x00007f2769849e7e in ssl23_get_server_hello (s=0x7f2754024050) at s23_clnt.c:799
#10 ssl23_connect (s=0x7f2754024050) at s23_clnt.c:228
#11 0x00007f2769aa10d8 in ssl_io_filter_handshake (filter_ctx=filter_ctx@entry=0x7f27480091d0) at ssl_engine_io.c:1086
#12 0x00007f2769aa22f4 in ssl_io_filter_output (f=0x7f2748009228, bb=0x7f2754004840) at ssl_engine_io.c:1616
#13 0x00007f2769a9ef9a in ssl_io_filter_coalesce (f=0x7f2748009200, bb=0x7f2754004840) at ssl_engine_io.c:1584
#14 0x00007f276b92c877 in ap_proxy_pass_brigade (bucket_alloc=bucket_alloc@entry=0x7f27540008e8, r=r@entry=0x7f2754002970, 
    p_conn=p_conn@entry=0x7f2754006990, origin=origin@entry=0x7f2748008bb0, bb=bb@entry=0x7f2754004840, flush=flush@entry=1)
    at proxy_util.c:3556
#15 0x00007f276a0d4061 in stream_reqbody_cl (old_cl_val=<optimized out>, input_brigade=0x7f2754010b38, header_brigade=0x7f2754004840, 
    origin=0x7f2748008bb0, p_conn=0x7f2754006990, r=0x7f2754002970, p=0x7f27540028f8) at mod_proxy_http.c:482
#16 ap_proxy_http_request (server_portstr=0x7f276532aa20 "", url=<optimized out>, uri=0x7f27540047a0, conf=0x55e8a88e27c0, 
    worker=<optimized out>, p_conn=0x7f2754006990, r=0x7f2754002970, p=0x7f27540028f8) at mod_proxy_http.c:933
#17 proxy_http_handler (r=<optimized out>, worker=<optimized out>, conf=0x55e8a88e27c0, 
    url=0x7f2754004546 "https://localhost:10443/test/", proxyname=0x0, proxyport=0) at mod_proxy_http.c:2022
#18 0x00007f276b923f34 in proxy_run_scheme_handler (r=r@entry=0x7f2754002970, worker=0x55e8a88e6ef8, conf=conf@entry=0x55e8a88e27c0, 
    url=0x7f2754004546 "https://localhost:10443/test/", proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0)
    at mod_proxy.c:2746
#19 0x00007f276b924e1d in proxy_handler (r=0x7f2754002970) at mod_proxy.c:1125
#20 0x000055e8a7e03f40 in ap_run_handler (r=r@entry=0x7f2754002970) at config.c:169
#21 0x000055e8a7e04489 in ap_invoke_handler (r=r@entry=0x7f2754002970) at config.c:439
#22 0x000055e8a7e190da in ap_process_async_request (r=r@entry=0x7f2754002970) at http_request.c:339
#23 0x000055e8a7e193c2 in ap_process_request (r=r@entry=0x7f2754002970) at http_request.c:374
#24 0x000055e8a7e15552 in ap_process_http_sync_connection (c=0x7f2758004e18) at http_core.c:190
#25 ap_process_http_connection (c=0x7f2758004e18) at http_core.c:231
#26 0x000055e8a7e0d570 in ap_run_process_connection (c=c@entry=0x7f2758004e18) at connection.c:41
#27 0x000055e8a7e0d988 in ap_process_connection (c=c@entry=0x7f2758004e18, csd=csd@entry=0x7f2758004c00) at connection.c:212
#28 0x00007f276bb3b76b in process_socket (bucket_alloc=0x7f27540008e8, my_thread_num=2, my_child_num=0, sock=0x7f2758004c00, 
    p=0x7f2758004b78, thd=0x55e8a8943d00) at worker.c:620
#29 worker_thread (thd=0x55e8a8943d00, dummy=<optimized out>) at worker.c:979
#30 0x00007f27749d3ea5 in start_thread (arg=0x7f276532b700) at pthread_create.c:307
#31 0x00007f27744f88dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Environment

  • Red Hat Enterprise Linux (RHEL) 7.8
    • httpd-2.4.6-95
    • mod_ssl-2.4.6-95
    • openssl-1.0.2k-19
    • openssl-libs-1.0.2k-19

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content