Why does HPE Vertica fail with "invalid fastbin entry (free)" error in Red Hat Enterprise Linux 7?

Issue

• During heavy load, HPE Vertica cluster crashes with the following error:

  *** Error in `/opt/vertica/bin/vertica': invalid fastbin entry (free):
  0x00007ef70f209800 ***
  ======= Backtrace: =========
  0x7f0614f0efe1(/lib64/libc.so.6):  + 0x7cfe1
  0x2a1e014(/opt/vertica/bin/vertica) CAT::TabColPair_pairToBytes2(void const*, void*, unsigned long)
  

• Java 8 application gets aborted on RHEL 7 while attempting to free memory.

  (gdb) info stack
  #0  0x00007ff0fe5c65d7 in raise () from /lib64/libc.so.6
  #1  0x00007ff0fe5c7cc8 in abort () from /lib64/libc.so.6
  #2  0x00007ff0fe606e07 in __libc_message () from /lib64/libc.so.6
  #3  0x00007ff0fe60e1fd in _int_free () from /lib64/libc.so.6
  #4  0x00007fefd9c30ba8 in lbm_event_delete (ev=0x7fee98560ae0) at ../../../../src/lib/lbm/lbmev.c:2608
  #5  0x00007fefd9c34b52 in lbm_event_dispatch (evq=0x7ff0f95bc910, tmo=0) at ../../../../src/lib/lbm/lbmev.c:1937
  #6  0x00007fefda0c718c in Java_com_latencybusters_lbm_LBMEventQueue_lbmEventDispatch (env=0x7ff0f95bd1f8, obj=<optimized out>, jevq=6, tmo=-1) at ../../../../src/jni/jni_lbmevq.c:283
  #7  0x00007ff0e37e8cf2 in ?? ()
  #8  0x00000004cb69fc68 in ?? ()
  #9  0x00007ff0e2b01a71 in ?? ()
  #10 0x00007ff0e2ade5d0 in ?? ()
  #11 0x00000004ec717280 in ?? ()
  #12 0x00007fee97af9818 in ?? ()
  #13 0x0000000000000000 in ?? ()
  

• Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap corruption.