Java crash in libc.so.6 in malloc_consolidate

Issue

• Java crashes with Automatic Bug Reporting Tool (ABRT) enabled.
• A fatal error log is not created.
• The fatal error header sent to standard out shows the following:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa65a96c419, pid=22249, tid=0x00007fa634085700
#
# JRE version: OpenJDK Runtime Environment (8.0_262-b10) (build 1.8.0_262-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.262-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x80419]# [ timer expired, abort... ]
*** JBossAS process (22249) received ABRT signal ***                                                   

• The backtrace shows the current thread is WatcherThread:

Thread 1 (Thread 0x7f972628a700 (LWP 26482)):
#0  0x00007f97499f1387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f97499f2a78 in __GI_abort () at abort.c:90
#2  0x00007f974925c459 in os::die () at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1581
#3  0x00007f9749419ff3 in WatcherThread::run (this=0x5625a791f000) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:1379
#4  0x00007f974925b382 in java_start (thread=0x5625a791f000) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:847
#5  0x00007f974a3b9ea5 in start_thread (arg=0x7f972628a700) at pthread_create.c:307
#6  0x00007f9749ab996d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

• The backtrace shows a thread crashing doing malloc:

Thread 79 (Thread 0x7f9722e32700 (LWP 27012)):
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007f9749a43b12 in _L_lock_16654 () at malloc.c:5190
#2  0x00007f9749a40753 in __GI___libc_malloc (bytes=140287755691872) at malloc.c:2903
#3  0x00007f974925307d in os::malloc (size=size@entry=24, memflags=memflags@entry=mtInternal, stack=...)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/runtime/os.cpp:626
#4  0x00007f9748e2ac57 in AllocateHeap (alloc_failmode=AllocFailStrategy::RETURN_NULL, stack=..., flags=mtInternal, size=24)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:56
#5  CHeapObj<(MemoryType)7>::operator new (stack=..., nothrow_constant=..., size=24)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:113
#6  0x00007f9748e2accd in CHeapObj<(MemoryType)7>::operator new (nothrow_constant=..., size=24)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:122
#7  0x00007f9748e2b0ed in create_decoder () at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/utilities/decoder.cpp:74
#8  get_error_handler_instance () at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/utilities/decoder.cpp:59
#9  Decoder::decode (addr=0x80419 <Address 0x80419 out of bounds>, buf=0x7f97499b5dc0 <VMError::report(outputStream*)::buf> "/lib64/libc.so.6", buflen=2000, offset=0x7f9722e2ef10, 
    modulepath=0x7f974a7e6ed0 "/lib64/libc.so.6") at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/utilities/decoder.cpp:108
#10 0x00007f974925c5a6 in os::dll_address_to_function_name (
    addr=addr@entry=0x7f9749a3b419 <malloc_consolidate+281> "H\213C\bL\213{\020I\211\304I\203\344\372J\215,#L\213m\bI\203\345\370\250\001uVH\213\003H)\303I\001\304H\213C\bH\203\340\370H;\004\003\017\205\367\001", buf=buf@entry=0x7f97499b5dc0 <VMError::report(outputStream*)::buf> "/lib64/libc.so.6", buflen=buflen@entry=2000, offset=offset@entry=0x7f9722e2ef10)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1724
#11 0x00007f9748ebab62 in frame::print_C_frame (st=st@entry=0x7f9722e2f650, buf=buf@entry=0x7f97499b5dc0 <VMError::report(outputStream*)::buf> "/lib64/libc.so.6", buflen=buflen@entry=2000, 
    pc=0x7f9749a3b419 <malloc_consolidate+281> "H\213C\bL\213{\020I\211\304I\203\344\372J\215,#L\213m\bI\203\345\370\250\001uVH\213\003H)\303I\001\304H\213C\bH\203\340\370H;\004\003\017\205\367\001")
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/runtime/frame.cpp:702
#12 0x00007f9748ebae40 in frame::print_on_error (this=this@entry=0x7f9722e2efd0, st=st@entry=0x7f9722e2f650, buf=buf@entry=0x7f97499b5dc0 <VMError::report(outputStream*)::buf> "/lib64/libc.so.6", 
    buflen=buflen@entry=2000, verbose=verbose@entry=false) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/runtime/frame.cpp:768
#13 0x00007f9749476852 in VMError::report (this=0x7f9722e2f750, st=st@entry=0x7f9722e2f650)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:507
#14 0x00007f9749478558 in VMError::report_and_die (this=this@entry=0x7f9722e2f750) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:992
#15 0x00007f97492665e5 in JVM_handle_linux_signal (sig=11, info=0x7f9722e2f9f0, ucVoid=0x7f9722e2f8c0, abort_if_unrecognized=<optimized out>)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
#16 0x00007f97492595f8 in signalHandler (sig=11, info=0x7f9722e2f9f0, uc=0x7f9722e2f8c0) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4591
#17 <signal handler called>
#18 malloc_consolidate (av=av@entry=0x7f9749d82760 <main_arena>) at malloc.c:4145
#19 0x00007f9749a3d0a5 in _int_malloc (av=av@entry=0x7f9749d82760 <main_arena>, bytes=bytes@entry=8192) at malloc.c:3444
#20 0x00007f9749a406fc in __GI___libc_malloc (bytes=8192) at malloc.c:2905
#21 0x00007f973aa23281 in ?? ()
#22 0x00007f9722e30060 in ?? ()
#23 0x00007f97492fc00f in OptoRuntime::register_finalizer (obj=<optimized out>, thread=0x6e7bb2020)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/opto/runtime.cpp:1520
...

• The backtrace shows another thread with a failed malloc (AllocateHeap (alloc_failmode=AllocFailStrategy::EXIT_OOM):

Thread 50 (Thread 0x7f97229e9700 (LWP 26821)):
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007f9749a43b12 in _L_lock_16654 () at malloc.c:5190
#2  0x00007f9749a40753 in __GI___libc_malloc (bytes=140287755691872) at malloc.c:2903
#3  0x00007f974925307d in os::malloc (size=size@entry=24, memflags=memflags@entry=mtInternal, stack=...)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/runtime/os.cpp:626
#4  0x00007f9748c20299 in AllocateHeap (alloc_failmode=AllocFailStrategy::EXIT_OOM, stack=..., flags=mtInternal, size=24)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:56
#5  CHeapObj<(MemoryType)7>::operator new (size=size@entry=24, stack=...) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:99
#6  0x00007f9748c20355 in CHeapObj<(MemoryType)7>::operator new (size=size@entry=24)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:107
#7  0x00007f97492b6ef3 in add_seen_thread (action=PlaceholderTable::LOAD_INSTANCE, thread=0x5625aaafa000, this=0x5625a7745c58)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/classfile/placeholders.hpp:274
#8  PlaceholderTable::find_and_add (this=<optimized out>, index=index@entry=995, hash=hash@entry=63422699, name=name@entry=0x5625ad7799c0, loader_data=loader_data@entry=0x5625a9ed9540, 
    action=action@entry=PlaceholderTable::LOAD_INSTANCE, supername=supername@entry=0x0, thread=thread@entry=0x5625aaafa000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/classfile/placeholders.cpp:140
#9  0x00007f97493eb234 in SystemDictionary::resolve_instance_class_or_null (name=name@entry=0x5625ad7799c0, class_loader=..., protection_domain=..., __the_thread__=0x5625aaafa000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp:781
#10 0x00007f97493eb94e in SystemDictionary::resolve_or_null (class_name=class_name@entry=0x5625ad7799c0, class_loader=..., class_loader@entry=..., protection_domain=..., protection_domain@entry=..., 
    __the_thread__=__the_thread__@entry=0x5625aaafa000) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp:245
#11 0x00007f97493eba98 in SystemDictionary::resolve_or_fail (class_name=0x5625ad7799c0, class_loader=..., protection_domain=..., throw_error=<optimized out>, __the_thread__=__the_thread__@entry=0x5625aaafa000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp:184
#12 0x00007f974905c040 in find_class_from_class_loader (env=0x5625aaafa2d0, name=<optimized out>, init=<optimized out>, loader=..., protection_domain=..., throwError=<optimized out>, 
    __the_thread__=0x5625aaafa000) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/prims/jvm.cpp:4250
#13 0x00007f974905c3fd in JVM_FindClassFromCaller (env=env@entry=0x5625aaafa2d0, name=name@entry=0x7f97229e6230 "org/xmlpull/mxp1_serializer/MXSerializer", init=init@entry=1 '\001', 
    loader=loader@entry=0x7f97229e6310, caller=caller@entry=0x7f97229e6318) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/hotspot/src/share/vm/prims/jvm.cpp:1054
#14 0x00007f9747acbc90 in Java_java_lang_Class_forName0 (env=0x5625aaafa2d0, this=<optimized out>, classname=0x7f97229e6300, initialize=<optimized out>, loader=0x7f97229e6310, caller=0x7f97229e6318)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/openjdk/jdk/src/share/native/java/lang/Class.c:135
...

• The sosreport created by ABRT shows plenty of available physical memory.