Java 11.0.16.0 / 17.0.4.0 suddenly spikes in memory and crashes

Issue

• We recently updated our JDK to 11.0.16.0 or 17.0.4.0. Now at times we see the java process suddenly spikes in memory usage before crashing soon after. hs_err_pid files show large malloc attempts from a jvm compiler thread like below in a compile task for sun.security.ssl.SSLEngineInputRecord::decodeInputRecord

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1073741840 bytes for Chunk::new
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Zero Based Compressed Oops mode in which the Java heap is
#     placed in the first 32GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 32GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (arena.cpp:197), pid=2907, tid=2927
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el7_9) (11.0.16+8) (build 11.0.16+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-11.0.16.0.8-1.el7_9) (11.0.16+8-LTS, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007f5134b1b000):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=2927, stack(0x00007f5138229000,0x00007f513832a000)]


Current CompileTask:
C2:299829840 17165   !   4       sun.security.ssl.SSLEngineInputRecord::decodeInputRecord (812 bytes)

Stack: [0x00007f5138229000,0x00007f513832a000],  sp=0x00007f5138324360,  free space=1004k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xf077e2]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1c2
V  [libjvm.so+0xf08793]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x43
V  [libjvm.so+0x6fc210]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0x110
V  [libjvm.so+0x47d741]  Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x1f1
V  [libjvm.so+0x47d982]  Arena::Arealloc(void*, unsigned long, unsigned long, AllocFailStrategy::AllocFailEnum)+0x1f2
V  [libjvm.so+0xc2417a]  Node::clone() const+0x29a
V  [libjvm.so+0xde42d0]  PhaseIdealLoop::split_up(Node*, Node*, Node*) [clone .part.114]+0xc50
V  [libjvm.so+0xde6090]  PhaseIdealLoop::do_split_if(Node*)+0x660
V  [libjvm.so+0xafb9f9]  PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&, bool)+0x129
V  [libjvm.so+0xaf61a5]  PhaseIdealLoop::build_and_optimize()+0x11e5
V  [libjvm.so+0x69b05b]  Compile::optimize_loops(int&, PhaseIterGVN&, LoopOptsMode)+0x2bb
V  [libjvm.so+0x69cb3b]  Compile::Optimize()+0xb5b
V  [libjvm.so+0x69e61d]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, bool, DirectiveSet*)+0x11ad
V  [libjvm.so+0x5bffb4]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x124
V  [libjvm.so+0x6a765b]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x43b
V  [libjvm.so+0x6a8e68]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xe9ee72]  JavaThread::thread_main_inner()+0x232
V  [libjvm.so+0xe9b1ca]  Thread::call_run()+0x16a
V  [libjvm.so+0xc5d8d8]  thread_native_entry(Thread*)+0xe8