Using JVM MaxNodeLimit on OpenJDK compilation

Issue

• What is the usage of MaxNodeLimit?
• How this flag affects Shenandoah and CMS collectors?
• Having Live Node limit exceeded limit in Shenandoah:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error
(/home/buildbot/worker/build-jdk8u-redhat-linux/build/hotspot/src/share/vm/opto/node.cpp:75),
pid=101496, tid=0x00007fd2b0cb1700
#  assert(Compile::current()->live_nodes() <
Compile::current()->max_node_limit()) failed: Live Node limit exceeded limit
#
# JRE version: OpenJDK Runtime Environment (8.0) (build
1.8.0-builds.shipilev.net-openjdk-jdk8-redhat-b175-20201026-debug-testing-b00)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b00-debug mixed mode linux-amd64
compressed oops)
# Core dump written. Default location:
/home/e21170/Downloads/core/segv/core or core.101496
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007fd2ac204830):  JavaThread "C2 CompilerThread0"
daemon [_thread_in_native, id=101889,
stack(0x00007fd2b0bb2000,0x00007fd2b0cb2000)]

Stack: [0x00007fd2b0bb2000,0x00007fd2b0cb2000],  sp=0x00007fd2b0cab2a0,
 free space=996k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V  [libjvm.so+0xe0a497]  VMError::report(outputStream*)+0xf0d
V  [libjvm.so+0xe0b903]  VMError::report_and_die()+0x475
V  [libjvm.so+0x68cef0]  report_vm_error(char const*, int, char const*,
char const*)+0xa5
V  [libjvm.so+0xb92ec3]  Node::verify_construction()+0x155
V  [libjvm.so+0xb93d18]  Node::Node(Node*, Node*)+0x62
V  [libjvm.so+0x3dc8a1]  CastX2PNode::CastX2PNode(Node*)+0x25
V  [libjvm.so+0xd1381f]
 ShenandoahBarrierC2Support::pin_and_expand(PhaseIdealLoop*)+0x2629
V  [libjvm.so+0xab474c]  PhaseIdealLoop::build_and_optimize(bool,
bool)+0xc82
V  [libjvm.so+0x61eb88]  PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&,
bool, bool)+0xac
V  [libjvm.so+0xd0b2b9]  ShenandoahBarrierC2Support::expand(Compile*,
PhaseIterGVN&)+0x67
V  [libjvm.so+0x614694]  Compile::Optimize()+0xf22
V  [libjvm.so+0x60dc0e]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*,
int, bool, bool, bool)+0x12ba
V  [libjvm.so+0x53f1ad]  C2Compiler::compile_method(ciEnv*, ciMethod*,
int)+0x119
V  [libjvm.so+0x626c2f]
 CompileBroker::invoke_compiler_on_method(CompileTask*)+0x4d3
V  [libjvm.so+0x626051]  CompileBroker::compiler_thread_loop()+0x32b
V  [libjvm.so+0xdae72c]  compiler_thread_entry(JavaThread*, Thread*)+0x57