Red Hat OpenJDK's Exit codes in podman/Openshift crio container

Updated -

When deploying Java in case of an problem, the following table can be used for troubleshooting by verifying the respective: pod log and the pod yaml, both will be available on the namespace's inspect bundle (not must-gather).

Scenario/JVM flag Expected Exit code
HeapDumpOnOutOfMemoryError Exit code 3
ExitOnOutOfMemoryError Exit code 3
CrashOnOutOfMemoryError Exit code 0
Kill -11 PID Exit code 0
Setting Xmx/Xms> container size (with Always Pre Touch enabled) Exit code 137
Any breach container limit (cgroups) Exit code 137
Slowness (Kubelet probe) Exit code 137

Example where to track the code:

$ oc log $POD
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /opt/infinispan/server/data/java_pid163.hprof ...
Heap dump file created [15866007 bytes in 0.036 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space
…
$ oc get pod PODNAME -o yaml
   lastState:
      terminated:
        containerID: cri-o://9a88ef7f3db3aaa0b89e9a587349aa07beeee86e378908fcd11f013b38b75e41
        exitCode: 3

Example JVM flags

Below example sets GC logs for /tmp/gc.log, OnError ( = to detect crashes), and HeapDumpOnOutOfMemoryError (=to detect OOME) on JAVA_OPTS - both files should go to a Persistence Volume(PV):

env:
      JAVA_OPTS: '-Xlog:gc*=info:file=/tmp/gc.log:time,level,tags,uptimemillis:filecount=10,filesize=1m
              -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/data OnError="touch /path/crash.txt"'

Default Java settings and behavior

As discussed on the Default Red Hat OpenJDK container JVM settings, the OpenJDK container image (and other products such as EAP,DG, AMQ) and will have default JVM flags and this influences the default behavior. This behavior however can be overwritten via JAVA_OPTS for example, which replaces the default flags.

Other References

Problem Issue
Verify cgroups v2 in OpenJDK Cgroups v2 in OpenJDK container in Openshift 4
JVM crash file Interpreting JVM crash file in OpenJDK/OracleJDK
Troubleshooting options Troubleshoot options for Data Grid pod crash
Customizing JVM flags in Openshift Guidelines for customizing JVM flags in Data Grid 8 images
VM.info file interpreting Interpreting VM.info file in OpenJDK/OracleJDK

Comments