Cassandra pods are crashing due to OutOfMemoryError on RHOCP 3.x

Issue

• hawkular-cassandra-1-xxxx pod is down with the below error:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"

• hawkular-metrics-xxxx pod is down with the below error:

ESC[0mESC[33m2021-07-07 13:52:30,836 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [87] Retrying connecting to Cassandra cluster in [3]s...
ESC[0mESC[0m2021-07-07 13:52:30,840 INFO  [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceOverrideMapper] (ServerService Thread Pool -- 59) Could not connect to Cassandra. This could mean Cassandra is not yet up and running. Will try again. Error message: All host(s) tried for query failed (tried: hawkular-cassandra/10.37.159.184:9042 (com.datastax.driver.core.exceptions.TransportException: [hawkular-cassandra/10.37.159.184:9042] Cannot connect))
(END)

• heapster-xxxx pod is down with the below error:

E0707 13:52:04.998946       1 client.go:203] Could not update tags: Put https://hawkular-metrics:443/hawkular/metrics/gauges/pod%2Fcd9daf5f-ded7-11eb-95ba-fa163e47c7a2%2Ffilesystem%2Fusage%2FVolume%3Amongodb-certs/tags: dial tcp 10.37.181.38:443: getsockopt: no route to host