Cassandra and hawkular-metrics pod continuously getting restarted after upgrading OpenShift
Issue
- Cassandra pod getting restarted at regular intervals
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-lpq9j 0/1 Running 176 21h
hawkular-metrics-jfxmw 0/1 Running 1007 14d
heapster-zj6xl 1/1 Running 0 3d
- Readiness probes for all the pods in
openshift-infranamespace failing. - Cassandra pod logs might show these error messages:
$ oc logs <cassandra-pod>
ERROR [MemtableFlushWriter:3] 2019-11-08 12:21:33,073 LogTransaction.java:262 - [/cassandra_data/data/hawkular_metrics/data_temp_2019110410-13869ff0fe4111e991c5dfe028fea12c/mc_txn_flush_4ce856a0-0222-11ea-b41b-b94758806e55.log] was not completed, trying to abort it now
ERROR [MemtableFlushWriter:3] 2019-11-08 12:21:33,075 LogTransaction.java:265 - Failed to abort [/cassandra_data/data/hawkular_metrics/data_temp_2019110410-13869ff0fe4111e991c5dfe028fea12c/mc_txn_flush_4ce856a0-0222-11ea-b41b-b94758806e55.log]
java.lang.RuntimeException: java.nio.file.AccessDeniedException: /cassandra_data/data/hawkular_metrics/data_temp_2019110410-13869ff0fe4111e991c5dfe028fea12c/mc_txn_flush_4ce856a0-0222-11ea-b41b-b94758806e55.log
Caused by: java.nio.file.AccessDeniedException: /cassandra_data/data/hawkular_metrics/data_temp_2019110410-13869ff0fe4111e991c5dfe028fea12c/mc_txn_flush_4ce856a0-0222-11ea-b41b-b94758806e55.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[na:1.8.0_222]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_222]
Environment
- Red Hat OpenShift Container Platform
- 3.9
- 3.10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.