Hawkular metrics pods fail or unready due to incorrect image version
Issue
Hawkular-cassandra and hawkular-metrics pod shows either status Running, Error or CrashLoopBackOff and is unready. This can happen in one cluster, or simultaneously in a few different OpenShift clusters as seen below:
[root@server1 ~]# oc get pod
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-cwg6p 0/1 CrashLoopBackOff 33 2h
hawkular-metrics-k78t7 0/1 Running 865 5d
hawkular-metrics-schema-tw6kh 0/1 Completed 0 11d
heapster-tbkt9 0/1 Running 867 5d
[root@server2 ~]# oc get pod
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-82gpn 0/1 CrashLoopBackOff 928 12d
hawkular-metrics-cqgr5 0/1 Running 526 12d
hawkular-metrics-schema-7kw4z 0/1 Completed 0 12d
heapster-lpsfm 0/1 Running 523 12d
[root@server3 ~]# oc get pod
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-gtjzr 0/1 Error 927 14d
hawkular-metrics-schema-5zjs6 0/1 Completed 0 14d
hawkular-metrics-sprtv 0/1 Running 532 14d
heapster-vhh6g 0/1 Running 523 14d
Pod logs may show permissions errors:
sed: cannot rename /opt/apache-cassandra/conf/sed6b3JJH: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedIBVSPF: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedQyRWxG: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedtayTwJ: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedWAzVmK: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedun6iVJ: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sed0lEnoI: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sed4LogeJ: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedfEYpVI: Operation not permitted
Pod logs show a schema error:
2019-06-20 02:04:55,466 INFO [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2019-06-20 02:05:05,470 INFO [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
Environment
OpenShift Container Platform 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.