Hawkular metrics pods fail or unready due to incorrect image version

Solution In Progress - Updated -

Issue

Hawkular-cassandra and hawkular-metrics pod shows either status Running, Error or CrashLoopBackOff and is unready. This can happen in one cluster, or simultaneously in a few different OpenShift clusters as seen below:

[root@server1 ~]# oc get pod
NAME                            READY     STATUS             RESTARTS   AGE
hawkular-cassandra-1-cwg6p      0/1       CrashLoopBackOff   33         2h
hawkular-metrics-k78t7          0/1       Running            865        5d
hawkular-metrics-schema-tw6kh   0/1       Completed          0          11d
heapster-tbkt9                  0/1       Running            867        5d

[root@server2 ~]# oc get pod
NAME                            READY     STATUS             RESTARTS   AGE
hawkular-cassandra-1-82gpn      0/1       CrashLoopBackOff   928        12d
hawkular-metrics-cqgr5          0/1       Running            526        12d
hawkular-metrics-schema-7kw4z   0/1       Completed          0          12d
heapster-lpsfm                  0/1       Running            523        12d

[root@server3 ~]# oc get pod
NAME                            READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-gtjzr      0/1       Error       927        14d
hawkular-metrics-schema-5zjs6   0/1       Completed   0          14d
hawkular-metrics-sprtv          0/1       Running     532        14d
heapster-vhh6g                  0/1       Running     523        14d

Pod logs may show permissions errors:

sed: cannot rename /opt/apache-cassandra/conf/sed6b3JJH: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedIBVSPF: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedQyRWxG: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedtayTwJ: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedWAzVmK: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedun6iVJ: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sed0lEnoI: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sed4LogeJ: Operation not permitted
sed: cannot rename /opt/apache-cassandra/conf/sedfEYpVI: Operation not permitted

Pod logs show a schema error:

2019-06-20 02:04:55,466 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2019-06-20 02:05:05,470 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist

Environment

OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content