Hawkular metrics is unavailable due to Cassandra cluster in CrashLoopBackOff in OCP

Solution In Progress - Updated -

Issue

  • Hawkular metrics is unavailable due to Cassandra cluster in CrashLoopBackOff- Cassandra logs show the following error:
ERROR [main] - No columns found for table hawkular_metrics.data_temp_2019013006 in system_schema.columns.  This may be due to corruption or concurrent dropping and altering of a table.  If this table is supposed to be dropped, restart cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and run the following query: "DELETE FROM system_schema.tables WHERE keyspace_name = 'hawkular_metrics' AND table_name = 'data_temp_2019013006';".If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.
Exception (org.apache.cassandra.schema.SchemaKeyspace$MissingColumns) encountered during startup: Columns not found in schema table for hawkular_metrics.data_temp_2019013006
ERROR [main]  CassandraDaemon.java:710 - Exception encountered during startup
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for hawkular_metrics.data_temp_2019013006
    at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1042) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:895) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:883) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for hawkular_metrics.data_temp_2019013006
    at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1042)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:895)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:883)
    at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136)
    at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)

Environment

  • Red Hat OpenShift Container Platform (OCP)
    • 3.9
    • 3.10
    • 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content