Hawkular metrics is unavailable due to Cassandra cluster in CrashLoopBackOff in OCP
Issue
Hawkular metrics
is unavailable due toCassandra
cluster in CrashLoopBackOff- Cassandra logs show the following error:
ERROR [main] - No columns found for table hawkular_metrics.data_temp_2019013006 in system_schema.columns. This may be due to corruption or concurrent dropping and altering of a table. If this table is supposed to be dropped, restart cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and run the following query: "DELETE FROM system_schema.tables WHERE keyspace_name = 'hawkular_metrics' AND table_name = 'data_temp_2019013006';".If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.
Exception (org.apache.cassandra.schema.SchemaKeyspace$MissingColumns) encountered during startup: Columns not found in schema table for hawkular_metrics.data_temp_2019013006
ERROR [main] CassandraDaemon.java:710 - Exception encountered during startup
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for hawkular_metrics.data_temp_2019013006
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1042) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:895) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:883) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for hawkular_metrics.data_temp_2019013006
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1042)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:895)
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:883)
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136)
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
Environment
- Red Hat OpenShift Container Platform (OCP)
- 3.9
- 3.10
- 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.