Deploying Metrics into 3.4 fails
Environment
OpenShift Container Platform 3.4.0
Issue
We have automation under 3.3 to deploy OpenShift Metrics that deploys using the supplied template and lets the deployment with a single pod complete before removing the Cassandra portion and deploying our own 3 node cluster with local EBS storage. This was working fine in 3.3.
Under 3.4 it looks to work but then Hawkular-Metrics starts complaining about missing schema components.
Root Cause
Metrics components in particular cassandra were using images for Openshift 3.3 instead of versions for Openshift 3.4.
Redeploy cassandra with latest 3.4 image.
Diagnostic Steps
logs from hawkular-metrics pod show
14:06:34,201 ERROR [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 71) Failed schema check: com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace system_schema does not exist
EXECUTING CQL:
SELECT table_name FROM system_schema.tables
WHERE keyspace_name='hawkular_alerts' AND table_name = 'triggers';
14:06:34,201 WARN [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 71) [15] Keyspace detected but schema not fully created. Retrying in [3000] ms...
14:06:37,204 ERROR [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 71) Failed schema check: com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace system_schema does not exist
EXECUTING CQL:
SELECT table_name FROM system_schema.tables
WHERE keyspace_name='hawkular_alerts' AND table_name = 'triggers';
14:06:37,204 WARN [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 71) [14] Keyspace detected but schema not fully created. Retrying in [3000] ms...
Checking description of cassandra pods showed old image version being used
$ grep Image *cassandra*.describe
hawkular-cassandra-1-69s43.describe: Image: registry:5000/openshift3/metrics-cassandra:3.3.1
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
