Hawkular-metrics presenting issues communicating with cassandra

Solution In Progress - Updated -

Issue

  • Metrics are not reporting and the hawkular-metrics pod is reporting errors about cassandra, but cassandra appears to be healthy
  • Hawkular-metrics is reporting error like this:
[DATETIME] WARN  [org.hawkular.metrics.scheduler.impl.SchedulerImpl] (RxComputationScheduler-3) Job execution of JobDetailsImpl{jobId=[HASH], jobType=TEMP_DATA_COMPRESSOR, jobName=TEMP_DATA_COMPRESSOR, parameters={}, trigger=RepeatingTrigger{triggerTime=1559638800000, interval=7200000, delay=60000}, status=NONE} for time slice 1559638800000 failed: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write)
...
[DATETIME] ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-4) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))
[DATETIME] ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-7) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))

Environment

  • Red Hat OpenShift Container Platform
    • 3.9

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In