Hawkular-metrics presenting issues communicating with cassandra

Solution In Progress - Updated -

Issue

  • Metrics are not reporting and the hawkular-metrics pod is reporting errors about cassandra, but cassandra appears to be healthy
  • Hawkular-metrics is reporting error like this:
[DATETIME] WARN  [org.hawkular.metrics.scheduler.impl.SchedulerImpl] (RxComputationScheduler-3) Job execution of JobDetailsImpl{jobId=[HASH], jobType=TEMP_DATA_COMPRESSOR, jobName=TEMP_DATA_COMPRESSOR, parameters={}, trigger=RepeatingTrigger{triggerTime=1559638800000, interval=7200000, delay=60000}, status=NONE} for time slice 1559638800000 failed: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write)
...
[DATETIME] ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-4) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))
[DATETIME] ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-7) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/[IP:PORT] (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/[IP:PORT]] Timed out waiting for server response))

Environment

  • Red Hat OpenShift Container Platform
    • 3.9

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content