Hawkular-metrics service fails with UnavailableException

Solution Verified - Updated -

Issue

$ oc logs hawkular-metrics-5kqfz
... 
Caused by: java.lang.RuntimeException: It appears that not all nodes in the Cassandra cluster are up after 15 checks. Schema updates cannot proceed without all nodes being up.
....
2019-11-14T14:39:08.573236624Z 2019-11-14 14:39:08,567 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-1) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)
2019-11-14T14:39:08.573251623Z  at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
2019-11-14T14:39:08.573256823Z  at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
2019-11-14T14:39:08.573261123Z  at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
2019-11-14T14:39:08.573265623Z  at rx.observable.ListenableFutureObservable$2$1.run(ListenableFutureObservable.java:78)
2019-11-14T14:39:08.573270223Z  at rx.observable.ListenableFutureObservable$1$1.call(ListenableFutureObservable.java:50)
2019-11-14T14:39:08.573288023Z  at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$1.call(EventLoopsScheduler.java:172)
2019-11-14T14:39:08.573293723Z  at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
2019-11-14T14:39:08.573298223Z  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2019-11-14T14:39:08.573313223Z  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2019-11-14T14:39:08.573318423Z  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2019-11-14T14:39:08.573322823Z  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2019-11-14T14:39:08.573327323Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-14T14:39:08.573342923Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-14T14:39:08.573348123Z  at java.lang.Thread.run(Thread.java:748)
2019-11-14T14:39:08.573352923Z Caused by: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)

Environment

  • OpenShift Container Platform 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content