Hawkular-metrics service fails with UnavailableException
Issue
$ oc logs hawkular-metrics-5kqfz
...
Caused by: java.lang.RuntimeException: It appears that not all nodes in the Cassandra cluster are up after 15 checks. Schema updates cannot proceed without all nodes being up.
....
2019-11-14T14:39:08.573236624Z 2019-11-14 14:39:08,567 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-1) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)
2019-11-14T14:39:08.573251623Z at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
2019-11-14T14:39:08.573256823Z at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
2019-11-14T14:39:08.573261123Z at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
2019-11-14T14:39:08.573265623Z at rx.observable.ListenableFutureObservable$2$1.run(ListenableFutureObservable.java:78)
2019-11-14T14:39:08.573270223Z at rx.observable.ListenableFutureObservable$1$1.call(ListenableFutureObservable.java:50)
2019-11-14T14:39:08.573288023Z at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$1.call(EventLoopsScheduler.java:172)
2019-11-14T14:39:08.573293723Z at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
2019-11-14T14:39:08.573298223Z at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2019-11-14T14:39:08.573313223Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2019-11-14T14:39:08.573318423Z at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2019-11-14T14:39:08.573322823Z at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2019-11-14T14:39:08.573327323Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-14T14:39:08.573342923Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-14T14:39:08.573348123Z at java.lang.Thread.run(Thread.java:748)
2019-11-14T14:39:08.573352923Z Caused by: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)
Environment
- OpenShift Container Platform 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.