Readiness and Liveness probe failed for hawkular metrics

Solution Verified - Updated -

Issue

  • In fresh RHOCP v3.11.43 cluster installation, hawkular-metrics pod does not come up and goes into crashloopbackoff state.
# oc get pods
NAME                            READY     STATUS             RESTARTS   AGE
hawkular-cassandra-1-r6ckd      1/1       Running            1          6d
hawkular-metrics-94hxj          0/1       CrashLoopBackOff   2803       6d
hawkular-metrics-schema-d65rc   0/1       Completed          0          6d
heapster-l6wcg                  0/1       Running            1064       6d

Events:
  Type     Reason     Age                   From                         Message
  ----     ------     ----                  ----                         -------
  Normal   Killing    1h (x2801 over 6d)    kubelet, xyz  Killing container with id docker://hawkular-metrics:Container failed liveness probe.. Container will be killed and recreated.
  Warning  Unhealthy  34m (x15716 over 6d)  kubelet, xyz Readiness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>. This may be due to Hawkular Metrics not being ready yet. Will try again.
  Warning  Unhealthy  15m (x8463 over 6d)   kubelet, xyz  Liveness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>.
Traceback (most recent call last):
  File "/opt/hawkular/scripts/hawkular-metrics-liveness.py", line 48, in <module>
    if int(uptime) < int(timeout):

Environment

  • Red Hat Openshift Container Platform
    • 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content