Elasticsearch pods fail to start with ImagePullBackOff status

Solution Verified - Updated -

Issue

Elasticsearch pods stay in failed state with ImagePullBackOff and on the 'oc get events' there is the warning message:

38m         Warning   Failed                         pod/elasticsearch-cdm-servicemeshjaeger-1-69f5f78754-gbq6n                  Failed to pull image "registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a": rpc error: code = Unknown desc = Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference&#32;&#35;132&#46;248c3554&#46;1596114557&#46;286726\n</BODY></HTML>\n"

Trying to pull manually the image the same error appears:

 $ podman --log-level=debug pull registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a
[Omitted]
DEBU[0003] Accessing "registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a" failed: Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference&#32;&#35;132&#46;72221102&#46;1596116349&#46;1e6e6df9\n</BODY></HTML>\n" 
DEBU[0003] Error pulling image ref //registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error initializing source docker://registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference&#32;&#35;132&#46;72221102&#46;1596116349&#46;1e6e6df9\n</BODY></HTML>\n" 
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference&#32;&#35;132&#46;72221102&#46;1596116349&#46;1e6e6df9\n</BODY></HTML>\n"
Error: unable to pull registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error initializing source docker://registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference&#32;&#35;132&#46;72221102&#46;1596116349&#46;1e6e6df9\n</BODY></HTML>\n"

Note: This was detected while installing Red Hat Openshift Service Mesh, but the Elasticsearch Operator used for ClusterLogging is the same so issue and resolution may be the same. On Service Mesh Elasticsearch failure will cause the jaeger-collector and jaeger-query pods to stay in CrashLoopBackOff.

Environment

Openshift Container Platform 4.4.z
Red Hat Elasticsearch Operator 4.4.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In