Missing Deployment and Failed Pods for ElasticSearch After RHOCP 4 Cluster Upgrade

Solution Verified - Updated -

Issue

  • ElasticSearch Deployment missing after RHOCP Cluster upgrade
  • ElasticSearch Pod fails to start after upgrade
$ oc get pods

elasticsearch-cdm-xxxxxxxx-1-xxxxxxxxx-xxxxxx   0/2    Pending  0         4s
elasticsearch-cdm-xxxxxxxx-2-xxxxxxxxx-xxxxxx   1/2    Running  0         21h

$ oc get deployment

NAME                          READY  UP-TO-DATE  AVAILABLE  AGE
cluster-logging-operator      1/1    1           1          21h
elasticsearch-cdm-xxxxxxxx-1  ??     1           ??         1d
elasticsearch-cdm-xxxxxxxx-2  ??     1           ??         1d
kibana                        1/1    1           1          1d

Logs of ElasctiSearch

2022-07-19T17:05:17.680297853Z [2022-07-19T17:05:17,680][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-xxxxxxxx-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-xxxxxxxx-2}{XXXXXXXXXXXXXXXXXXX}{XXXXXXXXXXXXXXXXXXX}{10.0.0.0}{10.0.0.0:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
2022-07-19T17:05:20.682835005Z [2022-07-19T17:05:20,682][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-xxxxxxxx-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-xxxxxxxx-2}{XXXXXXXXXXXXXXXXXXX}{XXXXXXXXXXXXXXXXXXX}{10.0.0.0}{10.0.0.0:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
2022-07-19T17:05:22.178699330Z [2022-07-19T17:05:22,178][ERROR][c.a.o.s.a.BackendRegistry] [elasticsearch-cdm-xxxxxxxx-2] Not yet initialized 
2022-07-19T17:05:23.683644884Z [2022-07-19T17:05:23,683][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-xxxxxxxx-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-xxxxxxxx-2}{XXXXXXXXXXXXXXXXXXX}{XXXXXXXXXXXXXXXXXXX}{10.0.0.0}{10.0.0.0:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
2022-07-19T17:05:26.684508897Z [2022-07-19T17:05:26,684][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-xxxxxxxx-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-xxxxxxxx-2}{XXXXXXXXXXXXXXXXXXX}{XXXXXXXXXXXXXXXXXXX}{10.0.0.0}{10.0.0.0:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL) 5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content