Openshift cluster not starting after the node restart.

Latest response

Hi,

We restarted Openshift cluster nodes both masters and workers. After restart we are not able to login in into cluster. Kubelet is working on all the nodes. when we are trying to login it is throwing error like 'Unable to connect to the server: EOF".

I0329 06:33:48.223880   21050 loader.go:375] Config loaded from file:  /root/.kube/config
I0329 06:33:48.224429   21050 round_trippers.go:423] curl -k -v -XHEAD  'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/'
I0329 06:33:48.252979   21050 round_trippers.go:443] HEAD https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/  in 28 milliseconds
I0329 06:33:48.253026   21050 round_trippers.go:449] Response Headers:
I0329 06:33:48.253101   21050 round_trippers.go:423] curl -k -v -XHEAD  'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/'
I0329 06:33:48.259049   21050 round_trippers.go:443] HEAD https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/ 403 Forbidden in 5 milliseconds
I0329 06:33:48.259066   21050 round_trippers.go:449] Response Headers:
I0329 06:33:48.259070   21050 round_trippers.go:452]     Content-Type: application/json
I0329 06:33:48.259074   21050 round_trippers.go:452]     X-Content-Type-Options: nosniff
I0329 06:33:48.259077   21050 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: 17003f8a-fa21-447b-a853-9863dbd328f7
I0329 06:33:48.259080   21050 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: e31b76b8-c325-4019-bc47-293e5f940792
I0329 06:33:48.259084   21050 round_trippers.go:452]     Content-Length: 186
I0329 06:33:48.259088   21050 round_trippers.go:452]     Date: Mon, 29 Mar 2021 10:33:48 GMT
I0329 06:33:48.259092   21050 round_trippers.go:452]     Audit-Id: f1594e82-af43-4e0a-9cd2-bf108f9f8e69
I0329 06:33:48.259095   21050 round_trippers.go:452]     Cache-Control: no-cache, private
I0329 06:33:48.259111   21050 request_token.go:86] GSSAPI Enabled
I0329 06:33:48.259144   21050 round_trippers.go:423] curl -k -v -XGET  -H "X-Csrf-Token: 1" 'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/.well-known/oauth-authorization-server'
I0329 06:33:48.259659   21050 round_trippers.go:443] GET https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/.well-known/oauth-authorization-server 200 OK in 0 milliseconds
I0329 06:33:48.259667   21050 round_trippers.go:449] Response Headers:
I0329 06:33:48.259670   21050 round_trippers.go:452]     Audit-Id: 60eeb747-9b25-47a8-b984-66c7dddef020
I0329 06:33:48.259673   21050 round_trippers.go:452]     Cache-Control: no-cache, private
I0329 06:33:48.259675   21050 round_trippers.go:452]     Content-Type: application/json
I0329 06:33:48.259678   21050 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: 17003f8a-fa21-447b-a853-9863dbd328f7
I0329 06:33:48.259680   21050 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: e31b76b8-c325-4019-bc47-293e5f940792
I0329 06:33:48.259683   21050 round_trippers.go:452]     Content-Length: 648
I0329 06:33:48.259685   21050 round_trippers.go:452]     Date: Mon, 29 Mar 2021 10:33:48 GMT
I0329 06:33:48.261108   21050 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/openshift (linux/amd64) kubernetes/b66f2d3" 'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/api/v1/namespaces/openshift/configmaps/motd'
I0329 06:33:48.261621   21050 round_trippers.go:443] GET https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/api/v1/namespaces/openshift/configmaps/motd 403 Forbidden in 0 milliseconds
I0329 06:33:48.261629   21050 round_trippers.go:449] Response Headers:
I0329 06:33:48.261632   21050 round_trippers.go:452]     Cache-Control: no-cache, private
I0329 06:33:48.261634   21050 round_trippers.go:452]     Content-Type: application/json
I0329 06:33:48.261637   21050 round_trippers.go:452]     X-Content-Type-Options: nosniff
I0329 06:33:48.261640   21050 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: 17003f8a-fa21-447b-a853-9863dbd328f7
I0329 06:33:48.261642   21050 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: e31b76b8-c325-4019-bc47-293e5f940792
I0329 06:33:48.261645   21050 round_trippers.go:452]     Content-Length: 303
I0329 06:33:48.261648   21050 round_trippers.go:452]     Date: Mon, 29 Mar 2021 10:33:48 GMT
I0329 06:33:48.261650   21050 round_trippers.go:452]     Audit-Id: bb1c2ecb-76aa-4703-9bad-520fe1d697c3
I0329 06:33:48.261671   21050 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
I0329 06:33:48.261974   21050 helpers.go:234] Connection error: Head https://oauth-openshift.apps.ocp4-cluster-001.int.dhdigital.co.in: EOF
F0329 06:33:48.261986   21050 helpers.go:115] Unable to connect to the server: EOF

Kindly provide some way to resolve this issue.

Attachments

FA

Responses