Openshift cluster not starting after the node restart.
Hi,
We restarted Openshift cluster nodes both masters and workers. After restart we are not able to login in into cluster. Kubelet is working on all the nodes. when we are trying to login it is throwing error like 'Unable to connect to the server: EOF".
I0329 06:33:48.223880 21050 loader.go:375] Config loaded from file: /root/.kube/config I0329 06:33:48.224429 21050 round_trippers.go:423] curl -k -v -XHEAD 'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/' I0329 06:33:48.252979 21050 round_trippers.go:443] HEAD https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/ in 28 milliseconds I0329 06:33:48.253026 21050 round_trippers.go:449] Response Headers: I0329 06:33:48.253101 21050 round_trippers.go:423] curl -k -v -XHEAD 'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/' I0329 06:33:48.259049 21050 round_trippers.go:443] HEAD https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/ 403 Forbidden in 5 milliseconds I0329 06:33:48.259066 21050 round_trippers.go:449] Response Headers: I0329 06:33:48.259070 21050 round_trippers.go:452] Content-Type: application/json I0329 06:33:48.259074 21050 round_trippers.go:452] X-Content-Type-Options: nosniff I0329 06:33:48.259077 21050 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 17003f8a-fa21-447b-a853-9863dbd328f7 I0329 06:33:48.259080 21050 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: e31b76b8-c325-4019-bc47-293e5f940792 I0329 06:33:48.259084 21050 round_trippers.go:452] Content-Length: 186 I0329 06:33:48.259088 21050 round_trippers.go:452] Date: Mon, 29 Mar 2021 10:33:48 GMT I0329 06:33:48.259092 21050 round_trippers.go:452] Audit-Id: f1594e82-af43-4e0a-9cd2-bf108f9f8e69 I0329 06:33:48.259095 21050 round_trippers.go:452] Cache-Control: no-cache, private I0329 06:33:48.259111 21050 request_token.go:86] GSSAPI Enabled I0329 06:33:48.259144 21050 round_trippers.go:423] curl -k -v -XGET -H "X-Csrf-Token: 1" 'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/.well-known/oauth-authorization-server' I0329 06:33:48.259659 21050 round_trippers.go:443] GET https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/.well-known/oauth-authorization-server 200 OK in 0 milliseconds I0329 06:33:48.259667 21050 round_trippers.go:449] Response Headers: I0329 06:33:48.259670 21050 round_trippers.go:452] Audit-Id: 60eeb747-9b25-47a8-b984-66c7dddef020 I0329 06:33:48.259673 21050 round_trippers.go:452] Cache-Control: no-cache, private I0329 06:33:48.259675 21050 round_trippers.go:452] Content-Type: application/json I0329 06:33:48.259678 21050 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 17003f8a-fa21-447b-a853-9863dbd328f7 I0329 06:33:48.259680 21050 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: e31b76b8-c325-4019-bc47-293e5f940792 I0329 06:33:48.259683 21050 round_trippers.go:452] Content-Length: 648 I0329 06:33:48.259685 21050 round_trippers.go:452] Date: Mon, 29 Mar 2021 10:33:48 GMT I0329 06:33:48.261108 21050 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/openshift (linux/amd64) kubernetes/b66f2d3" 'https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/api/v1/namespaces/openshift/configmaps/motd' I0329 06:33:48.261621 21050 round_trippers.go:443] GET https://api.ocp4-cluster-001.int.dhdigital.co.in:6443/api/v1/namespaces/openshift/configmaps/motd 403 Forbidden in 0 milliseconds I0329 06:33:48.261629 21050 round_trippers.go:449] Response Headers: I0329 06:33:48.261632 21050 round_trippers.go:452] Cache-Control: no-cache, private I0329 06:33:48.261634 21050 round_trippers.go:452] Content-Type: application/json I0329 06:33:48.261637 21050 round_trippers.go:452] X-Content-Type-Options: nosniff I0329 06:33:48.261640 21050 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 17003f8a-fa21-447b-a853-9863dbd328f7 I0329 06:33:48.261642 21050 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: e31b76b8-c325-4019-bc47-293e5f940792 I0329 06:33:48.261645 21050 round_trippers.go:452] Content-Length: 303 I0329 06:33:48.261648 21050 round_trippers.go:452] Date: Mon, 29 Mar 2021 10:33:48 GMT I0329 06:33:48.261650 21050 round_trippers.go:452] Audit-Id: bb1c2ecb-76aa-4703-9bad-520fe1d697c3 I0329 06:33:48.261671 21050 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403} I0329 06:33:48.261974 21050 helpers.go:234] Connection error: Head https://oauth-openshift.apps.ocp4-cluster-001.int.dhdigital.co.in: EOF F0329 06:33:48.261986 21050 helpers.go:115] Unable to connect to the server: EOF
Kindly provide some way to resolve this issue.
Responses