[root@ocp4-bastion ~]# ./openshift-install create cluster --dir ./ocp4 --log-level=info
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s (until 12:53AM +04) for the Kubernetes API at https://api.ocp4.kodevirtual.com:6443...
INFO API v1.28.7+c1f5b34 up
INFO Waiting up to 1h0m0s (until 1:33AM +04) for bootstrapping to complete...
E0502 00:55:13.855095    2458 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184&timeoutSeconds=419&watch=true": http2: client connection lost - error from a previous attempt: unexpected EOF
W0502 00:55:24.944263    2458 reflector.go:535] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
I0502 00:55:24.944373    2458 trace.go:236] Trace[735717463]: "Reflector ListAndWatch" name:k8s.io/client-go/tools/watch/informerwatcher.go:146 (02-May-2024 00:55:14.779) (total time: 10165ms):
Trace[735717463]: ---"Objects listed" error:Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host 10165ms (00:55:24.944)
Trace[735717463]: [10.165196107s] [10.165196107s] END
E0502 00:55:24.944403    2458 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
W0502 00:55:28.336221    2458 reflector.go:535] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
E0502 00:55:28.336283    2458 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
W0502 00:55:36.912414    2458 reflector.go:535] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
E0502 00:55:36.912471    2458 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
W0502 00:55:50.800288    2458 reflector.go:535] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
E0502 00:55:50.800336    2458 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
W0502 00:56:13.201371    2458 reflector.go:535] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
E0502 00:56:13.201706    2458 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api.ocp4.kodevirtual.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=14184": dial tcp 172.21.240.254:6443: connect: no route to host
WARNING Failed to extract host addresses: could not extract IP with bootstrap MOID:
INFO Skipping VM console logs gather: no gather methods registered for "vsphere"
INFO Pulling debug logs from the bootstrap machine
WARNING Unable to stat /root/ocp4/serial-log-bundle-20240502013327.tar.gz, skipping
ERROR Cluster operator authentication Degraded is True with APIServerDeployment_UnavailablePod::IngressStateEndpoints_MissingSubsets::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError: APIServerDeploymentDegraded: 1 of 2 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()
ERROR IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
ERROR OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.40.251:443/healthz": dial tcp 172.30.40.251:443: connect: connection refused
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
ERROR Cluster operator authentication Available is False with APIServices_Error::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::ReadyIngressNodes_NoReadyIngressNodes: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.oauth.openshift.io: not available: failing or missing response from https://10.128.0.69:8443/apis/oauth.openshift.io/v1: Get "https://10.128.0.69:8443/apis/oauth.openshift.io/v1": dial tcp 10.128.0.69:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.user.openshift.io: not available: failing or missing response from https://10.128.0.69:8443/apis/user.openshift.io/v1: Get "https://10.128.0.69:8443/apis/user.openshift.io/v1": dial tcp 10.128.0.69:8443: connect: no route to host
ERROR OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.40.251:443/healthz": dial tcp 172.30.40.251:443: connect: connection refused
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found
ERROR ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 2 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).
INFO Cluster operator baremetal Disabled is False with :
INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected
INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected
INFO Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected
INFO Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected
INFO Cluster operator cloud-controller-manager CloudControllerOwner is True with AsExpected: Cluster Cloud Controller Manager Operator owns cloud controllers at 4.15.9
ERROR Cluster operator cluster-autoscaler Degraded is True with MissingDependency: machine-api not ready
ERROR Cluster operator etcd Degraded is True with InstallerPodContainerWaiting_ContainerCreating::NodeController_MasterNodesReady: InstallerPodContainerWaitingDegraded: Pod "installer-4-ocp4-wmng6-master-1" on node "ocp4-wmng6-master-1" container "installer" is waiting since 2024-05-01 21:07:23 +0000 UTC because ContainerCreating
ERROR NodeControllerDegraded: The master nodes not ready: node "ocp4-wmng6-master-1" not ready since 2024-05-01 21:07:00 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
INFO Cluster operator etcd Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 3; 0 nodes have achieved new revision 4
ERROR Cluster operator ingress Available is False with IngressUnavailable: The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)
INFO Cluster operator ingress Progressing is True with Reconciling: ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available...
INFO ).
INFO Not all ingress controllers are available.
ERROR Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod "router-default-7979d79c66-qk5mm" cannot be scheduled: 0/2 nodes are available: 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.. Pod "router-default-7979d79c66-dpt2r" cannot be scheduled: 0/2 nodes are available: 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.. Make sure you have sufficient worker nodes.)
INFO Cluster operator ingress EvaluationConditionsDetected is False with AsExpected:
INFO Cluster operator insights ClusterTransferAvailable is False with NoClusterTransfer: no available cluster transfer
INFO Cluster operator insights Disabled is False with AsExpected:
INFO Cluster operator insights SCAAvailable is False with NotFound: Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 404: {"code":"ACCT-MGMT-7","href":"/api/accounts_mgmt/v1/errors/7","id":"7","kind":"Error","operation_id":"90cf4075-c7ba-4621-b1b7-2322d738d7b8","reason":"The organization (id= 2GZgKNDYoUMj8bZk4KhpekomDpA) does not have any certificate of type sca. Enable SCA at https://access.redhat.com/management."}
ERROR Cluster operator kube-apiserver Degraded is True with GuardController_SyncError::NodeController_MasterNodesReady: GuardControllerDegraded: Missing operand on node ocp4-wmng6-master-2
ERROR NodeControllerDegraded: The master nodes not ready: node "ocp4-wmng6-master-1" not ready since 2024-05-01 21:07:00 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
INFO Cluster operator kube-apiserver Progressing is True with NodeInstaller: NodeInstallerProgressing: 2 nodes are at revision 0; 0 nodes have achieved new revision 10
ERROR Cluster operator kube-apiserver Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 2 nodes are at revision 0; 0 nodes have achieved new revision 10
INFO Cluster operator kube-apiserver EvaluationConditionsDetected is False with AsExpected: All is well
ERROR Cluster operator kube-controller-manager Degraded is True with NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ocp4-wmng6-master-1" not ready since 2024-05-01 21:07:00 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
INFO Cluster operator kube-controller-manager Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 7; 0 nodes have achieved new revision 11
ERROR Cluster operator kube-scheduler Degraded is True with NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ocp4-wmng6-master-1" not ready since 2024-05-01 21:07:00 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
INFO Cluster operator kube-scheduler Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 6; 0 nodes have achieved new revision 10
INFO Cluster operator machine-api Progressing is True with SyncingResources: Progressing towards operator: 4.15.9
ERROR Cluster operator machine-api Degraded is True with SyncingFailed: Failed when progressing towards operator: 4.15.9 because minimum worker replica count (2) not yet met: current running replicas 0, waiting for [ocp4-wmng6-worker-0-4cght ocp4-wmng6-worker-0-bwg9d ocp4-wmng6-worker-0-rqzmt]
ERROR Cluster operator machine-api Available is False with Initializing: Operator is initializing
ERROR Cluster operator machine-config Degraded is True with RequiredPoolsFailed: Failed to resync 4.15.9 because: error during syncRequiredMachineConfigPools: [context deadline exceeded, failed to update clusteroperator: [client rate limiter Wait returned an error: context deadline exceeded, error required MachineConfigPool master is not ready, retrying. Status: (total: 2, ready 1, updated: 2, unavailable: 1, degraded: 0)]]
ERROR Cluster operator monitoring Available is False with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded
ERROR Cluster operator monitoring Degraded is True with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded
INFO Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.
ERROR Cluster operator network Degraded is True with RolloutHung: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - pod ovnkube-node-snmgs is in CrashLoopBackOff State
ERROR DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - last change 2024-05-01T21:07:01Z
ERROR DaemonSet "/openshift-multus/multus" rollout is not making progress - pod multus-gjqwd is in CrashLoopBackOff State
INFO Cluster operator network ManagementStateDegraded is False with :
INFO Cluster operator network Progressing is True with Deploying: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 1 nodes)
INFO DaemonSet "/openshift-multus/multus" is not available (awaiting 1 nodes)
INFO DaemonSet "/openshift-multus/network-metrics-daemon" is not available (awaiting 1 nodes)
INFO DaemonSet "/openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes)
INFO Deployment "/openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready
ERROR Cluster operator network Available is False with Startup: The network is starting up
INFO Cluster operator node-tuning Progressing is True with ProfileProgressing: Waiting for 1/2 Profiles to be applied
ERROR Cluster operator openshift-apiserver Available is False with APIServices_Error: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.79:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/authorization.openshift.io/v1: Get "https://10.128.0.79:8443/apis/authorization.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.build.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/build.openshift.io/v1: Get "https://10.128.0.79:8443/apis/build.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/image.openshift.io/v1: Get "https://10.128.0.79:8443/apis/image.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/project.openshift.io/v1: Get "https://10.128.0.79:8443/apis/project.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/quota.openshift.io/v1: Get "https://10.128.0.79:8443/apis/quota.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.route.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/route.openshift.io/v1: Get "https://10.128.0.79:8443/apis/route.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.security.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/security.openshift.io/v1: Get "https://10.128.0.79:8443/apis/security.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.128.0.79:8443/apis/template.openshift.io/v1: Get "https://10.128.0.79:8443/apis/template.openshift.io/v1": dial tcp 10.128.0.79:8443: connect: no route to host
ERROR Cluster operator operator-lifecycle-manager-packageserver Available is False with ClusterServiceVersionNotSucceeded: ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install failed: deployment packageserver not ready before timeout: deployment "packageserver" exceeded its progress deadline
INFO Cluster operator storage Progressing is True with VSphereCSIDriverOperatorCR_VMwareVSphereDriverNodeServiceController_Deploying: VSphereCSIDriverOperatorCRProgressing: VMwareVSphereDriverNodeServiceControllerProgressing: Waiting for DaemonSet to deploy node pods
ERROR Bootstrap failed to complete: timed out waiting for the condition
ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.
WARNING The bootstrap machine is unable to resolve API and/or API-Int Server URLs
INFO pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
INFO pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
INFO pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
INFO Bootstrap gather logs captured here "/root/ocp4/log-bundle-20240502013327.tar.gz"
[root@ocp4-bastion ~]# ping 172.21.240.254
PING 172.21.240.254 (172.21.240.254) 56(84) bytes of data.
64 bytes from 172.21.240.254: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 172.21.240.254: icmp_seq=2 ttl=64 time=0.186 ms
64 bytes from 172.21.240.254: icmp_seq=3 ttl=64 time=0.185 ms
64 bytes from 172.21.240.254: icmp_seq=4 ttl=64 time=0.162 ms
64 bytes from 172.21.240.254: icmp_seq=5 ttl=64 time=0.183 ms
64 bytes from 172.21.240.254: icmp_seq=6 ttl=64 time=0.098 ms
^C
--- 172.21.240.254 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5145ms
rtt min/avg/max/mdev = 0.098/0.155/0.186/0.034 ms
[root@ocp4-bastion ~]# ping 172.21.240.253
PING 172.21.240.253 (172.21.240.253) 56(84) bytes of data.
64 bytes from 172.21.240.253: icmp_seq=1 ttl=64 time=2.60 ms
64 bytes from 172.21.240.253: icmp_seq=2 ttl=64 time=0.373 ms
64 bytes from 172.21.240.253: icmp_seq=3 ttl=64 time=0.464 ms
^C
--- 172.21.240.253 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2036ms
rtt min/avg/max/mdev = 0.373/1.145/2.599/1.028 ms