crc 1.29.1 Error waiting for apiserver

Latest response

crc version
CodeReady Containers version: 1.29.1+bc5f4409
OpenShift version: 4.7.18 (embedded in executable)

Operating System: Fedora 34 (Thirty Four)
CPE OS Name: cpe:/o:fedoraproject:fedora:34
Kernel: Linux 5.12.14-300.fc34.x86_64
Architecture: x86-64

Error waiting for apiserver: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err : Process exited with status 1\n (x25)

--kubeconfig /opt/kubeconfig\nerr : Process exited with status 1\n - sleeping 1s"
time="2021-07-20T13:01:00+02:00" level=debug msg="retry loop: attempt 80"
time="2021-07-20T13:01:00+02:00" level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
time="2021-07-20T13:01:00+02:00" level=debug msg="SSH command results: err: Process exited with status 1, output: "
time="2021-07-20T13:01:00+02:00" level=debug msg="The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?\n"
time="2021-07-20T13:01:00+02:00" level=debug msg="error: Temporary error: ssh command error:\ncommand : timeout 5s oc get nodes --context admin --cluster crc

Responses

Hello Jose Antonio Rodrigo,

Please also see this discussion https://access.redhat.com/discussions/6144752.

Kind Regards,
RJ

Hi RJ Hinton, first of all, thank you for your answer.

I have been try some of the responses, like

crc stop crc delete sudo chmod 666 /etc/hosts crc setup crc start

or crc start -n 8.8.8.8 but I still get the same problem.

Benet Uwalaka test with 2 scenarios vmware and virtualbox. My scenario is with fedora 34 and kvm-libvirt.x86_64 7.0.0-6.fc34

I can see tha CRC doesn´t create .kube folder with config file , I don´t know why. Either see kubeadmin user $ oc get user No resources found $ oc get identity No resources found

oc whoami system:admin

crc console --credentials To login as a regular user, run 'oc login -u developer -p developer https://api.crc.testing:6443'. To login as an admin, run 'oc login -u kubeadmin -p qEB7s-5sfLB-HHrBK-w4gYf https://api.crc.testing:6443' [wmaster@mifedus ~]$ oc login -u kuebadmin -p qEB7s-5sfLB-HHrBK-w4gYf Login failed (401 Unauthorized) Verify you have provided correct credentials.

I'm still investigation

kind regards, j.

Hi. Follow some of the adviced of discussions 6144752. Without success up to now for Fedora.

In my case, I test the same crc version (crc version CodeReady Containers version: 1.29.1+bc5f4409 OpenShift version: 4.7.18 (embedded in executable)) and the same hardware Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz 16GB RAM

With Centos 8 it works with Fedora 34 don´t works Operating System: Fedora 34 (Thirty Four)
CPE OS Name: cpe:/o:fedoraproject:fedora:34 Kernel: Linux 5.12.14-300.fc34.x86_64

Regards, j.

Hi.

ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@192.168.130.11

export KUBECONFIG=/opt/kubeconfig

[root@crc-4727w-master-0 core]# oc get pods -n openshift-kube-apiserver NAME READY STATUS RESTARTS AGE installer-7-crc-4727w-master-0 0/1 Completed 0 27m installer-8-crc-4727w-master-0 0/1 Completed 0 24m kube-apiserver-crc-4727w-master-0 5/5 Running 0 21m revision-pruner-6-crc-4727w-master-0 0/1 Completed 0 21d revision-pruner-8-crc-4727w-master-0 0/1 Completed 0 20m

I can see some errors in installer-7-crc-4727w-master-0 [root@crc-4727w-master-0 core]# oc logs -f installer-7-crc-4727w-master-0 -n openshift-kube-apiserver . . . I0723 16:41:01.768657 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert: secrets "user-serving-cert" not found I0723 16:41:02.165040 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-000: secrets "user-serving-cert-000" not found I0723 16:41:03.767240 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-001: secrets "user-serving-cert-001" not found I0723 16:41:08.268190 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-002: secrets "user-serving-cert-002" not found I0723 16:41:08.827474 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-003: secrets "user-serving-cert-003" not found I0723 16:41:09.337001 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-004: secrets "user-serving-cert-004" not found I0723 16:41:09.874359 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-005: secrets "user-serving-cert-005" not found I0723 16:41:11.532471 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-006: secrets "user-serving-cert-006" not found I0723 16:41:12.505492 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-007: secrets "user-serving-cert-007" not found I0723 16:41:13.040725 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-008: secrets "user-serving-cert-008" not found I0723 16:41:13.573887 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-009: secrets "user-serving-cert-009" not found . . {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kube-apiserver","namespace":"openshift-kube-apiserver","uid":"ccb7f42a-fa03-4b35-a890-89b1728167f8","creationTimestamp":null,"labels":{"apiserver":"true","app":"openshift-kube-apiserver","revision":"7"},"annotations":{"kubectl.kubernetes.io/default-logs-container":"kube-apiserver"}},"spec":{"volumes":[{"name":"resource-dir","hostPath":{"path":"/etc/kubernetes/static-pod-resources/kube-apiserver-pod-7"}},{"name":"cert-dir","hostPath":{"path":"/etc/kubernetes/static-pod-resources/kube-apiserver-certs"}},{"name":"audit-dir","hostPath":{"path":"/var/log/kube-apiserver"}}],"initContainers":[{"name":"setup","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0286a17f0671310c1610508127f4730aab5c30048f5fd5af88c0e439eaf5ac30","command":["/usr/bin/timeout","105","/bin/bash","-ec"],"args":["echo -n \"Fixing audit permissions.\"\nchmod 0700 /var/log/kube-apiserver \u0026\u0026 touch /var/log/kube-apiserver/audit.log \u0026\u0026 chmod 0600 /var/log/kube-apiserver/*\necho -n \"Waiting for port :6443 and :6080 to be released.\"\nwhile [ -n \"$(ss -Htan '( sport = 6443 or sport = 6080 )')\" ]; do\n echo -n \".\"\n sleep 1\ndone\n"],"resources":{},"volumeMounts":[{"name":"audit-dir","mountPath":"/var/log/kube-apiserver"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}],"containers":[{"name":"kube-apiserver","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0286a17f0671310c1610508127f4730aab5c30048f5fd5af88c0e439eaf5ac30","command":["/bin/bash","-ec"],"args":["if [ -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt ]; then\n echo \"Copying system trust bundle\"\n cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem\nfi\necho -n \"Waiting for port :6443 to be released.\"\ntries=0\nwhile [ -n \"$(ss -Htan '( sport = 6443 )')\" ]; do\n echo -n \".\"\n sleep 1\n (( tries += 1 ))\n if [[ \"${tries}\" -gt 105 ]]; then\n echo \"timed out waiting for port :6443 to be released\"\n exit 1\n fi\ndone\necho\nexec watch-termination --termination-touch-file=/var/log/kube-apiserver/.terminating --termination-log-file=/var/log/kube-apiserver/termination.log --graceful-termination-duration=135s --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig -- hyperkube kube-apiserver --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --advertise-address=${HOST_IP} -v=2\n"],"ports":[{"containerPort":6443}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"STATIC_POD_VERSION","value":"7"},{"name":"HOST_IP","valueFrom":{"fieldRef":{"fieldPath":"status.hostIP"}}}],"resources":{"requests":{"cpu":"265m","memory":"1Gi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"},{"name":"audit-dir","mountPath":"/var/log/kube-apiserver"}],"livenessProbe":{"httpGet":{"path":"healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":45,"timeoutSeconds":10},"readinessProbe":{"httpGet":{"path":"healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":10,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}},{"name":"kube-apiserver-cert-syncer","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64c9db97be9ef2b76b08812246f403afe5283c62ca693bc2c269e2accf76b36a","command":["cluster-kube-apiserver-operator","cert-syncer"],"args":["--kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig","--namespace=$(POD_NAMESPACE)","--destination-dir=/etc/kubernetes/static-pod-certs"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"5m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-cert-regeneration-controller","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64c9db97be9ef2b76b08812246f403afe5283c62ca693bc2c269e2accf76b36a","command":["cluster-kube-apiserver-operator","cert-regeneration-controller"],"args":["--kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig","--namespace=$(POD_NAMESPACE)","-v=2"],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"5m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-insecure-readyz","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64c9db97be9ef2b76b08812246f403afe5283c62ca693bc2c269e2accf76b36a","command":["cluster-kube-apiserver-operator","insecure-readyz"],"args":["--insecure-port=6080","--delegate-url=https://localhost:6443/readyz"],"ports":[{"containerPort":6080}],"resources":{"requests":{"cpu":"5m","memory":"50Mi"}},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-check-endpoints","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64c9db97be9ef2b76b08812246f403afe5283c62ca693bc2c269e2accf76b36a","command":["cluster-kube-apiserver-operator","check-endpoints"],"args":["--kubeconfig","/etc/kubernetes/static-pod-certs/configmaps/check-endpoints-kubeconfig/kubeconfig","--listen","0.0.0.0:17697","--namespace","$(POD_NAMESPACE)","--v","2"],"ports":[{"name":"check-endpoints","hostPort":17697,"containerPort":17697,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"}],"livenessProbe":{"httpGet":{"path":"healthz","port":17697,"scheme":"HTTPS"},"initialDelaySeconds":10,"timeoutSeconds":10},"readinessProbe":{"httpGet":{"path":"healthz","port":17697,"scheme":"HTTPS"},"initialDelaySeconds":10,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"}],"terminationGracePeriodSeconds":135,"hostNetwork":true,"tolerations":[{"operator":"Exists"}],"priorityClassName":"system-node-critical"},"status":{}}

Regards, j.