Cannot start openshift after the volume where docker registry was mapped got full.

Posted on

Hi,

I am a newbie with Openshift and we are evaluating it in our company. Openshift 3 was working fine until the volume where the docker registry was mapped got full. we added another 20 GB but since then openshift cannot be started.
Here is the output of openshift start:
I0824 16:09:13.286956 8223 start_master.go:308] Starting OpenShift master on 0.0.0.0:8443 (v3.0.1.0-525-geddc479)
I0824 16:09:13.287029 8223 start_master.go:309] Public master address is https://10.4.0.55:8443
2015/08/24 16:09:13 etcdserver: datadir is valid for the 2.0.1 format
2015/08/24 16:09:13 etcdserver: name = openshift.local
2015/08/24 16:09:13 etcdserver: data dir = openshift.local.etcd
2015/08/24 16:09:13 etcdserver: member dir = openshift.local.etcd/member
2015/08/24 16:09:13 etcdserver: heartbeat = 100ms
2015/08/24 16:09:13 etcdserver: election = 1000ms
2015/08/24 16:09:13 etcdserver: snapshot count = 0
2015/08/24 16:09:13 etcdserver: advertise client URLs = https://10.4.0.55:4001
2015/08/24 16:09:13 etcdserver: restart member 602afaf54cfd4796 in cluster 42ea72f487fdf709 at commit index 17
2015/08/24 16:09:13 raft: 602afaf54cfd4796 became follower at term 2
2015/08/24 16:09:13 raft: newRaft 602afaf54cfd4796 [peers: [], term: 2, commit: 17, applied: 0, lastindex: 17, lastterm: 2]
2015/08/24 16:09:13 etcdserver: set snapshot count to default 10000
2015/08/24 16:09:13 etcdserver: added local member 602afaf54cfd4796 [https://10.4.0.55:7001] to cluster 42ea72f487fdf709
I0824 16:09:13.803952 8223 etcd.go:68] Started etcd at 10.4.0.55:4001
2015/08/24 16:09:15 raft: 602afaf54cfd4796 is starting a new election at term 2
2015/08/24 16:09:15 raft: 602afaf54cfd4796 became candidate at term 3
2015/08/24 16:09:15 raft: 602afaf54cfd4796 received vote from 602afaf54cfd4796 at term 3
2015/08/24 16:09:15 raft: 602afaf54cfd4796 became leader at term 3
2015/08/24 16:09:15 raft.node: 602afaf54cfd4796 elected leader 602afaf54cfd4796 at term 3
2015/08/24 16:09:15 etcdserver: published {Name:openshift.local ClientURLs:[https://10.4.0.55:4001]} to cluster 42ea72f487fdf709
F0824 16:09:22.399214 8223 start_allinone.go:91] OpenShift could not start: could not reach etcd: 501: All the given peers are not reachable (failed to propose on members [https://10.4.0.55:4001] twice [last error: Get https://10.4.0.55:4001/v2/keys/?quorum=false&recursive=false&sorted=false: Forbidden]) [0]

output of [root@bcngpp1vsprdopshift01 byo]# systemctl status openshift-master.service
openshift-master.service - OpenShift Master
Loaded: loaded (/usr/lib/systemd/system/openshift-master.service; enabled)
Active: failed (Result: exit-code) since lun 2015-08-24 16:08:55 UTC; 1min 49s ago
Docs: https://github.com/openshift/origin
Process: 8206 ExecStart=/usr/bin/openshift start master --config=${CONFIG_FILE} $OPTIONS (code=exited, status=255)
Main PID: 8206 (code=exited, status=255)

.. openshift-master[8206]: 2015/08/24 16:08:47 raft: 602afaf54cfd4796 is starting a new election at term 2
.. openshift-master[8206]: 2015/08/24 16:08:47 raft: 602afaf54cfd4796 became candidate at term 3
.. openshift-master[8206]: 2015/08/24 16:08:47 raft: 602afaf54cfd4796 received vote from 602afaf54cfd4796 at term 3
.. openshift-master[8206]: 2015/08/24 16:08:47 raft: 602afaf54cfd4796 became leader at term 3
.. openshift-master[8206]: 2015/08/24 16:08:47 raft.node: 602afaf54cfd4796 elected leader 602afaf54cfd4796 at term 3
.. openshift-master[8206]: 2015/08/24 16:08:47 etcdserver: published {Name:openshift.local ClientURLs:[https://10.4.0.55:400...87fdf709
.. openshift-master[8206]: F0824 16:08:55.004343 8206 start_master.go:100] could not reach etcd: 501: All the given peers are not...
.. systemd[1]: openshift-master.service: main process exited, code=exited, status=255/n/a
.. systemd[1]: Failed to start OpenShift Master.
.. systemd[1]: Unit openshift-master.service entered failed state.

journalctl -xn
-- Logs begin at lun 2015-08-24 13:10:41 UTC, end at lun 2015-08-24 16:12:56 UTC. --
.. openshift-node[2388]: E0824 16:12:56.114452 2388 common.go:243] Could not find an allocated subnet for minion bcngpp1vsprdopshift
.. openshift-node[2388]: E0824 16:12:56.556384 2388 kubelet.go:1641] error getting node: node bcngpp1vsprdopshift not fo
.. docker[1601]: time="2015-08-24T16:12:56.557219878Z" level=info msg="GET /containers/json"
.. docker[1601]: time="2015-08-24T16:12:56.659246044Z" level=info msg="GET /containers/json"
.. docker[1601]: time="2015-08-24T16:12:56.760870303Z" level=info msg="GET /containers/json"
.. docker[1601]: time="2015-08-24T16:12:56.862565764Z" level=info msg="GET /containers/json"
.. openshift-node[2388]: E0824 16:12:56.915778 2388 reflector.go:180] pkg/kubelet/kubelet.go:194: Failed to list *api.Node: Get http
.. openshift-node[2388]: E0824 16:12:56.915860 2388 reflector.go:180] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: G
.. openshift-node[2388]: E0824 16:12:56.915898 2388 reflector.go:180] pkg/kubelet/kubelet.go:177: Failed to list *api.Service: Get h
.. docker[1601]: time="2015-08-24T16:12:56.964131400Z" level=info msg="GET /containers/json"

Can you advise me what to do
Thank you

po

Responses