OpenShift 4.x etcd container fails to start due to missing snapshot file "error: snap: snapshot file doesn't exist"
Issue
- Etcd container pod logs indicating they are unable to start due to missing snapshot file as per below example:
2023-01-04T01:42:51.006195153Z {"level":"warn","ts":"2023-01-04T01:42:51.006Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000eec40/xxx.xxx.xxx.254:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp xxx.xxx.xxx.65:2379: connect: connection refused\""}
2023-01-04T01:42:51.006195153Z Error: context deadline exceeded
2023-01-04T01:42:53.038570167Z {"level":"warn","ts":1672796573.038539,"caller":"fileutil/fileutil.go:57","msg":"check file permission","error":"directory \"/var/lib/etcd\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
2023-01-04T01:42:55.213103191Z {"level":"warn","ts":"2023-01-04T01:42:55.213Z","caller":"snap/db.go:88","msg":"failed to find [SNAPSHOT-INDEX].snap.db","snapshot-index":1340915912,"snapshot-file-path":"/var/lib/etcd/member/snap/000000004fecc0c8.snap.db","error":"snap: snapshot file doesn't exist"}
2023-01-04T01:42:55.213135518Z {"level":"panic","ts":"2023-01-04T01:42:55.213Z","caller":"etcdserver/server.go:518","msg":"failed to recover v3 backend from snapshot","error":"failed to find database snapshot file (snap: snapshot file doesn't exist)","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.NewServer\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdserver/server.go:518\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/remote-source/cachito-gomod-with-deps/app/server/embed/etcd.go:246\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/etcd.go:228\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/etcd.go:123\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/main.go:40\nmain.main\n\t/remote-source/cachito-gomod-with-deps/app/server/main.go:32\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:225"}
2023-01-04T01:42:55.215293807Z panic: failed to recover v3 backend from snapshot
2023-01-04T01:42:55.215308232Z go.uber.org/zap.(*Logger).Panic(0xc00012f860, 0x123828a2023-01-04T01:42:55.215314987Z , 0x2a, 0xc0004482c0, 0x1, 0x1)
Environment
- Red Hat OpenShift Container Platform (OCP) 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.