OpenShift 4.x etcd container fails to start due to missing snapshot file "error: snap: snapshot file doesn't exist"

Solution Verified - Updated -

Issue

  • Etcd container pod logs indicating they are unable to start due to missing snapshot file as per below example:
2023-01-04T01:42:51.006195153Z {"level":"warn","ts":"2023-01-04T01:42:51.006Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000eec40/xxx.xxx.xxx.254:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp xxx.xxx.xxx.65:2379: connect: connection refused\""}
2023-01-04T01:42:51.006195153Z Error: context deadline exceeded
2023-01-04T01:42:53.038570167Z {"level":"warn","ts":1672796573.038539,"caller":"fileutil/fileutil.go:57","msg":"check file permission","error":"directory \"/var/lib/etcd\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
2023-01-04T01:42:55.213103191Z {"level":"warn","ts":"2023-01-04T01:42:55.213Z","caller":"snap/db.go:88","msg":"failed to find [SNAPSHOT-INDEX].snap.db","snapshot-index":1340915912,"snapshot-file-path":"/var/lib/etcd/member/snap/000000004fecc0c8.snap.db","error":"snap: snapshot file doesn't exist"}
2023-01-04T01:42:55.213135518Z {"level":"panic","ts":"2023-01-04T01:42:55.213Z","caller":"etcdserver/server.go:518","msg":"failed to recover v3 backend from snapshot","error":"failed to find database snapshot file (snap: snapshot file doesn't exist)","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.NewServer\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdserver/server.go:518\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/remote-source/cachito-gomod-with-deps/app/server/embed/etcd.go:246\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/etcd.go:228\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/etcd.go:123\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/main.go:40\nmain.main\n\t/remote-source/cachito-gomod-with-deps/app/server/main.go:32\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:225"}
2023-01-04T01:42:55.215293807Z panic: failed to recover v3 backend from snapshot
2023-01-04T01:42:55.215308232Z go.uber.org/zap.(*Logger).Panic(0xc00012f860, 0x123828a2023-01-04T01:42:55.215314987Z , 0x2a, 0xc0004482c0, 0x1, 0x1)

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content