MongoDBs with persistant storage not starting after unexpected shutdown
Issue
-
Since our external NFS disk storage ran out of space (has been solved) the mongodb keep crashing during startup.
-
Pods showing
CrashLoopBackOffstatus from repeatedly restarting
Example output from a starting container:
2016-05-11T11:39:50.418-0400 [initandlisten] build info: Linux x86-036.build.eng.bos.redhat.com 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 BOOST_LIB_VERSION=1_53
2016-05-11T11:39:50.418-0400 [initandlisten] allocator: tcmalloc
2016-05-11T11:39:50.418-0400 [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { enabled: false }, port: 27017 }, processManagement: { pidFilePath: "/var/lib/mongodb/mongodb.pid" }, replication: { oplogSizeMB: 64 }, storage: { dbPath: "/var/lib/mongodb/data", preallocDataFiles: false, smallFiles: true }, systemLog: { quiet: true } }
2016-05-11T11:39:50.422-0400 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /var/lib/mongodb/data/mongod.lock. Is a mongod instance already running?, terminating
2016-05-11T11:39:50.422-0400 [initandlisten] dbexit:
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: going to close listening sockets...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: going to flush diaglog...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: going to close sockets...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: waiting for fs preallocator...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: lock for final commit...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: final commit...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: closing all files...
2016-05-11T11:39:50.422-0400 [initandlisten] closeAllFiles() finished
2016-05-11T11:39:50.422-0400 [initandlisten] dbexit: really exiting now
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
.....
(Where /var/lib/mongodb/data points to an external NFS-Storage)
- MongoDB fails to start after exceeding storage limit
Environment
- Red Hat OpenShift Enterprise 3.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
