MongoDBs with persistant storage not starting after unexpected shutdown

Issue

• Since our external NFS disk storage ran out of space (has been solved) the mongodb keep crashing during startup.

• Pods showing CrashLoopBackOff status from repeatedly restarting

Example output from a starting container:

2016-05-11T11:39:50.418-0400 [initandlisten] build info: Linux x86-036.build.eng.bos.redhat.com 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 BOOST_LIB_VERSION=1_53
2016-05-11T11:39:50.418-0400 [initandlisten] allocator: tcmalloc
2016-05-11T11:39:50.418-0400 [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { enabled: false }, port: 27017 }, processManagement: { pidFilePath: "/var/lib/mongodb/mongodb.pid" }, replication: { oplogSizeMB: 64 }, storage: { dbPath: "/var/lib/mongodb/data", preallocDataFiles: false, smallFiles: true }, systemLog: { quiet: true } }
2016-05-11T11:39:50.422-0400 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /var/lib/mongodb/data/mongod.lock. Is a mongod instance already running?, terminating
2016-05-11T11:39:50.422-0400 [initandlisten] dbexit:
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: going to close listening sockets...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: going to flush diaglog...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: going to close sockets...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: waiting for fs preallocator...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: lock for final commit...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: final commit...
2016-05-11T11:39:50.422-0400 [initandlisten] shutdown: closing all files...
2016-05-11T11:39:50.422-0400 [initandlisten] closeAllFiles() finished
2016-05-11T11:39:50.422-0400 [initandlisten] dbexit: really exiting now
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
.....

(Where /var/lib/mongodb/data points to an external NFS-Storage)

• MongoDB fails to start after exceeding storage limit