Service relocation failed due to umount of DRBD resource
Issue
One of our cluster provide NFS shared storage, which relies on a DRBD shared storage. The relocation of the NFS service failed because the file system could not be umounted
May 15 13:01:51 STATION_A clurgmgrd[20729]: <notice> Stopping service service:nfssvc
...
May 15 13:01:52 STATION_A clurgmgrd: [20729]: <info> Removing export: 192.168.189.0/24:/DS
May 15 13:02:26 STATION_A clurgmgrd: [20729]: <info> unmounting /DS
May 15 13:02:34 STATION_A clurgmgrd: [20729]: <info> unmounting /DS
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> unmounting /DS
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <err> 'umount /DS' failed, error=0
May 15 13:02:39 STATION_A clurgmgrd[20729]: <notice> stop on fs "DS" returned 2 (invalid argument(s))
May 15 13:02:39 STATION_A kernel: block drbd1: State change failed: Device is held open by someone
May 15 13:02:39 STATION_A kernel: block drbd1: state = { cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate r----- }
May 15 13:02:39 STATION_A kernel: block drbd1: wanted = { cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate r----- }
May 15 13:02:39 STATION_A clurgmgrd[20729]: <notice> stop on drbd "drbd_DS" returned 1 (generic error)
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Removing export: 10.25.189.0/24:/spdata
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> unmounting /data
May 15 13:02:39 STATION_A kernel: block drbd2: role( Primary -> Secondary )
May 15 13:02:39 STATION_A kernel: block drbd2: bitmap WRITE of 0 pages took 0 jiffies
May 15 13:02:39 STATION_A kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Executing /etc/init.d/launch_x stop
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Executing /etc/init.d/launch_y stop
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Removing IPv4 address 192.168.191.118/28 from bond2
May 15 13:02:49 STATION_A clurgmgrd: [20729]: <info> Removing IPv4 address 192.168.189.203/25 from bond0
May 15 13:02:59 STATION_A clurgmgrd[20729]: <crit> #12: RG service:nfssvc failed to stop; intervention required
May 15 13:02:59 STATION_A clurgmgrd[20729]: <notice> Service service:nfssvc is failed
...
May 15 13:03:04 STATION_A clurgmgrd[20729]: <warning> #70: Failed to relocate service:nfssvc; restarting locally
May 15 13:03:04 STATION_A clurgmgrd[20729]: <err> #43: Service service:nfssvc has failed; can not start.
May 15 13:03:05 STATION_A clurgmgrd[20729]: <alert> #2: Service service:nfssvc returned failure code. Last Owner: STATION_A
May 15 13:03:05 STATION_A clurgmgrd[20729]: <alert> #4: Administrator intervention required.
Environment
- Red Hat Enterprise Linux 5
- DRBD
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.