Service relocation failed due to umount of DRBD resource

Solution Unverified - Updated -

Issue

One of our cluster provide NFS shared storage, which relies on a DRBD shared storage. The relocation of the NFS service failed because the file system could not be umounted

May 15 13:01:51 STATION_A clurgmgrd[20729]: <notice> Stopping service service:nfssvc
...
May 15 13:01:52 STATION_A clurgmgrd: [20729]: <info> Removing export: 192.168.189.0/24:/DS
May 15 13:02:26 STATION_A clurgmgrd: [20729]: <info> unmounting /DS
May 15 13:02:34 STATION_A clurgmgrd: [20729]: <info> unmounting /DS
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> unmounting /DS
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <err> 'umount /DS' failed, error=0
May 15 13:02:39 STATION_A clurgmgrd[20729]: <notice> stop on fs "DS" returned 2 (invalid argument(s))
May 15 13:02:39 STATION_A kernel: block drbd1: State change failed: Device is held open by someone
May 15 13:02:39 STATION_A kernel: block drbd1:   state = { cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate r----- }
May 15 13:02:39 STATION_A kernel: block drbd1:  wanted = { cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate r----- }
May 15 13:02:39 STATION_A clurgmgrd[20729]: <notice> stop on drbd "drbd_DS" returned 1 (generic error)
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Removing export: 10.25.189.0/24:/spdata
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> unmounting /data
May 15 13:02:39 STATION_A kernel: block drbd2: role( Primary -> Secondary )
May 15 13:02:39 STATION_A kernel: block drbd2: bitmap WRITE of 0 pages took 0 jiffies
May 15 13:02:39 STATION_A kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Executing /etc/init.d/launch_x stop
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Executing /etc/init.d/launch_y stop
May 15 13:02:39 STATION_A clurgmgrd: [20729]: <info> Removing IPv4 address 192.168.191.118/28 from bond2
May 15 13:02:49 STATION_A clurgmgrd: [20729]: <info> Removing IPv4 address 192.168.189.203/25 from bond0
May 15 13:02:59 STATION_A clurgmgrd[20729]: <crit> #12: RG service:nfssvc failed to stop; intervention required
May 15 13:02:59 STATION_A clurgmgrd[20729]: <notice> Service service:nfssvc is failed
...
May 15 13:03:04 STATION_A clurgmgrd[20729]: <warning> #70: Failed to relocate service:nfssvc; restarting locally
May 15 13:03:04 STATION_A clurgmgrd[20729]: <err> #43: Service service:nfssvc has failed; can not start.
May 15 13:03:05 STATION_A clurgmgrd[20729]: <alert> #2: Service service:nfssvc returned failure code.  Last Owner: STATION_A
May 15 13:03:05 STATION_A clurgmgrd[20729]: <alert> #4: Administrator intervention required.

Environment

  • Red Hat Enterprise Linux 5
  • DRBD

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content