Issue

• A SAN rebooted unexpectedly resulting in a fence event and the GFS2 fs went read only on one node. Upon stabilizing the cluster, node 2 now throws a withdraw when attempting to mount the GFS2 filesystem:

Feb  4 07:21:38 node42 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "mycluster:myGFS2fs"
Feb  4 07:21:38 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: Joined cluster. Now mounting FS...
Feb  4 07:21:38 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: jid=1, already locked for use
Feb  4 07:21:38 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: jid=1: Looking at journal...
Feb  4 07:21:39 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: jid=1: Done
Feb  4 07:21:39 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: fatal: invalid metadata block
Feb  4 07:21:39 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1:   bh = 65972 (type: exp=14, found=4)
Feb  4 07:21:39 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1:   function = gfs2_quota_init, file = fs/gfs2/quota.c, line = 1248
Feb  4 07:21:39 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: about to withdraw this file system
Feb  4 07:21:40 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: telling LM to unmount
Feb  4 07:21:40 node42 kernel: GFS2: fsid=mycluster:myGFS2fs.1: withdrawn
Feb  4 07:21:40 node42 kernel: Pid: 3602, comm: mount.gfs2 Not tainted 2.6.32-358.18.1.el6.x86_64 #1
Feb  4 07:21:40 node42 kernel: Call Trace:
Feb  4 07:21:40 node42 kernel: [<ffffffffa035af48>] ? gfs2_lm_withdraw+0x128/0x160 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffffa0345077>] ? gfs2_getbuf+0xc7/0x140 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffffa035b050>] ? gfs2_metatype_check_ii+0x50/0x60 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffffa03515d7>] ? gfs2_quota_init+0x357/0x3c0 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffffa0357e2e>] ? gfs2_make_fs_rw+0xbe/0x160 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffffa0357dbb>] ? gfs2_make_fs_rw+0x4b/0x160 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffff81063465>] ? wake_up_process+0x15/0x20
Feb  4 07:21:40 node42 kernel: [<ffffffffa034bcba>] ? gfs2_get_sb+0x95a/0xa20 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffffa033f289>] ? gfs2_glock_nq_num+0x59/0xa0 [gfs2]
Feb  4 07:21:40 node42 kernel: [<ffffffff81160c3a>] ? alloc_pages_current+0xaa/0x110
Feb  4 07:21:40 node42 kernel: [<ffffffff81183beb>] ? vfs_kern_mount+0x7b/0x1b0
Feb  4 07:21:40 node42 kernel: [<ffffffff81183d92>] ? do_kern_mount+0x52/0x130
Feb  4 07:21:40 node42 kernel: [<ffffffff811a3ef2>] ? do_mount+0x2d2/0x8d0
Feb  4 07:21:40 node42 kernel: [<ffffffff811a4580>] ? sys_mount+0x90/0xe0
Feb  4 07:21:40 node42 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b