file system resource does not fail its status checks after a file system has failed and remounted read-only in a RHEL 6 Update 6 High Availability cluster with rgmanager
Issue
- After updating to RHEL 6 Update 6, when I pull a fiber cable to test the cluster's reaction, our
fsresource never fails its status checks even though the file system has remounted read-only.
# tail -f /var/log/messages
Jan 19 10:42:44 rhel6-node1 rgmanager[2836]: Service service:fstest started
[...]
Jan 19 10:43:05 rhel6-node1 multipathd: mpathc: remaining active paths: 0
Jan 19 10:43:31 rhel6-node1 multipathd: mpathc: Disable queueing
Jan 19 10:43:31 rhel6-node1 kernel: Buffer I/O error on device dm-3, logical block 131584
Jan 19 10:43:31 rhel6-node1 kernel: lost page write due to I/O error on dm-3
Jan 19 10:43:31 rhel6-node1 kernel: JBD: I/O error detected when updating journal superblock for dm-3.
Jan 19 10:43:31 rhel6-node1 kernel: Aborting journal on device dm-3.
Jan 19 10:43:31 rhel6-node1 kernel: Buffer I/O error on device dm-3, logical block 131584
Jan 19 10:43:31 rhel6-node1 kernel: lost page write due to I/O error on dm-3
Jan 19 10:43:31 rhel6-node1 kernel: JBD: I/O error detected when updating journal superblock for dm-3.
Jan 19 10:43:39 rhel6-node1 kernel: EXT3-fs (dm-3): error: ext3_journal_start_sb: Detected aborted journal
Jan 19 10:43:39 rhel6-node1 kernel: EXT3-fs (dm-3): error: remounting filesystem read-only
[root@rhel6-node1 ~]# tail -f /var/log/cluster/rgmanager.log
Jan 19 10:41:11 rgmanager Starting changed resources.
Jan 19 10:42:44 rgmanager Starting disabled service service:fstest
Jan 19 10:42:44 rgmanager [fs] mounting /dev/dm-3 on /mnt/testmnt
Jan 19 10:42:44 rgmanager [fs] mount -t ext3 /dev/dm-3 /mnt/testmnt
Jan 19 10:42:44 rgmanager Service service:fstest started
Jan 19 10:42:59 rgmanager [fs] Checking fs "mpathc", Level 20
Jan 19 10:43:09 rgmanager [fs] Checking fs "mpathc", Level 20
Jan 19 10:43:19 rgmanager [fs] Checking fs "mpathc", Level 20
Jan 19 10:43:29 rgmanager [fs] Checking fs "mpathc", Level 20
Jan 19 10:43:39 rgmanager [fs] Checking fs "mpathc", Level 20
Jan 19 10:43:49 rgmanager [fs] Checking fs "mpathc", Level 20
Jan 19 10:43:59 rgmanager [fs] Checking fs "mpathc", Level 20
fswrite tests during the status checks are incorrectly succeeding after a file system has failed
Environment
- Red Hat Enterprise Linux (RHEL) 6 Update 6 with the High Availability Add On
rgmanagerresource-agentsreleases starting with3.9.5-12.el6RHBA-2014:1428 and prior to3.9.5-12.el6_6.3RHBA-2015:0130- One or more
fs,clusterfs, ornetfsresource resource agents configured in/etc/cluster/cluster.conf
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.