File system going in to read-only

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (All versions)

Issue

  • Filesystem remounted read-only after losing paths to storage (multipath devices).
  • Access to any files in the affected filesystem shows error 'Read-only file system'
Feb 20 08:48:25 HOSTNAME multipathd: checker failed path 8:48 in map mpath1 
Feb 20 08:48:25 HOSTNAME multipathd: mpath1: remaining active paths: 1 

Feb 23 20:49:45 HOSTNAME multipathd: sdb: readsector0 checker reports path is down 
Feb 23 20:49:45 HOSTNAME multipathd: checker failed path 8:16 in map mpath1 
Feb 23 20:49:45 HOSTNAME kernel: device-mapper: multipath: Failing path 8:16.
Feb 23 20:49:45 HOSTNAME multipathd: mpath1: remaining active paths: 1 
Feb 23 20:49:45 HOSTNAME multipathd: sdd: readsector0 checker reports path is down 
Feb 23 20:49:45 HOSTNAME multipathd: checker failed path 8:48 in map mpath1 
Feb 23 20:49:45 HOSTNAME kernel: device-mapper: multipath: Failing path 8:48.
Feb 23 20:49:45 HOSTNAME multipathd: mpath1: remaining active paths: 0 
Feb 23 20:49:45 HOSTNAME multipathd: dm-5: add map (uevent) 
Feb 23 20:49:45 HOSTNAME multipathd: dm-5: devmap already registered 
Feb 23 20:49:45 HOSTNAME multipathd: dm-5: add map (uevent) 
Feb 23 20:49:45 HOSTNAME multipathd: dm-5: devmap already registered 
Feb 23 20:49:48 HOSTNAME kernel: Aborting journal on device dm-6.
Feb 23 20:49:48 HOSTNAME kernel: Buffer I/O error on device dm-6, logical block 1545
Feb 23 20:49:48 HOSTNAME kernel: lost page write due to I/O error on dm-6
Feb 23 20:49:48 HOSTNAME kernel: ext3_abort called.
Feb 23 20:49:48 HOSTNAME kernel: EXT3-fs error (device dm-6): ext3_journal_start_sb: Detected aborted journal
Feb 23 20:49:48 HOSTNAME kernel: Remounting filesystem read-only
Feb 23 20:49:50 HOSTNAME multipathd: sdb: readsector0 checker reports path is up 
Feb 23 20:49:50 HOSTNAME multipathd: 8:16: reinstated 
Feb 23 20:49:50 HOSTNAME multipathd: mpath1: remaining active paths: 1 
Feb 23 20:49:50 HOSTNAME multipathd: sdd: readsector0 checker reports path is up 
Feb 23 20:49:50 HOSTNAME multipathd: 8:48: reinstated 
Feb 23 20:49:50 HOSTNAME multipathd: mpath1: remaining active paths: 2 
Feb 23 20:49:50 HOSTNAME multipathd: dm-5: add map (uevent) 
Feb 23 20:49:50 HOSTNAME multipathd: dm-5: devmap already registered 
Feb 23 20:49:50 HOSTNAME multipathd: dm-5: add map (uevent) 
Feb 23 20:49:50 HOSTNAME multipathd: dm-5: devmap already registered 
Feb 23 20:49:57 HOSTNAME CA eHealth SystemEDGE[26318]: sysedge: failed to open /proc/net/snmp6, 2

Resolution

  • Restore paths to LUN

Root Cause

  • All paths to LUN were lost. If the kernel finds fatal corruption on the disk or if certain key IOs like journal writes start failing, it will remount the filesystem as read-only.

Diagnostic Steps

  • Obtain sosreport and/or copy of /var/log/messages

    Jun 25 08:34:51 HOSTNAME multipathd: dm-6: devmap already registered
Jun 25 08:34:54 HOSTNAME kernel: Buffer I/O error on device dm-8, logical block 12132957
Jun 25 08:34:54 HOSTNAME kernel: lost page write due to I/O error on dm-8
Jun 25 08:34:54 HOSTNAME kernel: Aborting journal on device dm-8.
Jun 25 08:34:54 HOSTNAME kernel: Buffer I/O error on device dm-8, logical block 2057
.
.
.
Jun 25 08:35:02 HOSTNAME kernel: EXT3-fs error (device dm-8): ext3_journal_start_sb: Detected aborted journal
Jun 25 08:35:02 HOSTNAME kernel: Remounting filesystem read-only

  • Check multipath -ll -v4

    mpath1 (360000970000192602121533032434535) dm-5 EMC,SYMMETRIX
[size=109G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 0:0:0:1 sdb 8:16  [active][ready]
 \_ 1:0:0:1 sdd 8:48  [active][ready]

# grep dm-6 sos_commands/devicemapper/multipath_-v4_-ll 
dm-6: blacklisted

# grep dm-5 sos_commands/devicemapper/multipath_-v4_-ll 
dm-5: blacklisted
mpath1 (360000970000192602121533032434535) dm-5 EMC,SYMMETRIX

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments