Filesystem went into read-only mode after storage array firmware upgradation.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux

Issue

  • Filesystem went into read-only mode due to unavailability of multipath.
  • While upgrading storage array firmware, filesystem went into read-only mode.

Resolution

Root Cause

  • LVM access underlying disks instead of multipath due to which kernel remounted the filesystem as read-only at the time of issue.

  • LVM filter was not set for multipath devices.

Diagnostic Steps

  • Crosscheck multipath configuration, if it is not set as per recommended by storage vendor then configure multipath as per recommendation from the storage vendor.

  • Check whether LVs are using the multipath device or underlying device.

  • Check configured LVM filter. If it is not set configure LVM filter

  • Check sar logs:


# less sar06 |grep -iE "dev8-96|dev8-192|dev65-128|dev65-224|dev66-64|dev66-160|dev67-0|dev67-96|dev67-192|dev68-32|dev68-128|dev65-32" |grep -i average Average: dev8-96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-192 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev65-32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev65-128 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev65-224 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev66-64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev66-160 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev67-0 14.89 0.55 6171.51 414.44 0.43 29.02 0.28 0.41 <<<< Average: dev67-96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev67-192 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev68-32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev68-128 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 mpathX (253:33) |- (68:128) |- (68:32) |- (67:192) |- (67:96) |- (67:0) |- (66:160) |- (66:64) |- (65:224) |- (65:128) |- (65:32) |- (8:192) `- (8:96)

Though File system is created on multipath devices, I/Os are happening on only one path/disk.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.