Certain SCSI Sense Keys can cause all paths in a multipath map to fail repeatedly in Red Hat Enterprise Linux

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 5
    • In RHEL 5: All kernel releases
  • Red Hat Enterprise Linux (RHEL) 6
    • In RHEL 6: kernel releases prior to 2.6.32-220.el6 or 2.6.32-279.el6
  • device-mapper-multipath

Issue

  • All of my paths fail in quick succession after a SCSI sense code is received from the storage
  • Changing a LUN from read-write to read-only on the storage array causes all paths to fail repeatedly and load average to climb.
  • multipath does not distinguish between a target failure and path failure. This can cause multipath to retry an operation that failed on the target on all paths, causing each one to fail in succession.
  • When the kernel issues a discard request to the storage, and the storage doesn't support that method of discarding, all paths fail one after another:
Feb  7 14:39:33 example kernel: sd 1:0:0:0: [sdb] Discard failure
Feb  7 14:39:33 example kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb  7 14:39:33 example kernel: sd 1:0:0:0: [sdb] Sense Key : Illegal Request [current]
Feb  7 14:39:33 example kernel: Info fld=0x0
Feb  7 14:39:33 example kernel: sd 1:0:0:0: [sdb] Add. Sense: Invalid command operation code
Feb  7 14:39:33 example kernel: sd 1:0:0:0: [sdb] CDB: Write same(16): 93 08 00 00 00 00 00 00 00 00 00 40 00 00 00 00
Feb  7 14:39:33 example kernel: end_request: I/O error, dev sdb, sector 0
Feb  7 14:39:33 example kernel: device-mapper: multipath: Failing path 8:16
  • errors such as the following are seen:
Mar  7 16:46:36 example kernel: sd 7:0:2:23: [sdb] Unhandled sense code
Mar  7 16:46:36 example kernel: sd 7:0:2:23: [sdb] Result: hostbyte=invalid driverbyte=DRIVER_SENSE
Mar  7 16:46:36 example kernel: sd 7:0:2:23: [sdb] Sense Key : Data Protect [current]
Mar  7 16:46:36 example kernel: sd 7:0:2:23: [sdb] Add. Sense: Write protected
Mar  7 16:46:36 example kernel: sd 7:0:2:23: [sdb] CDB: Write(10): 2a 00 00 00 00 00 00 00 08 00
Mar  7 16:46:36 example kernel: end_request: I/O error, dev sdb, sector 0
Mar  7 16:46:36 example kernel: device-mapper: multipath: Failing path 69:336.
Mar  7 16:46:36 example kernel: sd 6:0:1:23: [sdc] Unhandled sense code
Mar  7 16:46:36 example kernel: sd 6:0:1:23: [sdc] Result: hostbyte=invalid driverbyte=DRIVER_SENSE
Mar  7 16:46:37 example kernel: sd 6:0:1:23: [sdc] Sense Key : Data Protect [current]
Mar  7 16:46:37 example kernel: sd 6:0:1:23: [sdc] Add. Sense: Write protected
Mar  7 16:46:37 example kernel: sd 6:0:1:23: [sdc] CDB: Write(10): 2a 00 00 00 00 00 00 00 08 00
Mar  7 16:46:37 example kernel: end_request: I/O error, dev sdc, sector 0
Mar  7 16:46:37 example kernel: device-mapper: multipath: Failing path 69:320.
Mar  7 16:46:37 example multipathd: 69:320: mark as failed
Mar  7 16:46:37 example multipathd: mpatha: remaining active paths: 3
Mar  7 16:46:37 example multipathd: 69:336: mark as failed
Mar  7 16:46:37 example multipathd: mpatha: remaining active paths: 2
Mar  7 16:46:37 example kernel: sd 6:0:0:23: [sdd] Unhandled sense code
Mar  7 16:46:37 example kernel: sd 6:0:0:23: [sdd] Result: hostbyte=invalid driverbyte=DRIVER_SENSE
Mar  7 16:46:37 example kernel: sd 6:0:0:23: [sdd] Sense Key : Data Protect [current]
Mar  7 16:46:37 example kernel: sd 6:0:0:23: [sdd] Add. Sense: Write protected
Mar  7 16:46:37 example kernel: sd 6:0:0:23: [sdd] CDB: Write(10): 2a 00 00 00 00 00 00 00 08 00
Mar  7 16:46:37 example kernel: end_request: I/O error, dev sdd, sector 0
Mar  7 16:46:37 example kernel: device-mapper: multipath: Failing path 69:304.
Mar  7 16:46:37 example kernel: sd 7:0:1:23: [sde] Unhandled sense code
Mar  7 16:46:37 example kernel: sd 7:0:1:23: [sde] Result: hostbyte=invalid driverbyte=DRIVER_SENSE
Mar  7 16:46:37 example kernel: sd 7:0:1:23: [sde] Sense Key : Data Protect [current]
Mar  7 16:46:37 example kernel: sd 7:0:1:23: [sde] Add. Sense: Write protected
Mar  7 16:46:37 example kernel: sd 7:0:1:23: [sde] CDB: Write(10): 2a 00 00 00 00 00 00 00 08 00
Mar  7 16:46:37 example kernel: end_request: I/O error, dev sde, sector 0
Mar  7 16:46:37 example kernel: device-mapper: multipath: Failing path 69:304.
Mar  7 16:46:38 example kernel: end_request: I/O error, dev sde, sector 0
Mar  7 16:46:38 example kernel: device-mapper: multipath: Failing path 69:288.
Mar  7 16:46:38 example multipathd: 69:288: mark as failed
Mar  7 16:46:38 example multipathd: mpatha: remaining active paths: 1
Mar  7 16:46:38 example multipathd: 69:304: mark as failed
Mar  7 16:46:38 example multipathd: mpatha: remaining active paths: 0

Resolution

RHEL6

  • Update to kernel-2.6.32-220.el6 or later to correct most types of path failures resulting from failed or unsupported operations on the storage target.

  • In the case of SCSI discard requests causing path failures, update to kernel-2.6.32-279.el6 or later.

RHEL 5

  • RHEL 5 unfortunately does not contain functionality to distinguish between a general path failure and a target-failure that should not be retried on alternate paths. As such, you may see indefinite path flapping when a target error occurs such as when a LUN is read-only or a reservation conflict occurs. One option for avoiding this situation would be to disable queue_if_no_path and no_path_retry (or set it to a very low value so a device failure can occur before the path_checker can reinstate a path and retry the I/O that triggered the problem). In order to do this, set the following in /etc/multipath.conf:
no_path_retry N

Where N is the number of times the system should retry a path. For more information, refer to RHEL 5's DM Multipath guide.

  • If a device is already in this state where paths are failing, returning, failing, returning etc due to no_path_retry or queue_if_no_path set, you can try manually setting fail_if_no_path on the map to cause that I/O to fail. For example:
# dmsetup message mpath1 0 fail_if_no_path
  • If the SCSI discard requests are causing path failures, then this issue could be work-around by creating the filesystem with discard support disabled. This could be done by using -K option with mkfs command at the time of filesystem creation:
# mkfs.ext4 -K <device>

Root Cause

There are certain types of storage errors or failures that do not necessarily indicate that a device is in a failed state, but rather that the operation in question could not be completed. In these situations, the storage device or target may return an error that includes an additional sense code that indicates what the problem was. For instance, if a LUN is set to be read-only but the host issues a write operation to it, the target will respond with a DATA_PROTECT sense code:

Mar  7 16:46:37 rhel2-t kernel: sd 5:0:3:23: [sdma] Unhandled sense code
Mar  7 16:46:37 rhel2-t kernel: sd 5:0:3:23: [sdma] Result: hostbyte=invalid driverbyte=DRIVER_SENSE
Mar  7 16:46:37 rhel2-t kernel: sd 5:0:3:23: [sdma] Sense Key : Data Protect [current]
Mar  7 16:46:38 rhel2-t kernel: sd 5:0:3:23: [sdma] Add. Sense: Write protected
Mar  7 16:46:38 rhel2-t kernel: sd 5:0:3:23: [sdma] CDB: Write(10): 2a 00 00 00 00 00 00 00 08 00

This doesn't mean that the LUN or path has failed; it just means that it couldn't handle the write operation. There are a number of different conditions which can produce similar behavior.

In RHEL 6 prior to update 2, the kernel SCSI layer and device-mapper-multipath would not distinguish these types of errors from others, and as such they would cause a path failure. When a path failure occurs, device-mapper will reissue any outstanding I/O from the failed path down an alternate path. In these situations where the error is caused by the storage being unable to complete the operation rather than a true path failure, each path is going to return the same error. As a result, multipath will fail each path in succession as it retries that operation down each.

Eventually this will result in a situation where all paths in the multipath map have been marked as failed, even though they are still operational. Typically, as soon as the path checker runs again, it will recognize the path is up and reinstate it. However, if the map has the queue_if_no_path or no_path_retry feature enabled, this will cause that same I/O that caused all the initial path failures to be reissued, triggering the issue over and over indefinitely. If these features are not enabled, the I/O failure would be passed back to userspace or the file system layer and not retried, but if something were to issue another of those problematic operations (such as a write against a read-only LUN), it will trigger the issue once more.

As of RHEL 6 Update 2 and kernel release 2.6.32-220.el6, the SCSI layer and device-mapper will distinguish these specific errors from path failures based on the SCSI sense code that is returned. If it detects one of the relevant errors, rather than retrying that I/O down an alternate path, it will pass that I/O error back up to the next layer (userspace or the file system) to inform it that the operation could not be completed.

This functionality was implemented by Red Hat Engineering via Bugzilla #431754 and #698655.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments