Cluster node unable to access storage after being fenced with fence_scsi and rebooting in RHEL 6

Solution Unverified - Updated -

Issue

  • Cluster node unable to access storage while rejoining the cluster after being fenced with fence_scsi and rebooting
  • Cluster node is getting reservation conflicts even after unfencing itself
  • When we wanted to test if fencing with fence_scsi works correctly and removed one node out of the cluster (shut down the server) the key of that node was removed from the disks (which seem to be the normal behavior). However the stopped node could never return into the cluster because it stopped during the boot process, as it could not gain access to all the devices.
  • SCSI reservation conflicts, I/O errors, and multipath map path failures occur when a node is rebooting after it has been fenced by fence_scsi
Jul  4 13:35:04 node1 kernel: sd 2:0:1:11: reservation conflict
Jul  4 13:35:29 node1 kernel: sd 2:0:1:5: reservation conflict
Jul  4 13:35:32 node1 kernel: sd 2:0:1:11: reservation conflict
Jul  4 13:35:33 node1 kernel: sd 1:0:0:5: reservation conflict
Jul  4 13:35:41 node1 kernel: sd 2:0:0:1: reservation conflict
Jul  4 13:38:52 node1 kernel: sd 2:0:1:11: reservation conflict
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Device not ready
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Sense Key : Not Ready [current] 
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Add. Sense: Logical unit not ready, manual intervention required
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Jul  4 13:40:25 node1 kernel: end_request: I/O error, dev sde, sector 0
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Device not ready
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Sense Key : Not Ready [current] 
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Add. Sense: Logical unit not ready, manual intervention required
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] CDB: Read(10): 28 00 0c 7f ff 80 00 00 08 00

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add on
  • One or more nodes configured to use the fence_scsi fencing agent

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content