Cluster node unable to access storage after being fenced with fence_scsi and rebooting in RHEL 6
Issue
- Cluster node unable to access storage while rejoining the cluster after being fenced with
fence_scsi
and rebooting - Cluster node is getting reservation conflicts even after unfencing itself
- When we wanted to test if fencing with
fence_scsi
works correctly and removed one node out of the cluster (shut down the server) the key of that node was removed from the disks (which seem to be the normal behavior). However the stopped node could never return into the cluster because it stopped during the boot process, as it could not gain access to all the devices. - SCSI reservation conflicts, I/O errors, and
multipath
map path failures occur when a node is rebooting after it has been fenced byfence_scsi
Jul 4 13:35:04 node1 kernel: sd 2:0:1:11: reservation conflict
Jul 4 13:35:29 node1 kernel: sd 2:0:1:5: reservation conflict
Jul 4 13:35:32 node1 kernel: sd 2:0:1:11: reservation conflict
Jul 4 13:35:33 node1 kernel: sd 1:0:0:5: reservation conflict
Jul 4 13:35:41 node1 kernel: sd 2:0:0:1: reservation conflict
Jul 4 13:38:52 node1 kernel: sd 2:0:1:11: reservation conflict
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Device not ready
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Sense Key : Not Ready [current]
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Add. Sense: Logical unit not ready, manual intervention required
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Jul 4 13:40:25 node1 kernel: end_request: I/O error, dev sde, sector 0
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Device not ready
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Sense Key : Not Ready [current]
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Add. Sense: Logical unit not ready, manual intervention required
Jul 4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] CDB: Read(10): 28 00 0c 7f ff 80 00 00 08 00
Environment
- Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add on
- One or more nodes configured to use the
fence_scsi
fencing agent
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.