kernel: sd 1:0:0:0: reservation conflict
Hi,
I have a problem about Red Hat Cluster, it detected cluster services failure and then it restarted services automatically. Error message as the following:
Feb 16 10:06:22 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 16 10:06:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 16 10:06:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 0e 25 3f 00 00 08 00
Feb 16 10:06:22 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 927039
Please kindly help.
Thank you very much.
Thirat S.
Attachments
Responses
The "reservation conflict" error messages mean you're using SCSI-3 Persistent Reservations for cluster fencing, and your other cluster node(s) judged kcsandnp01 had failed, and fenced it out of the cluster by removThe "reservation conflict" error messages mean you're using SCSI-3 Persistent Reservations for cluster fencing, and your other cluster node(s) judged kcsandnp01 had failed, and fenced it out of the cluster by removing its write access to the shared disks. This obviously happens with no input from kcsandnp01, so kcsandnp01 may have some data in its disk cache that is waiting to be committed to the disk.
After fencing kcsandnp01, the other node(s) have executed appropriate recovery actions (filesystem journal recovery; possibly a filesystem check for non-cluster filesystems if required) and are now again using those disks. As a result, the data in kcsandnp01's cache is now stale: if it was written to the shared disks, it would overwrite newer information in an uncoordinated manner. So that must not be allowed to happen. Any filesystem errors resulting in missing those write operations are already corrected by the other nodes (if they are automatically correctable).
Unfortunately, there is no way for the cluster to tell the kernel to purge the stale data from the cache: in kcsandnp01's kernel's viewpoint, that means intentionally losing data, and the kernel just won't do that. This means that kcsandnp01 must now execute a crash-style reboot: it won't be allowed to rejoin the cluster properly until it has rebooted, to ensure that the stale data is definitely purged from kcsandnp01's disk cache.
Trying to reboot kcsandnp01 normally in this situation would very likely just hang, as at the end of the shutdown process the kernel would get stuck in a loop of "I must write out all the disk caches... But I can't write to those disks... But I must... But I can't..."
If you need a way to execute a crash-style reboot remotely, and don't have a hardware-based remote console available, I've found that this command can do the job:
echo b > /proc/sysrq-trigger
This will reboot the system immediately, ignoring any disk caches: if you do it in a SSH session, your session will most likely hang even before you can see your Enter keypress at the end of the command echoed back to your terminal.
The errors listed in the attached messages.zip might be mostly consequences of the SCSI reservation fencing: unless the processes specifically check for missing SCSI reservations, there will be no indication that the system is actually blocked from writing to the shared disks until they try. There are also some messages about "Forcing vmfence away from kcsandnp01 after 1000000 failures": to me, this suggests that there might be an alternate fencing mechanism at work that probably should have rebooted the kcsandnp01, but didn't, perhaps because its connection to VM hypervisor control is no longer available. Maybe check for expired passwords and/or locked accounts in the VM management system?
If your cluster is set up properly for SCSI fencing, the cluster should take care of unfencing as soon as kcsandnp01 has been rebooted and is rejoining the cluster. But it looks like the system might be not doing that properly, or perhaps some processes that are not managed by the clustering software are trying to access the shared disks.
This appears to be a Pacemaker cluster, so this RedHat Knowledgebase article might be helpful: https://access.redhat.com/articles/530533
There was no mention whether this is a RHEL 6 or 7 system: that would be important, as there are some differences in Pacemaker configuration for SCSI fencing between major releases.
If you need to implement a watchdog or some other type of test for SCSI fencing status, this might be helpful: https://access.redhat.com/solutions/65187
(And after all this writing, I notice that the original post is now quite old... well... I hope this will be of some use to someone else then!)
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
