Software-Emulated Watchdog Known Limitations
Overview
In Red Hat Enterprise Linux (RHEL) High Availability Cluster with for example storage based fencing (fence_sbd), it is recommended that a Watchdog is enabled. This Watchdog ensures that the node being fenced is able to reboot itself in order to prevent resources running on more than one node at the time and can cause data integrity issues.
Where possible, Red Hat recommends using hardware-based watchdog technology. However, there are increasingly situations (in public clouds, for example), where implementing a hardware-based solution is impractical. In these situations, software-emulated watchdog can be the best available solution, and is fully supported by Red Hat. This article provides a list of things to be aware of when using software-emulated watchdog.
Softdog, a fully supported configuration, operates within the limitations and available resources provided by the kernel, and cannot be guaranteed to carry out the necessary reboot action if the operating system is starved of resources or, like with any scenario, is malfunctioning. Under these scenarios, this can lead to other nodes in the cluster assuming the node has been fenced and thus rebooted while, in fact this is not the case. This can lead to inconsistent state of the cluster or possible data loss.
This article discusses known limitations and configurations where a cluster can potentially behave in an unpredictable way. We recommend that a user carefully reads this article and determine how to configure a cluster that is acceptable for their use case.
Known Issues
Software Lockup
By using a software-emulated watchdog on a node, the action to reboot itself in the event of a fence event, is executed in the same environment as all other processes on the node. In the case of a software lockup or the system is starved of resources, the kernel is not able to schedule any processes and thus is also no longer able to execute the reboot of itself. Users are advised to carefully determine and provide the necessary resources to avoid issues related to starvation.
Virtualized Platforms
When cluster nodes are running as a Virtual Machine on a hypervisor, they may be impacted by, for example, Live Migration or suspend/resume activities. The guest OS does not have control when and how often a Live Migration activity occurs. However during such activity (after the Live Migration preparation), the guest OS is temporarily suspended to complete the Live Migration and continue operations on a different hypervisor. If during the Live Migration, a fence event occurs, the software watchdog may be impacted by the temporary suspension and is thus may not be able to complete the self fencing and execute a reboot.
Particularly Sensitive Workloads
Shared Data Access
In a setup where cluster nodes have access to Resilient Storage (GFS2, SMB/CIFS), it is particularly important to ensure a consistent state of all cluster nodes. In the case of (but not limited to) one of the above mentioned circumstances, where a cluster node is not able to reboot itself after a fence event, already running processes may continue to write to the shared storage. In this case, other node(s) in the cluster trust that a particular node has been fenced and thus its processes are no longer running and not writing to the shared storage. If, because of the failed reboot, those processes are still able to write to the shared storage, it may lead to data inconsistency or even data loss. This situation differs from other shared storage that have for example SCSI Fencing configured that forcibly prevents the fenced node from writing to the shared storage.
For clusters in production environments we recommend using a hardware-specific watchdog driver. However, if no watchdog matches your hardware, softdog can be used as a kernel watchdog module.
Important: Softdog limitations
The softdog driver assumes that at least one CPU is still running. If all CPUs are stuck, the code in the softdog driver that should reboot the system will never be executed. In contrast, hardware watchdogs keep working even if all CPUs are stuck.
Comments