Chapter 6. Cold Failover Cluster Configuration

This chapter provides information on configuration a cold failover HA cluster. For information on configuring a RAC/GFS cluster, see Chapter 5, RAC/GFS Cluster Configuration.

Long before RAC (and its progenitor, OPS) was suitable for high availability, customers still needed Oracle databases to be more reliable. The best way to do this was with a (relatively) simple two-node cluster that provided a second server node to take over in the event the primary node crashed. These early clusters still required many of the shared attributes that OPS/RAC databases required, but mandated that only one Oracle instance could be running at once; the storage was shared, but Oracle access was not. The use of this “failover” configuration remains in wide use today.

The OS clustering layer must insure that Oracle is never running on both nodes at the same time. If this occurs, the database will be corrupted. The two nodes must be in constant contact, either through a voting disk, or a heartbeat network, or both. If something goes wrong with the primary node (the node currently running Oracle), then the secondary node must be able to terminate that server, take over the storage, and restart the Oracle database. Termination is also called fencing, and is most frequently accomplished by the secondary node turning off the power to the primary node; this is called power-managed fencing. There are a variety of fencing methods, but power-managed fencing is recommended.

The Oracle database must be installed on a shared storage array and this file system (or these file systems) can only be mounted on the active node. The clustering layer also has agents, or scripts that must be customized to the specific installation of Oracle. Once configured, this software can automatically start the Oracle database and any other relevant services (like Oracle network listeners). The job of any cluster product is to ensure that Oracle is only ever running on one node.

Clusters are designed specifically to handle bizarre, end-case operating conditions, but are at the mercy of the OS components that might fail too. The heartbeat network operates over standard TCP/IP networks, and is the primary mechanism by which the cluster nodes identify themselves to other members. This ensures the cluster is viable and that Oracle can continue to operate on the primary node. Some failure cases can cause the heartbeat to become erratic or unreliable, so modern clustering products provide a second check-in mechanism, which insures that quorum is maintained. Quorum voting causes each cluster member to identify itself by voting, in the form of a simple write to a shared vote, or quorum disk. The combination of heartbeat and quorum disk minimizes the risk of split-brain cluster states. Split-brain clusters occur when the two nodes think they are both in the correct cluster state, so both access the shared storage. Split-brain states create the highest risk of database corruption, so this is the functional core of the cluster.

The two most common examples of Oracle Cold Failover are HP ServiceGuard and Veritas Cluster Server (VCS). Red Hat Cluster Suite’s implementation for Oracle closely models these products, so customers familiar with them will be immediately familiar with Red Hat Cluster Suite Oracle HA. Of course, the devil is most definitely in the details.