Oracle HA cluster in RHEL 5.5

Latest response

Hi,

 

I've 2 HP DL380 G6 server with Redhat Enterprise Linux Advance Edition 5.5 x64 intalled (both the nodes having Fiber HBA cards and connected with SAN switches). I've assigned 5 common storage LUNs to both these servers on GFS2 filesystem and their mount points are /oradata, /data1, /index, /redo and /arch. I've setup 2 node Active / Passive cluster using Redhat cluster suite. I've Set the priority 1 to node 1 and 2 to node 2 in failover domain. The same volume and virtual ip ( or can say cluster ip for applications) will be available at one node at a time. In case of failure of node 1, the cluster resources (i.e. VIP, SAN vols) will switchover to node 2. As soon as the node 1 start working and get on the cluster resources are shifted back to node 1. This setup is tested and working fine.

Now, I've to configure the RHEL cluster in this way so that Oracle DB instance(s) should failover along with the other resources (vip and san vols).

Both servers having oracle 11g R2 Enterprise (non-RAC) installed on both servers. We're using GFS2 filesystem over CLVM on shared SAN volumes.

I want to clear few things that We're going to implement in this scenario.

 

1. we're not going to use Oracle RAC. We've configured OS level clustering in Active / Passive mode.
2. I want to start the Oracle DB services automatically by cluster service on Primary node while it will keep stopped in secondary node.
3. when the cluster services will shift from Primary to Secondary server, it should start the DB services on the other node, but before shifting to other node the DB services should stopped properly on the Primary node. For testing this, I've purposely stopped the rgmanager service (i.e. service rgmanager stop), which immediately starting to failover the cluster services and resources to secondary node. But what I've observed that the Oracle DB service still running on primary and the resources shifted to secondary node except those volumes which are still accessed by oracle or underlaying services (i.e. /index, /redo, /arch) on primary node. Rest volumes shifted to 2nd node. Later the cluster service failed to start (verified by "clustat" command).

 

I want such script which should be understandable and manageagle by cluster resource manager. It should work on both scenarios...

 

1. In case, 1st node get down completely. It should shift all the cluster resources (i.e. vip, san vols and oracle db service).
2. In case, if there any os dependent of application/db dependent service fails or not responds, cluster should be intelligent enough to understand that and failover the cluster services from 1st node to 2nd node properly and fence it (suppose, if i stop the rgmanager or oracle listener service, it should understand this and failover the cluster services to 2nd node and then fence it). But it's not happening.

 

Does anybody get any success in this. Pls help me to accomplish the same (either using conga or system-config-cluster).

 

Regards

Arunabh

Responses

Hi Arunabh-
Would you please paste your /etc/cluster/cluster.conf?

 Hi Phil Jensen,
 

 

<?xml version="1.0"?>

<cluster alias="ucm_cluster" config_version="43" name="ucm_cluster">

<fence_daemon post_fail_delay="0" post_join_delay="3"/>

<clusternodes>

<clusternode name="mpjbpdcucm1-internal" nodeid="1" votes="1">

<fence>

<method name="1">

<device name="ilo1"/>

</method>

</fence>

</clusternode>

<clusternode name="mpjbpdcucm2-internal" nodeid="2" votes="1">

<fence>

<method name="1">

<device name="ilo2"/>

</method>

</fence>

</clusternode>

</clusternodes>

<cman expected_votes="1" two_node="1"/>

<fencedevices>

<fencedevice action="off" agent="fence_ilo" hostname="10.10.40.92" login="admin" name="ilo1" passwd="admin@123"/>

<fencedevice action="off" agent="fence_ilo" hostname="10.10.40.93" login="admin" name="ilo2" passwd="admin@123"/>

</fencedevices>

<rm log_facility="local4" log_level="5">

<failoverdomains>

<failoverdomain name="domain_1" ordered="1" restricted="1">

<failoverdomainnode name="mpjbpdcucm1-internal" priority="1"/>

<failoverdomainnode name="mpjbpdcucm2-internal" priority="2"/>

</failoverdomain>

</failoverdomains>

<resources>

<ip address="10.10.34.93/24" monitor_link="1"/>

<clusterfs device="/dev/vgoradata/lvoradata" force_unmount="1" fsid="15674" fstype="gfs2" mountpoint="/oradata" name="oradata" options="rw"/>

<clusterfs device="/dev/vgdata1/lvdata1" force_unmount="1" fsid="11430" fstype="gfs2" mountpoint="/data1" name="data1" options="rw"/>

<clusterfs device="/dev/vgindex/lvindex" force_unmount="1" fsid="26369" fstype="gfs2" mountpoint="/index" name="index" options="rw"/>

<clusterfs device="/dev/vgredo/lvredo" force_unmount="1" fsid="27818" fstype="gfs2" mountpoint="/redo" name="redo" options="rw"/>

<clusterfs device="/dev/vgarch/lvarch" force_unmount="1" fsid="8778" fstype="gfs2" mountpoint="/arch" name="arch" options="rw"/>

<script file="/home/dbuser/dbstart.sh" name="DBSTART"/>

</resources>

<service autostart="1" domain="domain_1" exclusive="1" name="SRV_IP_GFS" recovery="relocate">

<ip ref="10.10.34.93/24"/>

<clusterfs ref="oradata"/>

<clusterfs ref="data1"/>

<clusterfs ref="index"/>

<clusterfs ref="redo"/>

<clusterfs ref="arch"/>

<script ref="DBSTART"/>

</service>

</rm>

</cluster>

 

Regards

Arunabh

 Hi Phil Jensen,
 

 

<?xml version="1.0"?>

<cluster alias="ucm_cluster" config_version="43" name="ucm_cluster">

<fence_daemon post_fail_delay="0" post_join_delay="3"/>

<clusternodes>

<clusternode name="mpjbpdcucm1-internal" nodeid="1" votes="1">

<fence>

<method name="1">

<device name="ilo1"/>

</method>

</fence>

</clusternode>

<clusternode name="mpjbpdcucm2-internal" nodeid="2" votes="1">

<fence>

<method name="1">

<device name="ilo2"/>

</method>

</fence>

</clusternode>

</clusternodes>

<cman expected_votes="1" two_node="1"/>

<fencedevices>

<fencedevice action="off" agent="fence_ilo" hostname="10.10.40.92" login="admin" name="ilo1" passwd="admin@123"/>

<fencedevice action="off" agent="fence_ilo" hostname="10.10.40.93" login="admin" name="ilo2" passwd="admin@123"/>

</fencedevices>

<rm log_facility="local4" log_level="5">

<failoverdomains>

<failoverdomain name="domain_1" ordered="1" restricted="1">

<failoverdomainnode name="mpjbpdcucm1-internal" priority="1"/>

<failoverdomainnode name="mpjbpdcucm2-internal" priority="2"/>

</failoverdomain>

</failoverdomains>

<resources>

<ip address="10.10.34.93/24" monitor_link="1"/>

<clusterfs device="/dev/vgoradata/lvoradata" force_unmount="1" fsid="15674" fstype="gfs2" mountpoint="/oradata" name="oradata" options="rw"/>

<clusterfs device="/dev/vgdata1/lvdata1" force_unmount="1" fsid="11430" fstype="gfs2" mountpoint="/data1" name="data1" options="rw"/>

<clusterfs device="/dev/vgindex/lvindex" force_unmount="1" fsid="26369" fstype="gfs2" mountpoint="/index" name="index" options="rw"/>

<clusterfs device="/dev/vgredo/lvredo" force_unmount="1" fsid="27818" fstype="gfs2" mountpoint="/redo" name="redo" options="rw"/>

<clusterfs device="/dev/vgarch/lvarch" force_unmount="1" fsid="8778" fstype="gfs2" mountpoint="/arch" name="arch" options="rw"/>

<script file="/home/dbuser/dbstart.sh" name="DBSTART"/>

</resources>

<service autostart="1" domain="domain_1" exclusive="1" name="SRV_IP_GFS" recovery="relocate">

<ip ref="10.10.34.93/24"/>

<clusterfs ref="oradata"/>

<clusterfs ref="data1"/>

<clusterfs ref="index"/>

<clusterfs ref="redo"/>

<clusterfs ref="arch"/>

<script ref="DBSTART"/>

</service>

</rm>

</cluster>

 

Regards

Arunabh

Hi Arunabh,

You mentioned that you are running RHEL 5.5, which I assume means you've got rgmanager-2.0.52-6.el5 installed.  This version had a bug in it which may prevent clusterfs resources from stopping when the service is stopped or relocated.  The following erratum resolves it:

 

  http://rhn.redhat.com/errata/RHSA-2011-1000.html

 

Can you check /var/log/messages at the time you were relocating the node and see if there are any messages similar to this:

 

  clurgmgrd: [xxxx]: <debug> Not unmounting clusterfs:<name> - still in use by 1 other service(s)

 

If there are, then you will need to install the above erratum in order to prevent this from occurring.

 

If there are not, then can you grab the snippet of /var/log/messages from both nodes that corresponds to when you performed this test, and paste them here?

 

I also have a few questions:

  • When this occurs, is it only the clusterfs file systems that are still running on the primary node, or is it the database also? 
  • What does 'clustat' report for that service after this occurs?
  • Is there a specific reason you are using GFS2 over a single-node file system? It sounds like you're only using an active/passive configuration, meaning the FS will never need to be accessed on both nodes at once. If that is the case, using ext3/4 is more appropriate since it will reduce the complexity of the cluster and you should be able to get better performance out of it.

Regards,

John Ruemker, RHCA

Red Hat Software Maintenance Engineer

Online User Groups Moderator

Hi John,

 

Thanks for your suggestion. I'll give a try the suggested solution.

 

Thanks Again.

 

Arunabh