Oracle DB corruption

Solution In Progress - Updated -

Issue

Hi,

We require assistance from Redhat to validate and isolate any possible OS as a root cause of the below scenario.

On Thursday (7/May) we performed a managed fail-over of an Oracle Database operating as part of a three node Veritas VSC cluster. We have one server which had hardware issues (GLEDB05) which needed to be shut down in order to perform the replacement of several components. We currently have two Oracle databases, one of which can fail over to a standby server as follows (state prior to the change):
- gledb04 (standby)
- gledb05 (active / DB 1)
- gledb06 (active / DB 2)

Our intention was to gracefully shutdown down oracle on GLEDB05 in order for VSC to bring up the database on GLEDB04. Unfortunately, this did not occur because when gledb04 took over, and Oracle attempted to bring up the database, it reported underlying data corruption.

For the next 20hrs we attempted to transfer a hot backup, tape backup data across to GLEDB04 so that Oracle could start on this server. On each occasion Oracle reported data corruption.

We were finally able to resolve the issue by performing a data transfer to GLEDB05 (which had parts replaced and was back in service). Oracle was able to successfully load the database backups without corruption and services were restored soon after.

As part of the root cause analysis we need to understand if the underlying OS contributed to the data corruption in any way. Either via transfer of data over the HBA to the VxFS file system used by VSC. We have 2/3 systems which did not display this behaviour while one did.

For your reference we have active service cases with multiple vendors relating to this issue:
Symantec/Veritas: #05635682
Oracle: SR 3-10705762771
Netapp: 2005649157 , 2003033348
Brocade (via Netapp): 2005654773
HP: TBC

Environment

Red Hat Enterprise Linux
VxFS
5.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.