Receiving ORACLE data block corrupted error messages

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Oracle Database

Issue

  • The following errors were seen in Oracle logs after changes were made to the underlining storage. Logical Volume Manager is being used along with an ext filesystem. The logical volume and ext filesystem were extended.

Block corruption reported by oracle database Caused by: java.sql.SQLException:

ORA-01578: ORACLE data block corrupted (file # x, block # x) 
ORA-01110: data file '<filename>.ddf'

Similar messages are seen in the Oracle logs:

Corrupt block relative dba: 0x<data block access> (file x, block x)
Completely zero block found during multiblock buffer read
Reading datafile '/<path>/<file name>.dbf' for corruption at <device>: 0x<address> (file x, block x)
Wed Feb 13 22:03:15 2013
Corrupt Block Found
Reread (file x, block x) found same corrupt data (no logical check)
         TSN = x, TSNAME = <data>
         RFN = x, BLK = x, RDBA = <relative data block access>
         OBJN = x, OBJD = x, OBJECT = <name>, SUBOBJECT =
         SEGMENT OWNER = CDM, SEGMENT TYPE = Table Segment
Corrupt Block Found
         TSN = <transparant network substract>, TSNAME = <data>
         RFN = x, BLK = x, RDBA = <relative data block access>
         OBJN = x, OBJD = x, OBJECT = <name>, SUBOBJECT =
         SEGMENT OWNER = CDM, SEGMENT TYPE = Table Segment
Errors in file /<path>/<name>.trc  (incident=29113):
ORA-01578: ORACLE data block corrupted (file # x, block # x)
ORA-01110: data file x: '/<path>/<name>.dbf'
Incident details in: /<path>/incdir_x/<name>.trc
DBMS_STATS: GATHER_STATS_JOB encountered errors.  Check the trace file.
Errors in file /<path>/<name>.trc:
ORA-20011: Approximate NDV failed: ORA-01578: ORACLE data block corrupted (file # x, block # x)
ORA-01110: data file x: '/<path>/<name>.dbf'

Resolution

The following steps should be taken to prevent corruption within RHEL after device changes have been made:

  • If Oracle is running, stop the database

If the filesystem is on a logical volume, and if that volume has run out of space:

The information provided below is from Oracle's documentation. It has not been confirmed from Red Hat.
As it is a product that is not supported at Red Hat, we are only providing this for your convenience.
Please contact Oracle for verification.

Once the storage has been confirmed to not contain errors, execute the following:

DBMS_REPAIR.CHECK_OBJECT to find corrupt blocks
DBMS_REPAIR.FIX_CORRUPT_BLOCKS to fix the corrupted blocks

You can find more information on how to recover the Oracle database by reading the following Oracle Knowledge Base article.
NOTE: An Oracle support contract is required in order to access the information

https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=556733.1

Again, please contact Oracle for support.

Root Cause

There is no known root cause for this error. We at Red Hat have confirmed that there were no error messages in
our messages logs and have also confirmed that LVM was created properly. These errors were only seen in the Oracle logs.

Diagnostic Steps

  • Check the messages file to verify there are no I/O errors or errors that can be seen shortly before the Oracle error.
  • Verify that LVM has been created/extended correctly through the lvmdump.
  • If there are no issues seen that lead to the Oracle errors, please contact Oracle for support.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments