Sun Storage 7000 Unified Storage System: Issue With Shadow Migration. Oracle DB ORA-01578 ORA-01110 errors

Asset ID:	1-72-1469932.1
Update Date:	2012-10-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1469932.1 : Sun Storage 7000 Unified Storage System: Issue With Shadow Migration. Oracle DB ORA-01578 ORA-01110 errors

Applies to:

Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7720 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Symptoms

Below is the error reported:

Mon Jun 11 22:33:37 2012
Hex dump of (file 405, block 82827) in trace file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc
Corrupt block relative dba: 0x6541438b (file 405, block 82827)
Completely zero block found during buffer read
Reading datafile '/psatoc/oradata/data01/a_txn_ind04.dbf' for corruption at rdba: 0x6541438b (file 405, block 82827)
Reread (file 405, block 82827) found same corrupt data (no logical check)
Mon Jun 11 22:33:37 2012
Corrupt Block Found
  TSN = 382, TSNAME = APPS_TS_TX_IDX
  RFN = 405, BLK = 82827, RDBA = 1698775947
  OBJN = -1, OBJD = 299514, OBJECT = APPS_TS_TX_IDX, SUBOBJECT =
  SEGMENT OWNER = , SEGMENT TYPE = Temporary Segment
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc (incident=65081):
ORA-01578: ORACLE data block corrupted (file # 405, block # 82827)
ORA-01110: data file 405: '/psatoc/oradata/data01/a_txn_ind04.dbf'
Incident details in: /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc:

Changes

Mon Jun 11 22:33:37 2012
Hex dump of (file 405, block 82827) in trace file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc
Corrupt block relative dba: 0x6541438b (file 405, block 82827)
Completely zero block found during buffer read
Reading datafile '/psatoc/oradata/data01/a_txn_ind04.dbf' for corruption at rdba: 0x6541438b (file 405, block 82827)
Reread (file 405, block 82827) found same corrupt data (no logical check)
Mon Jun 11 22:33:37 2012
Corrupt Block Found
TSN = 382, TSNAME = APPS_TS_TX_IDX
RFN = 405, BLK = 82827, RDBA = 1698775947
OBJN = -1, OBJD = 299514, OBJECT = APPS_TS_TX_IDX, SUBOBJECT =
SEGMENT OWNER = , SEGMENT TYPE = Temporary Segment
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc (incident=65081):
ORA-01578: ORACLE data block corrupted (file # 405, block # 82827) <-------------------------------------------------------------
ORA-01110: data file 405: '/psatoc/oradata/data01/a_txn_ind04.dbf'
Incident details in: /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc:

How to I diagnose and fix this ORA-01578 error?

Answer: Since you are getting a ORA-01578 running Oracle on a personal computer (Windows), it's a safe bet that you have a disk crash. Oracle is not designed to run on PC's, it made for industrial strength servers.

Cause

We have initiated a shadow copy to Sun storage :adc07stor21 from a netapp filer and it got finished @ below time
2012-6-10 11:25:53 6c121f9a-2626-ca71-98a8-eb14811a77ae Migration of 'psatoc_21/i_oradata' from 'nfs://adc07ntap29-bkp/vol/psatoc_29_data_3_2M0A0G0_MIGSRC_EXP_17062012/i_oradata' completed.

http://www.dba-oracle.com/t_ora_01578_data_block_corrupted.htm

In your case, the ORA-01578 block corruption error, with is related to the ORA-26040 corrupt block error:

ORA-01578: ORACLE data block corrupted

To diagnose the ORA-01578 error, use the oerr utility to look-up the ORA-01579 error:

ORA-01578: ORACLE data block corrupted (file # string, block # string)

Cause: The data block indicated was corrupted, mostly due to software errors.

Action: Try to restore the segment containing the block indicated.
This may involve dropping the segment and recreating it.
If there is a trace file, report the errors in it to your ORACLE representative.
Normally, block corruption is often due to a hardware error, and there is likely a trace file or log entry that was made when it happened.

By examining the trace file, you can often tel the root cause of the corruption, or an Oracle Representative must now tell you the root cause of the corruption.

Beware, you may need to perform an RMAN recovery on this database, up to the point of the corruption, usually a disk failure.

Solution

Solution:
You will need to open a Call with https://support.oracle.com/ with your Oracle Database CSI# to help salvage or clean up the ORA-01578: ORACLE data block corrupted (file # 405, block # 82827)

Action Plan:
Oracle technical Support can use the block editor (BBED) utility to inspect the data blocks and see EXACTLY why they became corrupted. You can also inspect the blocks with BBED yourself.

References

<NOTE:1578.1> - OERR: ORA-1578 "ORACLE data block corrupted (file # %s, block # %s)" Master Note
<NOTE:729433.1> - Cannot Reuse a Corrupt Block in Flashback Mode, ORA-1578

Attachments

This solution has no attachment