VTL - Data read from virtual media appears to be corrupt

Asset ID:	1-72-1022193.1
Update Date:	2011-11-04
Keywords:

Solution Type Problem Resolution Sure

Solution 1022193.1 : VTL - Data read from virtual media appears to be corrupt

Applies to:

Sun StorageTek VTL Value System - Version: 1.0 - Build 1323 to 1.0 - Build 1323 - Release: 1.0 to 1.0
Sun StorageTek VTL Plus Storage Appliance - Version: 1.0 - Build 1323 to 2.0 - Build 1656   [Release: 1.0 to 2.0]
Sun StorageTek VTL Prime System - Version: 1.0 - Build 1813 to 1.1 - Build 2076   [Release: 1.0 to 1.0]
Sun StorageTek VTL Storage Appliance - Version: 4.0 - Build 1221 to 4.0 - Build 1221   [Release: 4.0 to 4.0]
All Platforms
.
***Checked for relevance on 03-11-2011*** (dd-mm-yyyy)

Symptoms

Commvault media server is reporting "Data read from media appears to be corrupted". The restore job failed when reading from the virtual tape drive assigned thru the VTL target port.

Error messages in Commvault logs:

# ERROR CODE [40:91]: Failed to Copy or verify Chunk [463453] in media [000006], Storage Policy [], Copy [], Host [10.23.10.122], Path [scsidev@4:4.100.2], File Number [806], Backup Jobs [ ]. Data read from media appears to be corrupted. Data read from media appears to be corrupted.

Error messages in VTL messages file:

Jan 26 22:39:14 xxxVTL1 qla2x00fs: [ID 359633 kern.info] fffffe8000866c80 qla(1): [CTIO_24xx_TYPE] received with status SRR_RECV!
Jan 26 22:39:23 xxxVTL1 qla2x00fs: [ID 359633 kern.info] fffffe8000866c80 qla(1): [CTIO_24xx_TYPE] received with status SRR_RECV!

Cause

The restore failure was caused by the bad FC link.

Solution

Check/replace FC components for identified FC connection:

FC cable
Switch SFP
VTL HBA
NOTE: If HBA is bad, there will be associated scsi_log_error messages in VTL messages file. Otherwise do not replace (replace cable and SFP first and retest).

Example for determining port affected:
From the VTL messages log we see the restore job failed when reading from the virtual tape drive assigned thru the VTL target port with instance 1 (qla(1)). The restore failure was caused by the bad FC link as we can see the following SRR request sent from the initiator constantly.

       Jan 26 22:39:14 xxxVTL1 qla2x00fs: [ID 359633 kern.info] fffffe8000866c80 qla(1): [CTIO_24xx_TYPE] received with status SRR_RECV!
       Jan 26 22:39:23 xxxVTL1 qla2x00fs: [ID 359633 kern.info] fffffe8000866c80 qla(1): [CTIO_24xx_TYPE] received with status SRR_RECV!

So in this case, we checked the FC link between the client host initiator to the VTL target adaptor port 4 which has instance number 1 (from ipstor.conf file):.
<PhysicalAdapter number="4" info="qla2x00fs_1" WWPN="xxxxxxxxxxxxxxxx" mode="target" monitorWWPN="xxxxxxxxxxxxxxxx" moduleName="qla2x00fs"/>

Additional Information
Commvault message indicated data corruption, but in this case there was NO data corruption. Restore was successful after FC issue was corrected.

Just because the backup software is indicating the media is corrupt does not mean the HBA is faulty. There are several reasons why a tape could become corrupt. The best approach is to gather the necessary xrays, backup server logs and any additional information to have the issue analyzed by Oracle.

If the data is indeed corrupt and Falconstor or the backup software vendor cannot fix or retrieve the data then the virtual tape will need to be deleted and added back into the VTL Appliance.

Attachments

This solution has no attachment