Exadata Database Machine X2-8
 
Exadata Database Machine X2-2 Hardware
 
Enterprise Manager for Exadata
 
Oracle Exadata Hardware
 
Exadata Database Machine V2
 

PLA-Support>Sun Systems>x64>Engineered Systems HW>SN-x64: EXADATA
 

This document explains which I/O errors require disk replacement, which do not, and which should be investigated further. I/O errors can be reported in different places for different reasons, and not every I/O error is due to a physical hard disk problem that requires replacement.

In this Document

Purpose

Troubleshooting Steps

About Disk Error Handling:

Errors for which Disk Replacement is Recommended:

Errors for which Disk Replacement is NOT Recommended:

Conclusion:

References

Applies to:

Purpose

This document explains which I/O errors require disk replacement, which do not, and which should be investigated further. I/O errors can be reported in different places for different reasons, and not every I/O error is due to a physical hard disk problem that requires replacement.

Troubleshooting Steps

About Disk Error Handling:

The inability to read some sectors is not always an indication that a drive is about to fail. Even if the physical disk is damaged at one location, such that a certain sector is unreadable, the disk may be able to use spare space to replace the bad area, so that the sector can be overwritten. Physical hard disks are complex mechanical devices with spinning media, so media errors and other mechanical problems are a fact of life which is why redundancy was designed into Exadata in order to protect data against such errors. It is important to stay up to date on disk vendor's firmware which resolves known issues with internal drive mechanical control, media usage and re-allocation algorithms which are problems that can lead to premature failure if not attended to in a timely manner. The most recent Exadata patch image releases contain the latest disk firmware for each drive supported, as well as continuous improvements in ASM and Cellsrv management and handling of disk related I/O errors and failures. Refer to Note 888828.1 for the latest patch releases.

Physical hard disks used on Exadata support SMART (Self-Monitoring, Analysis, and Reporting Technology) and will report their own SMART status to the RAID HBA in the event of a problem. SMART events are based on thresholds which are vendor-defined for monitoring various internal disk mechanisms and will be different for different types of errors. By definition SMART has only 2 external states – predicted failure or not failed (OK). The SMART status does not necessarily indicate the drive's past or present reliability. Exadata Storage Cells reports disks in 2 places – on the cluster by ASM and on the individual cell server by cellsrv which accesses disks through LSI's RAID HBA Megacli utility.

When a failed disk has occurred, ASM will forcibly re-balance the data off the failed disk to restore redundancy if not replaced within the timout period. Performance may be reduced during a rebalance operation. The rebalance operation status should be checked to ensure that this rebalance has completed successfully, because if the rebalance fails, then redundancy will stay reduced until this is rectified or the disk replaced. On-site spare disks are provided such that failed disks can be replaced rapidly by the customer if they choose, in order to ensure maximum performance and full redundancy is maintained prior to the timeout expiring and forcing a rebalance.

In normal redundancy configuration for the disk groups, 1 disk failure can be survived before ASM rebalance re-establishes data redundancy for the whole cluster; if a 2^nd failure occurs before ASM rebalance has completed, then the DB may lose data and crash. In high redundancy configuration for the disk groups, 2 disk failures can be survived before ASM rebalance re-estabilishes redundancy for the whole cluster; if a 3^rd failure occurs before then, the DB may lose data and crash. While the statistical chance of a 2^nd disk failure is very low, the consequences are severe in normal redundancy mode. Redudancy configuration is a trade-off between higher availability for mission-critical and business-critical systems, vs. higher capacity disk groups available for data storage, and should be chosen according to individual customer need.

/opt/oracle.SupportTools/sundiag.sh is a utility used to collect data for Exadata service requests, and in particular contains data specific for diagnosing disk failures. The version in the Exadata software image may not be the latest. For more details and the latest version, refer to Note 761868.1. Each of the examples below are from outputs collected by sundiag.

Errors for which Disk Replacement is Recommended:

Case 1. Cell's alerthistory reports the drive has changed its SMART status to "Predictive Failure":

This indicates the drive has determined via SMART that it is predicting it will fail and a SR should be opened for a replacement as soon as is convenient, and a sundiag attached for data analysis purposes.

If the system is connected to Oracle via Automatic Service Request, then a SR will automatically be opened for this event.

Storage cell disks can be replaced by the customer using the onsite spare provided, if the customer chooses, or Oracle will send out an engineer with the disk. 

Case 2. Cell's alerthistory reports the drive lun has experienced a critical error for which it cannot recover from:

This indicates the drive has experienced a critical error during the transaction causing the RAID HBA to mark the volume as critical. This often occurs concurrently with Predictive Failure in Storage cells where each volume is a single-disk RAID0, but may occur by itself in the event of a problem writing. A SR should be opened for a replacement as soon as is convenient, and a sundiag attached for data analysis purposes.

If the system is connected to Oracle via Automatic Service Request, then a SR will automatically be opened for this event.

Storage cell disks can be replaced by the customer using the onsite spare provided, if the customer chooses, or Oracle will send out an engineer with the disk.

Internal Note: Review sundiag's for all disk failures, and look for symptoms of Note 1360343.1 Issues 50 and 65 for which disks are requested to be CPAS'd.

Issue 50 describes a scenario where the disk drive appears to be hot-removed and hot-inserted by itself on the RAID0 volumes on the storage cell within a few seconds of each other, where it would not be physically possible to remove and replace a drive that quickly. It may also be at a time when the customer was not at the data center. This will be evident in the cell alerts and LSI HBA firmware logs which appears as a disk remove followed by a disk insert event. This may offline the volume in addition to causing SCSI I/O error messages from the kernel and may have ASM write errors evident.

Issue 65 describes a scenario where the disk records a critical failure and when parsing the megacli event logs, there are lots of command timeouts which end in "Error 02" when the disk is marked failed.

Case 3. DB node's where the Megacli status is shown as "Firmware state: (Unconfigured Bad)" preceded by logged errors indicating the drive was Failed or Predictive Failed:

The above command output files are gathered by a sundiag.

Since DB node's have RAID5 with a hotspare, when a disk fails, the hotspare will turn on and the RAID will be rebuilt, so the volume status will be Degraded temporarily and then return to Optimal. This will be evident in the Megacli logs, however may not be obvious to an operator without analysis. A failed disk can be verified by collecting a sundiag output, and a SR should be opened for analysis.

Case 4.  DB node's where the "Predictive Failure Count" is >0 even if the drive status shows as "Online".

The above command output files are gathered by a sundiag.

In this case, the hotspare has not turned on due to an incorrect MegaRAID setting. To force the hotspare to turn on at the next failure, do the following:

If replacing the drive before the next failure, then hot-plug remove it and wait for the controller to start copyback to the hotspare due to the missing disk.

Case 5. Storage Cell's where the drive cell status is "Warning" and Megacli status is "Firmware State: (Unconfigured Bad)". The Cell's alerthistory may report the drive with a "not present" alert.

This case may occur when the drive fails during a boot cycle, before the Cell's management services are running so the Cell does not see it go offline, only that its no longer present in the OS configuration. This will be evident in the Megacli logs, however may not be obvious to an operator without analysis. A failed disk can be verified by collecting a sundiag output, and a SR should be opened for analysis.

Storage cell disks can be replaced by the customer using the onsite spare provided, if the customer chooses, or Oracle will send out an engineer with the disk. In this case, follow the procedure for Storage Cell disks in Predictive Failure status.
 

Errors for which Disk Replacement is NOT Recommended:

Case 1. The Media Error counters reported by MegaCli in PdList or LdPdInfo outputs in a sundiag. On Storage Servers, these are also reported by Cellsrv in the physical disk view:

The above command output files are gathered by a sundiag.

These are counters of how many times a single disk I/O transaction has experienced an error, and are not indicative of the health of a disk and its ability to keep operating. They are not thresholds of SMART and do not have any kind of specific threshold that can be used to determine if a disk needs replacement or not. On earlier Exadata image versions, some of these errors may have been generated during a Patrol Scrub operation which does a verify of all the blocks including those on the disk which have not been used by ASM yet. These may or may not cause a problem in the future, so should be left until ASM does use them and can manage any data and errors on them.

Errors being counted here should be ignored until such time as a disk SMART asserts it is critical or predicted failure and the RAID HBA will offline the disk, send an alert and change the "status" field accordingly.

If multiple disks in different cells are having errors counted, there is a possibility of multiple disks going to failure at the same time, as described above. Greater diligence should be taken to monitor for and replace each predicted or critical failure in a timely manner.

Case 2.  The Other Error counters reported by MegaCli in PdList or LdPdInfo outputs in a sundiag. On Storage Servers, these are also reported by Cellsrv in the physical disk view:

The above command output files are gathered by a sundiag.

These are counters of how many times a single disk I/O transaction has experienced a SCSI transaction error, and are most likely caused by a data path problem. On Exadata, this could be due to the RAID HBA, the SAS Cables, the SAS Expander, the Disk Backplane or the Disk. On occasion they may also cause a Disk to report as 'critical' due to a timeout responding to an I/O transaction or some other unexpected sense returned, and replacing the disk in this case most likely will not resolve the problem. In many cases these data path problems may appear on multiple disks which would indicate something other than the disk is faulted.

In the example shown, all the disks have had data path errors. Slot 2 disk had some corrected read errors as a side-effect of the data path errors that are not critical, hence status is normal and therefore this disk does not match the criteria outlined above that requires replacement. One of those data path errors has triggered Slot 8 to change to critical, although it has not shown any media errors. Disk replacements of slot 8 did not resolve this problem. Data analysis of the full history of the errors and their types identified the problem component to be the SAS Expander.

A sundiag output should be collected and a SR opened for further analysis to determine the source of the fault.

Case 3.  ASM logs on the DB node show I/O error messages in *.trc files similar to:

That may also be accompanied by ASM recovery messages such as this:

This is a single I/O Error on a read, which ASM has recovered and corrected. There are other similar ASM messages for different types of read errors.

This will likely also have a matching entry in the Storage Server's alert.log such as this:

This may also generate a Storage Server entry in /var/log/messages such as this:

and will probably match an entry in the RAID HBA logs gathered by sundiag.

These single errors are recoverable using the built-in redundancy of ASM so ASM will initiate re-write of the block that had the error, using the mirrored copy which allows the disk to re-allocate data around any bad blocks in the physical disk media. The disk should not be replaced until the failures are such that they trigger predictive failure or critical cell alerts.

Case 4. Oracle Enterprise Manager users of the Exadata plug-ins may see alerts marked "Critical" for all I/O errors.

Since read errors are correctable and not truly critical, this may be a false report. A sundiag output should be collected and a SR opened for further analysis to determine if the fault is critical or not that requires replacement.

A request for enhancement has been filed for EM to separate non-critical and critical write errors in EM.

Case 5.  A disk with Firmware status "Unconfigured(good)".

This is an indication the disk is good, but not configured into a RAID volume. This is not an expected status in Exadata except during transition periods after a replacement if something does not work correctly during a replacement.

In Storage Servers in particular, this may be an indication that the disk was replaced but the Management Service (MS) daemon did not function properly and did not create the RAID volume and subsequent cell and grid disks. Refer to Note 1281395.1 and Note 1312266.1 for more details.

Conclusion:

Any other disk or I/O errors for which a disk may be suspect for such as device not present, device missing or timeouts, should have a sundiag output collected and a SR opened for further analysis to determine if the fault is critical.

Any doubts or concerns about disk or I/O errors listed above, then a sundiag output should be collected and a SR opened for further analysis to determine whether action is necessary.

References

This solution has no attachment