Sun Storage 7410 Unified Storage System
 
Sun ZFS Storage 7320
 
Sun Storage 7310 Unified Storage System
 
Sun Storage 7210 Unified Storage System
 
Sun Storage 7110 Unified Storage System
 
Sun ZFS Storage 7420
 
Sun ZFS Storage 7120
 

PLA-Support>Sun Systems>DISK>NAS>SN-DK: 7xxx NAS
 
.Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
 

Applies to:

Purpose

The purpose of this document is to troubleshoot disk drive failures on a Sun Storage 7000 ZFS Appliance.

NAS head revision : not dependent
BIOS revision : not dependent
ILOM revision : not dependent
JBODs Model : J4400|J4410|J4500
CLUSTER related : not dependent

Troubleshooting Steps

1. Verify the Problems list from the Appliance

To aid serviceability, the appliance detects persistent hardware failures (faults) and software failures (defects, often included under faults) and reports them as active problems on this screen. If the phone home service is enabled, active problems are automatically reported to Oracle where a support case may be opened depending on the service contract and the nature of the fault.

From support bundle

cat /fm/fmadm.out

From CLI

maintenance problems show

From BUI

1. Click Maintenance
2. Click System
3. Click Problems

 The list below is a list of the descriptive text and the error code found on the system, as they may be seen in the customer report. Review the list shown by the appliance against the one below:

If there are no problems in the list, the drives in the Appliance are ok.  No further work is required.

• If there are more than one of the above problems in the list, go to Step 2.
• If the problem is for ZFS-8000-FD, ZFS-8000-0X, DISK-8000-4Q, the drive is failed or is about to fail, and requires replacement.  Please contact Oracle for further support.
• If the problem is for ZFS-8000-GH, go to Step 3.
• If the problem is for AK-8000-F0, go to Step 4.
• If the problem is for ZFS-8000-D3, go to Step 5.

2.  Are the problems all ZFS-8000-GH?

If so, go to Step 3.
If not, please contact Oracle for further support.

3.  Run a scrub to attempt to correct the errors. 

The ZFS subsystem can generate erroneous checksum errors on the system as part of a disk replacement or normal day to day action.  This does not require a disk replacement unless the
data checksum turns into an unrecoverable action (this does not cause data loss).  The scrub may flag additional drives as failed due to checksum errors, but will never fail enough drives to offline the storage pool.

1. Run the scrub to completion
   - BUI: Configuration -> Storage -> Click on the pool -> Click Scrub
   - CLI: configuration storage scrub start
2. Mark the check sum fault as cleared in the problems log
   - BUI: Maintenance ->Problems -> Click on Problem->Click Marked Repaired
   - CLI: maintence problems select <problem-id> markrepaired
3. Repeat this process until there are no checksum errors or there is an Unrecoverable Problem logged by the system.

Contact Oracle if there are unrecoverable errors generated during this process.

4.  Verify Part is in the Correct Enclosure

AK-8000-F0 is an indication that the disk is not compatible with the enclosure, i.e. a SAS-1 drive in a SAS-2 enclosure. Unfortunately, the error is often spurious, and can disappear with after reseating a drive. This fault will almost exclusively be seen on 7410 and 7420 systems.

Please reseat the drive in question and mark the problem as cleared.  If the problem returns, please review your configuration.  There may be a SAS-1 or SAS-2 system in the wrong enclosure.

5.  Verify your system release for whether CR 6999699 has been fixed on your system AND whether you have replaced a disk drive recently

<SUNBUG: 6999699> This issue is due to the creation of multiple entries for drives in the storage pools.  This has been resolved in the 2010Q3.4.0 release

Get the AK release:

• BUI->Click on the Sun/Oracle symbol in the upper left corner
• CLI->maintenance system updates show

The release will be of format:

If you are at a version prior the release listed above and you have replaced a disk drive recently, please contact Oracle for further support.

In case the above steps do not provide a conclusion about the disk problem you should collect a full support bundle and upload it to Oracle or attach it to a Service Request you have open for this problem.

6.  Check debug.sys to see if there were several command timeouts for which the disk was offlined.

Get the drive serial number from the problems list:

From ./fm/fmadm.out

From ./logs/debug.sys

if timeouts exist && consistent step 7
if timeouts exist && intermittent step 9
if timeouts do not exist the drive should be replaced(unless replaced previously on the system).

7. Verify whether the drive is in the head or expansion tray

From the fault in fmadm.out.  Review whether the drive is in the appliance head, versus the expansion tray:

If the device is in the head, go to Step 9.
If the device is in an expansion as denoted by product-id=SUN-Storage-J4400 or :product-id=SUN-Storage-J4410:server-id=: go to Step 8

8) Verify the status of the SIMs/SIM paths

Look for the chassis serial number found in ./fm/fmadm.out:

Now look for that same number in the ./hw/fmtopo.txt.  The serial number should be listed TWICE, once
for each SIM.  

In this string we are only looking for the whether the revision is listed or not for this entry, as shown in bold above.
This should be shown for both SIMs.  The controller value in the output above indicates the SIM slot (0 = left, 1 = right)

If both SIMs show the revision in fmtopo.txt, then go to Step 9
If one SIM does not show a revision, then

1. reseat the SIM
2. reseat the drive
3. verify fm fault is cleared

9.  Check whether debug.sys shows other drives impacted by timeouts

Get the drive serial number from the problems list:

Note the times of the timeouts

Do disconnected command timeouts exist for other Drive serial numbers at the same times as those
for the faulted drive?

if so collaborate with L2
if not go to step 10

10.  Check for any other failed system components in hw.aksh

Perform a quick review of component status in hw.aksh for any other faulty part status.

if so collaborate with L2
if not have the drive replaced

This solution has no attachment