Troubleshooting Sun Storage[TM] 6580/6780 Cache Memory Backup Device Faults

Asset ID:	1-75-1120693.1
Update Date:	2010-07-01
Keywords:

Solution Type Troubleshooting Sure

Solution 1120693.1 : Troubleshooting Sun Storage[TM] 6580/6780 Cache Memory Backup Device Faults

Applies to:

Sun Storage 6780 Array - Version: Not Applicable and later   [Release: NA and later ]
Sun Storage 6580 Array - Version: Not Applicable and later    [Release: NA and later]
Sun Storage Common Array Manager (CAM) - Version: 6.2 to 6.5   [Release: 6.2 to 6.5]
Information in this document applies to any platform.

Purpose

This document is intended to provide a basic overview on how to troubleshoot faults with the array RAID controller cache backup device.

Please validate that each troubleshooting step below is true for your environment. Each step will provide instructions via a link to a document, for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Please do not skip a step.

Symptoms:

Critical Fault for Cache Backup Device Failed(xx.66.1251)
Array Log Events for Cache Backup Device (Event 0x7500)
Cache Backup device shows as Failed in the user interface
Cache Backup device shows as Unknown in the user interface
Controller Seven Segment Display shows OS+ CF+ F#+ blank-

Last Review Date

June 9, 2010

Instructions for the Reader

A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.

Troubleshooting Details

Step 1) Verify the existence of a critical fault for Cache Backup Device Failed Critical Fault

Reference <Document: 1021057.1> Verify Sun StorageTek[TM] 2500 and Sun Storage[TM] 6000 Critical Faults via the User Interface

If there is no critical fault, it is unlikely that a backup device has completely failed, but may require additional checks. Continue to Step 2.
If there is a critical fault, go to Step 5.

Step 2) Verify the 7-segment display on the array controller module.

Another indicator of a Cache Backup Device fault is the 7-Segment Display on the Array RAID Controller. This display also serves as the array controller module Tray ID indicator. It is located in the rear(cable side) of the tray. For more information on the 7-segment display:

Reference <Document: 1021110.1> Sun Storage[TM] 6180, 6580, and 6780 Array Controller 7-Segment Display.

If the display shows a repeating pattern:

OS+ CF+ F#+ blank-

Note: The hash(#) symbol represents a number 1 through 4 which represents the faulty cache backup device slot.

Go to Step 5.

If the display shows another repeating pattern other than that shown above, the problem is not a Cache Backup Problem, but something else with the controller. Please reference <Document: 1021113.1> Troubleshooting Sun StorageTek[TM], Sun StorEdge[TM], and Sun Storage[TM] RAID Controller Failures
If the display shows the tray ID(defaults to 99) of the module, OR you are unable to confirm the display, continue to Step 3.

Step 3) Verify the state/status of the Cache Backup Device on your controller

Follow the instructions below to look at the state and status of your Cache Backup Devices, based on your user environment:

Sun StorageTek Common Array Manager:

Browser:

Expand Storage Arrays in the left menu pane
Expand your storage array name in the left menu pane
Expand Troubleshooting in the left menu pane
Click on FRUs
In the right display pane, click on Cache Backup Device

SSCS CLI:

sscs list -d array_name -t cacheBackup fru

Sun StorageTek SANtricity Storage Manager:

GUI:

Launch SANtricity
Double Click on your array name to open the Array Management Window
In the left pane click on the controller icon for the controller DIMMs you want to view.
In the right pane the Cache Backup Device status, for each slot, will be listed after the base controller information.

SMcli:

SMcli -n <array name> -c "show storageArray profile;"

NOTE: The Cache Backup Device information will be listed in the CONTROLLERS section of the resultant SMcli output.

If your Backup Device is in an Enabled/OK/Optimal state/status, continue to Step 4
If your Backup Device is in a Failed state or status, go to Step 5.
If your Backup Device is in an Unknown status, and the Controller is ONLINE, go to Step 6.

NOTE: The Cache Backup Device information will be Unknown or Unavailable if the controller it resides on is currently OFFLINE. This applies for all management software.

Step 4) Verify existence of Cache Backup Device Failed messages in array event logs

The Cache Backup Device can have a handful of messages

Sun StorageTek Common Array Manager:

Browser

Expand Storage Arrays in the left menu pane
Expand your storage array name in the left menu pane
Expand Troubleshooting in the left menu pane
Click on Events
In the right pane, click on the -|-> icon. If you mouse over it it will state Advanced Filter.
Set Event to Log Events
Set Event Type to Component
Set Read the last X Kbytes From Log File to 100
Set String Filter to CacheBackupDevice
Click on the Details of any alarm that is shown.
Review the Description Field
Get the value of the array log event ID from the description

Note: The filter in item "9." above is case sensitive.

Example: Description : Apr 08 21:31:31 6780-array Tray.99.Controller.A.CacheBackupDevice.USB3: [ID 0x1901] NOTICE: Cache memory dimm is missing

SSCS CLI:

Get the list of events:

sscs list -d array_name -t LogEvent -f CacheBackupDevice event

Note: The -f option's argument is case sensitive.

Get the event details:

sscs list -d array_name event event_id

Get the value of the array log event ID from the description:

Example: Description : Apr 08 21:31:31 6780-array Tray.99.Controller.A.DIMM01: [ID 0x7500] NOTICE: Persistent cache backup device has failed

SANtricity Storage Manager

GUI

Launch SANtricity
Double Click on your array name to open the Array Management Window
Click on the Advanced Menu
Click on the Troubleshooting Sub-Menu
Click on View Event Log
Un-Check View Only Critical Events
Click on the Component Type field header to sort the events
Look for Cache Backup in the list of events
For any Cache DIMM event, highlight it, and check the View Details box
Get the value of the Event type field for each DIMM event.

SMcli

Get the list of events by saving off the event log.

SMcli -n array_name -c "save storageArray allEvents file=\"some/file/path/log.txt\";"

Open a text viewing application to look at the individual events.
Get the value of the Event type field for each Cache Backup event.

Example Event Date/Time: 6/8/10 21:52:00 ET Sequence Number: 12345 Event Type: 7500 Description: Persistent cache backup device has failed

If there is the existence of an event log ID of 0x7500, go to Step 5.
If there are no 0x7500 Event ID's, and your array controller is Online/OK, you have verified that the Cache Backup Device is working optimally. No further work is required.

Step 5) Open a Service Call with Oracle to have the Cache Backup Device indicated replaced

The Cache Backup Device slot for the controller is indicated in either the 0x7500 Event, the Seven Segment Display, or the critical fault, is failed, and requires replacement. No further troubleshooting is necessary.

Please open a service call with Oracle with:

Support Data Collection

Reference <Document: 1002514.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] Common Array Manager

Reference <Document: 1014074.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] SANtricity Storage Manager

Cache Backup Slot location
Array Critical Faults
Array Event Log
Seven Segment Display Code cited.

Step 6) Open a Service Call with Oracle to for further research

Please provide a Support Data Collection:

Reference <Document: 1002514.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] Common Array Manager

Reference <Document: 1014074.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] SANtricity Storage Manager

Attachments

This solution has no attachment