Asset ID: |
1-73-1307357.1 |
Update Date: | 2012-06-20 |
Keywords: | |
Solution Type
FAB (standard) Sure
Solution
1307357.1
:
A SIM card in a JBOD Diskshelf attached to an S7000 at f/w 3420 while its peer SIM card is at f/w 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".
Related Items |
- Sun Storage 7310 Unified Storage System
- Sun Storage 7410 Unified Storage System
- Sun ZFS Storage 7120
- Sun ZFS Storage 7320
- Sun ZFS Storage 7420
|
Related Categories |
- PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: Sun FAB
- .Old GCS Categories>Sun Microsystems>Sun FAB>Standard>Reactive
|
In this Document
Oracle Confidential (PARTNER). Do not distribute to customers.
Reason: FABs available to Internals and Partners only
Applies to:
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7320 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7420 - Version Not Applicable to Not Applicable [Release N/A]
Information in this document applies to any platform.
__________
SUNBUG: 7020272
Affected Parts:
542-0194 - Disk Shelf 24x3.5" SAS-2 Interface Module (SIM)
Symptoms
When a SIM with 3520 is in the same JBOD with a SIM that has 3524, the Appliance BUI will not detect that SIM and all associated devices down that SIM's path will not go online.
An example of out put would look like this.
From the appliance CLI this situation may look like the SIM is absent:
pt7420-020:maintenance hardware> list
NAME STATE MANUFACTURER MODEL SERIAL
chassis-000 pt7420-020 ok Oracle Sun ZFS Storage 7420 1027DF0013
chassis-001 1031QAQ03A ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ03A
chassis-002 1031QAQ030 ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ030
chassis-003 1031QAQ042 faulted Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ042
chassis-004 1020QAQ007 ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1020QAQ007
chassis-005 1031QAQ029 ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ029
chassis-006 1031QAQ021 ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ021
chassis-007 1020QAQ01A ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1020QAQ01A
chassis-008 1031QAQ022 ok Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ022
pt7420-020:maintenance hardware> select chassis-003 select slot show
Slots:
LABEL STATE MANUFACTURER MODEL SERIAL
slot-000 SIM 0 ok Sun Microsystems, Inc. 3753633 2029QTF-1004QC1271
slot-001 SIM 1 absent Sun Microsystems, Inc. - -
There will likely be a problem reported with the JBOD:
pt7420-020:maintenance hardware>
pt7420-020:maintenance problems> show
Problems:
COMPONENT DIAGNOSED TYPE DESCRIPTION
problem-000 2011-3-9 14:41:17 Major Defect The JBOD '1031QAQ042' is
configured incorrectly
pt7420-020:maintenance problems> select problem-000 show
Properties:
uuid = 8c16786a-04e4-c7f9-e9b1-f4c059c4edd9
code = AK-8000-H7
diagnosed = 2011-3-9 14:41:17
phoned_home = never
severity = Major
type = Defect
url = http://sun.com/msg/AK-8000-H7
description = The JBOD '1031QAQ042' is configured
incorrectly.
impact = The disks contained within the enclosure
cannot be used as part of a storage pool.
response = None.
action = Correct the disk configuration to conform to a
valid configuration.
Components:
component-000 100% 1031QAQ042: hc://:product-id=SUN-Storage-J4410:chassis-id=1031QAQ042:serial=1031QAQ042:part=Storage-J4410:revision=3524/ses-enclosure=2 (degraded)
Manufacturer: Sun Microsystems, Inc.
Part number: Storage-J4410
Model: Sun Disk Shelf (SAS-2)
Revision: 3524
pt7420-020:maintenance problems>
Impact
The Diskshelf JBOD will be running in a degraded state.
Having a SIM card in a JBOD Diskshelf attached to an S7000 appliance at firmware revision 3420 while its peer SIM card is at firmware revision 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".
It will also show that the SIM card with 3520 FW loaded on it is reported as "absent", and the "maintenance problems> show" will indicate the that "The JBOD 'xxxxxxxxxx' is configured incorrectly". In certain conditions the entire JBOD will not be noticeable from the "HW View".
Changes
Contributing Factors
There are 3 Scenarios leading to how the firmware inconsistency could happen:
1) At the time of an installation or initial upgrade to Q3.1.1 the installation/upgrade
is interrupted leaving the SIM cards in different states.
2) A FRU replacement that has FW 3520 on it is installed into a chassis where
the other SIM card is already running Q3.1.1 (FW 3524).
3) A Customer is running at a version of AK build prior to Q3.1.1 and a SIM FRU
is swapped into their JBOD that has 3524 on it.
Cause
Root Cause
The cause of this is the incompatibility between SIM FW 3520 and 3524.
The formal fix is To Be Determined. Running the sesupgrade command from the Solaris Shell is the workaround for now.
Solution
Workaround
Upgrading the SIM with FW 3520 to FW 3524 using the manual sesupgrade command will alleviate this incompatibility.
IMPORTANT! It is imperative that all customers upgrade their Amber Road appliance to a minimum level of Q3.1.1 (aka 2010.08.17.1.1) so that this issue will not reoccur.
Resolution
Note: The following should only be performed by trained Oracle service personnel.
To rectify this problem, if the customer is not already at Q3.1.1, upgrade their appliance first.
IMPORTANT: WAIT FOR ALL HARDWARE UPDATES TO FULLY COMPLETE BEFORE PROCEDING WITH THIS FAB!
Check the versions of your system, including the SIM FW in the Diskshelf. If you still have the FW mismatched between SIMs, execute the rest of this FAB.
To manually upgrade the SIMs in your system, perform the following steps:
1-1) drop into the Solaris shell, identify that there is a downlevel SIM, then manually update.
pt7420-020:maintenance problems> confirm shell
Executing raw shell; "exit" to return to appliance shell ...
+-----------------------------------------------------------------------------+
| You are entering the operating system shell. By confirming this action in |
| the appliance shell you have agreed that THIS ACTION MAY VOID ANY SUPPORT |
| AGREEMENT. If you do not agree to this -- or do not otherwise understand |
| what you are doing -- you should type "exit" at the shell prompt. EVERY |
| COMMAND THAT YOU EXECUTE HERE IS AUDITED, and support personnel may use |
| this audit trail to substantiate invalidating your support contract. The |
| operating system shell is NOT a supported mechanism for managing this |
| appliance, and COMMANDS EXECUTED HERE MAY DO IRREPARABLE HARM. |
| |
| NOTHING SHOULD BE ATTEMPTED HERE BY UNTRAINED SUPPORT PERSONNEL UNDER ANY |
| CIRCUMSTANCES. This appliance is a non-traditional operating system |
| environment, and expertise in a traditional operating system environment |
| in NO WAY constitutes training for supporting this appliance. THOSE WITH |
| EXPERTISE IN OTHER SYSTEMS -- HOWEVER SUPERFICIALLY SIMILAR -- ARE MORE |
| LIKELY TO MISTAKENLY EXECUTE OPERATIONS HERE THAT WILL DO IRREPARABLE |
| HARM. Unless you have been explicitly trained on supporting this |
| appliance via the operating system shell, you should immediately return |
| to the appliance shell. |
| |
| Type "exit" now to return to the appliance shell. |
+-----------------------------------------------------------------------------+
1-2) pt7420-020# sesupgrade /dev/scsi/ses/*
device: /dev/scsi/ses/c0t500163600024BE7Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ03A
current status: none
device: /dev/scsi/ses/c0t500163600024CC7Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ03A
current status: none
device: /dev/scsi/ses/c0t500163600025643Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ030
current status: none
device: /dev/scsi/ses/c0t5001636000256CFDd0
vendor: SUN
product: Storage J4410
revision: 3520
serial: 1031QAQ042
current status: none
device: /dev/scsi/ses/c0t50016360002572FDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ042
current status: none
device: /dev/scsi/ses/c0t50016360002576FDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1020QAQ007
current status: none
device: /dev/scsi/ses/c0t5001636000257DBDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ029
current status: none
device: /dev/scsi/ses/c0t500163600025803Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1020QAQ007
current status: none
device: /dev/scsi/ses/c0t50016360002590BDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ021
current status: none
device: /dev/scsi/ses/c0t50016360002593BDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ021
current status: none
device: /dev/scsi/ses/c0t5001636000259DFDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ029
current status: none
device: /dev/scsi/ses/c0t5001636000259EBDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ030
current status: none
device: /dev/scsi/ses/c0t500163600026423Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1020QAQ01A
current status: none
device: /dev/scsi/ses/c0t500163600026433Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1020QAQ01A
current status: none
device: /dev/scsi/ses/c0t500163600026633Dd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ022
current status: none
device: /dev/scsi/ses/c0t50016360002664FDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ022
current status: none
pt7420-020#
Verify the specific device:
1-3) pt7420-020# sesupgrade /dev/scsi/ses/c0t5001636000256CFDd0
device: /dev/scsi/ses/c0t5001636000256CFDd0
vendor: SUN
product: Storage J4410
revision: 3520
serial: 1031QAQ042
current status: none
pt7420-020#
Change to the firmware repository and manually update the SIM:
1-4) pt7420-020# cd /usr/lib/ak/fw/SUN-Storage-J4410-3524/
1-5) pt7420-020# ls
J4410.3524.fw manifest.xml plugin.so
1-6) pt7420-020# sesupgrade -i J4410.3524.fw /dev/scsi/ses/c0t5001636000256CFDd0
device: /dev/scsi/ses/c0t5001636000256CFDd0
vendor: SUN
product: Storage J4410
revision: 3520
serial: 1031QAQ042
current status: none
downloading 424224 bytes: ok
status: completed (available)
Wait 3 minutes for the upgrade to complete and the SIM to reset and then verify the upload was successful:
1-7) pt7420-020# sesupgrade /dev/scsi/ses/c0t5001636000256CFDd0
device: /dev/scsi/ses/c0t5001636000256CFDd0
vendor: SUN
product: Storage J4410
revision: 3524
serial: 1031QAQ042
current status: none
NOTE: Please be sure to wait 3 minutes between upgrades of each SIM.
Please repeat steps 1-1 through 1-7 for each SIM this has the mismatched FW residing on it.
**If the bui or cli does not show correct jbod information for any of the effected jbods it might require a restart of akd** If so please do the following from the appliance cli:
2-1) mall-maguro1:> maintenance
2-2) small-maguro1:maintenance> show
2-3) small-maguro1:maintenance> system
2-4) small-maguro1:maintenance system> restart
2-5) This will restart the management system. Are you sure? (Y/N)
At this point you will experience temporary loss of the bui or management cli (approximately 2 minutes). Once the akd restarts please check to make sure proper information about the affected jbod is now displayed (correct number of paths, correct sim firmware etc).
If for any reason the SIM seems to fail the reboot after the firmware update you might have to reseat the SIM card to do a cold reboot. After this you need to check the firmware version and maybe have to reapply the manual update.
Identification of Affected Parts (how to):
Once the akd restarts please check to make sure proper information about the affected jbod is now displayed (correct number of paths , correct SIM firmware etc. Access the SIMs via the BUI after insertion of the SIM to asses if the BUI can properly read that SIM and FW.
Contacts:
Contributor: [email protected]
Responsible Engineer: [email protected]
Responsible Manager: [email protected]
Business Unit Group: NWS (Network Storage)
For information about FAB documents, its release processes, implementation strategies and billing information, go to the following URL:
https://sunspace.sfbay.sun.com/display/Onestop/FAB%20(Field%20Action%20Bulletin)
In addition to the above you may email:
* [email protected]
Attachments
This solution has no attachment