Document Audience:INTERNAL
Document ID:A0209-1
Title:Sun Fire 15K & Sun Fire 12K with Crystal+ cards may experience panics in the pcisch driver.
Copyright Notice:Copyright © 2007 Sun Microsystems, Inc. All Rights Reserved
Update Date:Fri Jun 13 00:00:00 MDT 2003

----------------------------------------------------------------------------
             - Sun Proprietary/Confidential: Internal Use Only -
----------------------------------------------------------------------------

                             FIELD CHANGE ORDER
            (For Authorized Distribution by Enterprise Services)

FCO #: A0209-1
Status: inactive
Synopsis: Sun Fire 15K & Sun Fire 12K with Crystal+ cards may experience panics in the pcisch driver.
Date: Jun/13/2003
SunAlert: No
Top FIN/FCO Report: No
Products Reference: Sun Fire 15K/12K
Product Category: Server / System Component
Product Affected: 
Mkt_ID     Platform     Model    Description          Serial Number
------	   --------     -----    -----------	      -------------
  -         F15K          -      Sun Fire 15K               -
  -         F12K          -      Sun Fire 12K               -

X-Options Affected

Mkt_ID     Platform     Model    Description               Serial Number
------	   --------     -----    -----------	           -------------
X6727A     F15K/F12K      -      PCI Dual FC Network Adapter+    -
Parts Affected: 
Part Number     Description                             Model
-----------     -----------                             -----
375-3030-xx     PCI Dual FC Network Adapter+              -


(SCSI Devices)
Type   Vendor    Model     SerialNumber(Min)   SerialNumber(Max)   Firmware
----   ------    -------   -----------------   -----------------   --------
N/A
References: 
ESC: 537306
  FIN: IO852-1
  BugID: 4699182
Issue Description: 
Sun Fire 15K and Sun Fire 12K systems with PCI Dual FC Network Adapter+
(Crystal+) used in the 66MHz slots may experience pcisch driver panics
due to a parity error on the PCI Bus.

Panics in the pcisch driver cover a wide range of possible failures.
In this case, the control status register (CSR) calls out the detection
of bad parity on the PCI bus:

  WARNING: pcisch-19: PCI fault log start:
  PCI SERR
  PCI error occurred on device #0
  dwordmask=0 bytemask=0
  pcisch-19: PCI primary error (0):pcisch-19: PCI secondary error (0):pcisch-19:
       PBM AFAR 0.00000000:WARNING: pcisch19: PCI config space
       CSR=0xc2a0
  pcisch-19: PCI fault log end.

  panic[cpu128]/thread=2a10001fd20: pcisch-19: PCI bus 3 error(s)!

  000002a10001bea0 pcisch:pbm_error_intr+148 (30000b643d8, 2772, 30000b84548, 3,
        30000b643d8, 3)
    %l0-3: 00000300008b9860 0000000000004000 0000000000000000 0000030000b86584
    %l4-7: 00000300009978c8 0000030008d03ea8 0000000000000000 0000030008d03ed0
  000002a10001bf50 unix:current_thread+44 (0, ffffffffffffffff, 0, 300335b3528,
        0, 1044f340)
    %l0-3: 0000000010007450 000002a10001f061 000000000000000e 0000000000000016
    %l4-7: 0000000000010000 00000300339922a8 000000000000000b 000002a10001f910
  000002a10001f9b0 unix:disp_getwork+40 (1044e398, 0, 1044f340, 10457310, 2, 0)
    %l0-3: 000000001010e2d8 0000000010509e00 00000300335bd518 000002a100c37d20
    %l4-7: 000002a100cebd20 0000000002736110 0000000000000000 000002a10001f9c0
  000002a10001fa60 unix:idle+a4 (0, 0, 80, 1044e398, 3000096d980, 0)
    %l0-3: 0000000010043d58 2030205b275d2076 616c20696e646578 000002a10011dd20
    %l4-7: 70636220290a2020 202e22202073703a 20222031205b275d 2076616c20696e64

NOTE:  The stack itself can be different, depending on each specific case.  What
       matters is the CSR values (specifically the "detected-parity-error" bit).

Although this type of panic can result from a hardware issue on any adapter,
this FCO is only addressing those with a PCI Dual FC Network Adapter+.  In
addition, this FCO is only legitimate for failures in the 66Mhz slots (bottom
slots of an hsPCI assembly).

With every other panic of this nature, a hardware replacement has resolved
the case.  However, with one customer, repeated hardware replacements did not
resolve the issue.  The customer's issue has since been replicated on multiple
machines in an engineering environment.  There are some unique factors that
are needed to create this scenario:

  A. To date, this problem has only been seen on 375-3030 (Crystal+)
     cards.
  B. All the panics have been in either slot 0 or slot 2 of the I/O Boat.
     (Slots 0 and 2 is the lower 66 MHz slots)
  C. Schizo 2.3 seems to bring the problem out with more regularity.
  D. Veritas software (specifically adding mirrors to volumes) seems
     to increase the likelihood of failure.

Please review the Steps for Diagnosis in the Special Considersations
section below before implementing any corrective action.
Parts Affected: 
N/A
Implementation: 
---
|   |   MANDATORY (Fully Pro-Active)
 ---

 ---
|   |   CONTROLLED PRO-ACTIVE (per Sun Geo Plan)
 ---

 ---
| X |   UPON FAILURE
 ---
Replacement Time Estimate: 
4.0 hours
Special Considerations: 
Steps for Diagnosis
===================

 1)  Isolate the offending PCI bus:

 As a reminder, when looking at a starcat I/O boat, the slots
 are designated:

   |--------------------------|--------------------------|
   | Schizo 1, leaf B (33Mhz) | Schizo 0, leaf B (33Mhz) |
   |--------------------------|--------------------------|
   | Schizo 1, leaf A (66Mhz) | Schizo 0, leaf A (66Mhz) |
   |--------------------------|--------------------------|

   OR

   |--------|--------|
   | Slot 3 | Slot 1 |
   |   OR   |   OR   |
   | X.1.1.1| X.1.0.1|
   |--------|--------|
   | Slot 2 | Slot 0 |
   |   OR   |   OR   |
   | X.1.1.0| X.1.0.0|
   |--------|--------|

   NOTE: X = hsPCI number (0-17)

To diagnose the pcisch panic from the above stack, follow these steps:

  Use the /etc/path_to_inst on the domain or the cfgadm/rcfgadm commands
  to isolate the slot.  For example, using the two methods with the panic
  above (pcisch-19):

      # grep pcisch /etc/path_to_inst
      "/pci@3d,600000" 7 "pcisch"
      "/pci@1c,700000" 0 "pcisch"
      "/pci@3c,700000" 4 "pcisch"
  --> "/pci@9d,600000" 19 "pcisch"
      "/pci@9c,600000" 17 "pcisch"
      "/pci@3c,600000" 5 "pcisch"
      "/pci@5d,600000" 11 "pcisch"
      "/pci@7d,600000" 15 "pcisch"


     In this case, instance 19 is "/pci@9d,600000".  To
     translate that into a slot location, break down the 9d into
     binary <10011101>, then add spaces to obtain <100 1110 1>.
     That address now breaks down to slot 4 (100), skip the
     middle section (1110), pci 1 (or the pci slot on the
     left).

     The other option is to leverage the conversion the dynamic
     reconfiguration interface provides:

  # rcfgadm -d a -la | grep pcisch
  pcisch0:e00b1slot1      pci-pci/hp   connected    configured   ok
  pcisch10:e02b1slot3     unknown      connected    unconfigured unknown
  pcisch11:e02b1slot2     pci-pci/hp   connected    configured   ok
  pcisch12:e03b1slot1     pci-pci/hp   connected    configured   ok
  pcisch13:e03b1slot0     pci-pci/hp   connected    configured   ok
  pcisch14:e03b1slot3     unknown      connected    unconfigured unknown
  pcisch15:e03b1slot2     pci-pci/hp   connected    configured   ok
  pcisch16:e04b1slot1     unknown      connected    unconfigured unknown
  pcisch17:e04b1slot0     pci-pci/hp   connected    configured   ok
  pcisch18:e04b1slot3     unknown      connected    unconfigured unknown
     --> pcisch19:e04b1slot2     unknown      empty   unconfigured unknown
  pcisch1:e00b1slot0      unknown      empty   unconfigured unknown
  pcisch20:e08b1slot1     unknown      empty   unconfigured unknown
  pcisch21:e08b1slot0     pci-pci/hp   connected    configured   ok
  pcisch22:e08b1slot3     unknown      empty   unconfigured unknown
  pcisch23:e08b1slot2     unknown      empty   unconfigured unknown
  pcisch2:e00b1slot3      unknown      connected    unconfigured unknown
  pcisch3:e00b1slot2      pci-pci/hp   connected    configured   ok
  pcisch4:e01b1slot1      pci-pci/hp   connected   configured   ok
  pcisch5:e01b1slot0      unknown      empty   unconfigured unknown
  pcisch6:e01b1slot3      unknown      connected    unconfigured unknown
  pcisch7:e01b1slot2      pci-pci/hp   connected    configured   ok
  pcisch8:e02b1slot1      pci-pci/hp   connected    configured   ok
  pcisch9:e02b1slot0      unknown      connected    unconfigured unknown

     In this case, the issue is on expander 4 (ex4), I/0 board
     (b1), slot 2.

  b) Once the offending FRU has been identified, follow FIN IO852-1
and replace the hsPCI and the cassette called out in the panic.
Once completed, replace ALL x6272A's within the domain with
x6768A (Crystal2A), including x6727A that have not generated
panics.

     So, for the example above, we would replace the hsPCI in
     slot 4, the cassette in slot 2 (lower left), the x6727A
     with a x6768A and all other x6727A's in this domain.

     It is expected that some customers may wish to take the
     down time and replace all x6727A's in their entire platform
     where applicable.  This action has been approved under this
     FCO.

      EXCEPTION:  If customer attached A3500FC (540-4026 or 540-4027)
      to F12/15K via Crystal+, then x6799A (Amber) must be used in
      place of x6768A (Crystal 2A).

  c) There are some hardware prerequisites you might have to
  contend with:

  - The cables used for the x6768A differ from the cables
    used for the x6727A.  Before performing this FCO,
    verify and replace all required cables or use LC ->
    SC adapters.

 replace 537-1004 2 Meter SC-SC with 537-1035 2 Meter LC-SC

 replace 537-1020 5 Meter SC-SC with 537-1033 5 Meter LC-SC

 replace 537-1004 15 Meter SC-SC with 537-1034 15 Meter LC-SC

 If a custom length SC-SC cable is in use, order 0.4 Meter LC-SC
 cable 537-1036 and SC-SC Female to Female coupler 130-4723.


  d) There are some software prerequisites you might have to
  contend with:

  - Slot 1 DR will be available with SMS1.3.  The current
    target date (always subject to change) for SMS1.3 is
    sometime near the end of Janurary 2003.  Until then,
    the system will have to incur a downtime for
    replacement.

  - If the boot device is on the to-be-replaced hsPCI, it will be
    necessary to have planned the system configuration to
    allow a boot-device DR (i.e. multipathing/mirroring,
    etc).  If you do not have such a capability, the
    domain will have to incur a downtime for
    replacement.

  - Once the x6727A has been replaced with a x6768A, the
    controller number for the disk will change unless you
    follow the procedure below.

  - The customer will need to download the drivers for the x6768A.
    Reference the procedure below.  NOTE: DO NOT FORGET
    TO UPDATE/PATCH THE JUMPSTART IMAGE, IF APPLICABLE.

    At the time of authoring this FCO, the driver
    required according to:

      Sun StorEdge 2G FC PCI Dual Channel Network Adapter
      Product Notes (Part Number: Part No.816-5002-11
      June 2002, Revision A)

      Before installing the Sun StorEdge 2G FC PCI Dual
      Channel Network Adapter card, the host must have
      both the Solaris 8 update 4 operating environment
      release with the recommended patch cluster and the
      Sun StorEdge 2G FC PCI Dual Channel Network Adapter
      driver.

      Check http://www.sun.com/download/ or
      http://www.sun.com/storage/ san for updates.  There
      is one set of packages for the Solaris 8 operating
      environment and another for the Solaris 9 operating
      environment available under the respective links
      for the operating environments. The SUNWsan package
      is interchangeable between the releases.

      Packages:

      SUNWsan
      SUNWcfpl
      SUNWcfplx

      Available at: http://www.sun.com/download/ or
      http://www.sun.com/storage/san

      Patches (NOTE: patches might be uprev'ed.  These
      are the minimum requirements):

      Solaris 8  Solaris 9
      ---------  ---------
      Sun StorEdge Traffic Manager patch 111412-09  113039-01
      fctl/fp/fcp/usoc driver     111095-10  113040-01
      fcip driver   111096-04  113041-01
      qlc driver    111097-10  113042-02
      luxadm/liba5k and libg_fc patch    111413-08  113043-01
      cfgadm fp plug-in library patch    111846-04  113044-01
      SAN Foundation Kit patch    111847-04  111847-04

      Available at: sunsolve.sun.com

  - There is a known issue with replacing an crystal with
  an encapsulated boot device.
    If not donce correctly, device major number will be
    incorrectly set forcing panics.  Please reference the
    procedure below and specifcally the name_to_major
    file references.

  - Replacement Procedure:

  Replace Crystal+ cards with Crystal2A on a F15K/12K

==================================================================

  I. Prerequisites

    - Crystal2A drivers, patches and packages.
      [ refer to Sun StorEdge 2G FC PCI Dual Channel
      Network Adapter Product Notes ]

    - Solaris 8 2/02 with current recommended patch
    cluster and san patches.


    - Dedicated Solaris 8 2/02 network boot/jumpstart image with
      Crystal2A drivers, packages and patches.

    - A good backup of all filesystems.

==================================================================

  II. Preparation

  1. If you are replacing a controller that contains the
  boot device or a Veritas Volume Manager device in
  rootdg, you will have to create a boot server image
  that contains the Crystal2A drivers. Otherwise you may
  skip this step.


     To do this, first create a  Solaris 8 02/02
     JumpStart Boot server.  Then, install the Crystal2A
     drivers, patches and packages into this image and
     copy the /etc/name_to_major file from the domain
     onto the boot image. (This will prevent problems
     with differing major numbers between the domain and
     the boot image).

     Example using a Solaris 8 02/02 boot server image located at
    /jumpstart/5.8_HW202:

   For Packages:
   =============
   cd [ location of packages ]
   pkgadd -R /jumpstart/5.8_HW202/Solaris_8/Tools/Boot -d .

   For Patches:
   ============
   cd [ location of patches ]
   patchadd -C /jumpstart/5.8_HW202/Solaris_8/Tools/Boot
   ./[patchid]

   For /etc/name_to_major:
   =======================
   cd /jumpstart/5.8_HW202/Solaris_8/Tools/Boot/etc
   cp name_to_major name_to_major.orig
   ftp domain
   ftp> cd /etc
   ftp> get name_to_major

  2. Bring the domain down to single user mode

   OK> boot -s

  3. Install patches and packages on the domain.

   Follow normal patchadd and pkgadd procedures.

  4. Verify the controller number[s] for the card being replaced

   # format
   # ls -l /dev/dsk
   # ls -l /dev/ses
     ( You may want to save this output for reference. )

==================================================================

  III. Replacing a controller NOT used for the boot device.

  (This example uses c1 as the controller to be changed.)

  1. If Volume Manager is being used, disable it from starting.

 # touch /etc/vx/reconfig.d/state.d/install-db

  2. Reboot domain into single user mode

 # init 0
 OK> boot -s

     Volume manager should not be running at this point.

 # ps -ef | grep vx   (this should show no volume manager
  processes)

  3. Remove the devices associated with the controller to be
     replaced.

 # cd /dev/dsk
 # rm c1*

 # cd /dev/rdsk
 # rm c1*

 # cd /dev/cfg
 # rm c1 [ this entry may or may not exist ]

 # cd /a/dev/ses ( if applicable )
 # rm ses2 ses3  ( for the ses devices associated with c1 )

  4. Shutdown and replace the cards.  Make sure auto-boot? is
     false.

 # init 0
 OK> setenv auto-boot? false

     Shut off the domain from the SC

 setkeyswitch -d [domainid] off

     Replace the card, turn on the domain.

 setkeyswitch -d [domainid] on

     Verify that the new controller is available from OBP.

 OK> probe-scsi-all

     Do a single user reconfiguration boot

 OK> boot -sr

  5. Verify that the devices were created as expected:

 # format
 # ls -l /dev/dsk/c1* /dev/rdsk/c1*
 # ls -lL /dev/dsk/c1* /dev/rdsk/c1*
 # ls -l /dev/es

     If Veritas Volume Manager was NOT used, check that the
     devices can be mounted.  If all looks good continue to
     multiuser.

     # mountall

  6. If Veritas was disabled previously (Step 1), re-enable it and
     reboot.

 # rm /etc/vx/reconfig.d/state.d/install-db
 # init 6

     Verify that veritas started correctly and all volumes are
     available.

==================================================================

  IV. Replacing a controller that IS used for the boot device.

     (This example uses c0 as the controller to be changed.)

  1. If the boot disk is encapsulated, you must first
     unencapsulate the boot device.

     Reboot and verify that the boot disk has been successfully
     unencapsulated.

  2. If Volume Manager is being used, disable it from starting.

 # touch /etc/vx/reconfig.d/state.d/install-db

  3. Reboot domain into single user mode

 # init 0
 OK> boot -s

     Volume manager should not be running at this point.

 # ps -ef | grep vx   (this should show no volume manager
  processes)

  4. Shutdown and replace the cards.  Make sure auto-boot? is
     false.

 # init 0
 OK> setenv auto-boot? false

     Shut off the domain from the SC

 setkeyswitch -d [domainid] off

     Replace the card, turn on the domain.

 setkeyswitch -d [domainid] on

     Verify that the new controller is availble from OBP.

 OK> probe-scsi-all

  5. Boot from the Crystal-2A enabled JumpStart Boot server (as
     described under preparation.  Verify that the
     devices are   visible.

 OK> boot net -s  (from Crystal-2a patched jumpstart)

 # format

  6. Mount the boot device's / partition at /a.  Remove the
     previous controller's device nodes.

 # mount /dev/dsk/ /a
 # rm /a/dev/dsk/c0* /a/dev/rdsk/c0*
 # rm /a/dev/cfg/c0  (this may or may not exist)

 # rm /a/dev/es/ses0 /a/dev/es/ses1
     ( for the ses devices associated with c0 )

  7. Build and verify the new device nodes and reset-all.

 # devfsadm -r /a -p /a/etc/path_to_inst

 # ls -l /a/dev/dsk/c0* /a/dev/rdsk/c0*
 # ls -l /a/dev/dsk/c0* /a/dev/rdsk/c0*
 # ls -l /a/dev/es

 # umount /a
 # halt

 OK> reset-all

  8. Determine the new boot device path.

   The device path WILL change the Crystal2A has a different FW prom.

     For example, original boot device:

/pci@3d,600000/pci@1/SUNW,qlc@4/fp@0,0/disk@w220000203733433b,0:a
     New boot device:

/pci@3d,600000/SUNW,qlc@1/fp@0,0/disk@w220000203733433b,0:a

 OK> show-disks      (check for new card's device)
 OK> probe-scsi-all  (check that disk are visible)

  Verify the new path of the boot device and use nvunalias and
  nvalias to record it.

 OK> nvunalias [old-boot-device-alias]
 OK> nvalias [device-alias] [device-path]
 OK> setenv boot-device [device-alias]
 OK> setenv diag-device [device-alias]  (if desired)

  9. Boot off the new path into single user mode.  Verify that the
     devices were created as expected

 ok> boot -s

 # format

     If Veritas Volume Manager was NOT used, check that the
     devices can be mounted.  If all looks good continue
     to      multiuser.

 # mountall


  10. If Veritas was disabled previously (Step 1), re-enable it
   and reboot.

 # rm /etc/vx/reconfig.d/state.d/install-db
 # init 6

     Verify that Veritas started correctly and all volumes are
     available.
Corrective Action: 
Important! Troubleshoot pcisch driver panics as outlined above and
           in FIN I0852-1 and follow instructions outlined in the
           Special Considerations section.

  A. Replace all 375-3030-xx (Crystal+) cards with 375-3108-xx
     (Crystal-2A) cards in the affected domain.

OR

  B. If customer attached A3500FC (540-4026 or 540-4027) to F12/15K
     via Crystal+, replace all 375-3030-xx (Crystal+) cards  with
     375-3019-xx (Amber) cards in the affected domain.

Either action will require new drivers to be installed and LC-SC
or LC-LC Fibre Cables.  See Product Note 816-5002 for details:

   http://infoserver.central/data/816/816-5002/pdf/816-5002-11.pdf
Comments: 
Billing Type: 
Warranty: Sun will provide parts at no charge under Warranty
           Service. On-Site Labor Rates are based on how the
           system was initially installed.

 Contract: Sun will provide parts at no charge. On-Site Labor Rates
           are based on the type of service contract.

 Non Contract: Sun will provide parts at no charge. Installation by
               Sun is available based on the On-Site Labor Rates
               defined in the Price List.

--------------------------------------------------------------------------
Implementation Footnote: 
________________________

i)   In case of Mandatory FCOs, Sun Services will attempt to contact
      all known customers to recommend the part upgrade.

ii)  For controlled proactive swap FCOs, Sun Services mission critical
     support teams will initiate proactive swap efforts for their respective
     accounts, as required.

iii) For Replace upon Failure FCOs, Sun Services partners will implement
     the necessary corrective actions as and when they are required.

--------------------------------------------------------------------------

All released FINs and FCOs can be accessed using your favorite network
browser as follows:

SunWeb Access:
______________

* Access the top level URL of http://sdpsweb.Central/FIN_FCO/

* From there, select the appropriate link to query or browse the FIN and
  FCO Homepage collections.

SunSolve Online Access:
_______________________

* Access the SunSolve Online URL at http://sunsolve.Central/

* From there, select the appropriate link to browse the FIN or FCO index.

Internet Access:
_______________

* Access the top level URL of  https://spe.sun.com

--------------------------------------------------------------------------
General:
________

Send questions or comments to [email protected]

---------------------------------------------------------------------------
Statusinactive