Asset ID:	1-77-1019883.1
Update Date:	2011-11-03
Keywords:

---

Solution Type  Sun Alert Sure

Solution  1019883.1 :   Sun Blade 6000 Chassis Monitoring Module (CMM) or Sun Blade T6300 Server Module may Exhibit Erroneous "Failed", "Hot Insertion", and "Removal" Messages  


PreviouslyPublishedAs
248186

Bug Id
<SUNBUG: 6720809>, <SUNBUG: 6695705>, <SUNBUG: 6762599>

Product
Sun Blade 6000 Modular System
Sun Blade T6300 Server Module

Date of Resolved Release
07-Jan-2009

Sun Blade 6000 Chassis Monitoring Module (CMM) or Sun Blade T6300 Server Module may exhibit erroneous "Failed", "Hot Insertion", and "Removal" messages:

1. Impact

Sun Blade 6000 Chassis Monitoring Module (CMM) and Sun Blade T6300 Server Module may exhibit erroneous "Failed", "Hot Insertion", and "Removal" Messages.

2. Contributing Factors

This issue can occur on the following platforms:

• Sun Blade 6000 Chassis Monitoring Module (CMM)
  
• Sun Blade T6300 Server Module

Note 1: This issue only occurs with the following T6300 Blade Module part numbers:

• 541-2317-05 (or below)
• 541-2318-05 (or below)
• 541-2319-05 (or below)
• 541-2320-04 (or below)

To determine the T6300 Blade Module part number, visually inspect the motherboard.
If the blades cannot be inspected visually, the following procedure can be used to identify affected blade(s):

1. Access the SP of the Target blade

2. Type "shownetwork"

3. Set the SP to "ssqa" mode by typing the following:

    setsc sc_ssqamode true "xyz"

where "xyz" is the last nibble from the three last bytes from the mac address.

4. Get the "Four Eyes" firmware version:

    sc> i2cp 0x20 3 2 0 0 1.

If the above command returns a value == 12, the blade is affected by this issue.
If the command returns a value == 14, the blade is not affected.

Note 2: The error messages mentioned above are completely harmless and do not affect the systems performance, functionality or reliability.

Note 3:  This issue can occur in Sun Blade 6000 configurations with any population of  Sun Blade T6300 server modules.  They are most apparent when Sun Blade T6300 server modules are plugged into slots 6,7,8, or 9. This issue rarely occurs in Sun Blade 6000 configurations that are less populated with Sun Blade T6300 server modules when plugged into the lower slots.

Note 4: This issue can occur on the systems listed above irrespective of the OS/ALOM versions in use.

3. Symptoms

If the described issue occurs, messages similar to the following will be seen when the T6300 Service Processor (SP) is powered on:

Example of T6300 SP messages:

    SC Alert: PSU at MP/PS1 has been removed.
    SC Alert: PSU at MP/PS1 has been inserted.
    SC Alert: NEM at MP/NEM0 has been removed.
    SC Alert: PSU at MP/PS0 has been removed.
    SC Alert: PSU at MP/PS0 has been inserted.
    SC Alert: SYS_FAN at MP/FM3/FIN has FAILED.
    SC Alert: SYS_FAN at MP/FM3/FOUT has FAILED.
    SC Alert: SYS_FAN at MP/FM6/FIN has FAILED.
    SC Alert: SYS_FAN at MP/FM6/FOUT has FAILED.
    SC Alert: SYS_FAN at MP/FM7/FIN has FAILED.
    SC Alert: SYS_FAN at MP/FM7/FOUT has FAILED.
    SC Alert: NEM at MP/NEM1 has been removed.
    SC Alert: NEM at MP/NEM1 has been inserted.
    SC Alert: PSU at MP/PS0 has FAILED.
    SC Alert: PSU at MP/PS1 has FAILED.

Example of chassis CMM events:

    1319  Thu Jan  1 00:01:56 1970  Chassis  Action  major  Hot insertion of /CH/NEM1
    1318  Thu Jan  1 00:01:56 1970  Chassis  Action  major  Hot insertion of /CH/NEM0
    1317  Thu Jan  1 00:01:56 1970  Chassis  Action  major  Hot insertion of /CH/PS1
    1316  Thu Jan  1 00:01:56 1970  Chassis  Action  major  Hot insertion of /CH/PS0
    1315  Thu Jan  1 00:01:56 1970  Chassis  Action  major  Hot insertion of /CH/BL6
    1314  Thu Jan  1 00:01:56 1970  Chassis  Action  major  Hot insertion of /CH/BL5

4. Workaround

All T6300 blades should be upgraded to the latest firmware version 6.7.2 (delivered in patch139438-02) as this new firmware may decrease the chances of the described issue from occurring.

In Sun Blade 6000 configurations populated with 2 or 3  Sun Blade T6300's,  move the blades to the lower chassis slots while moving the other blades (i.e T6320) to the higher slots. Also, CMM ILOM firmware should be at 2.0.3.2 or later.  This has been known to decrease the chances of the described issue from occurring.

Alternatively, setting the "sys_eventlevel" to 0 or 1 in the T6300 ALOM will suppress the messages from being logged to the "/var/adm/messages" logs. Level 1 is for critical messages and 0 is for zero messages to be logged to the "/var/adm/messages" file. It is recommended that the "sys_eventlevel" be set at level 1 so that critical messages are still logged.

Note: This will not stop the messages from being logged on the CMM or the T6300 SP's as there is no way to suppress these.

5. Resolution

This issue is addressed in the following platforms:

• Sun Blade 6000 Chassis Monitoring Module (CMM) with motherboard level 371-1447-06 (or above) and CMM ILOM firmware 2.0.3.2 (or later).
  

• Sun Blade T6300 Server Module with I2C bridge chip firmware v1.4 (or later).

as delivered in the following T6300 Blade Module part numbers:

• 541-2317-07 (or above)
  
• 541-2318-07 (or above)
  
• 541-2319-07 (or above)
  
• 541-2320-06 (or above)

Note: In addition to the required motherboard levels above the System Firmware must also be upgraded to sysfw 6.7.2 (or later), which contains ALOM code to enable the dynamic arbitration/priority feature within the bridge chip.

Change History

03-Now-2011- Corrected the document state from "workaround" to "resolved"

Internal Comments
The ultimate fix will be a two part fix including T6300 blade replacement to update the motherboard FourEyes I2C bus bridge chip firmware to v1.4 which provides I2C bus Dynamic Arbitration and Dynamic Priority features. All blades must use this polling mechanism to ensure fair access to the I2C Bus and to prevent the timeouts that produce the false messaging.


In addition the system firmware must be upgraded to v6.7.1 which
contains ALOM code to enable the dynamic arbitration/priority feature
within the bridge chip. These fixes will be released in an upcoming FAB
once final testing is complete.



Please send technical questions to the following email:

  [email protected]

and CC the following persons:

  Internal Contributor/Submitter

  Internal Eng Responsible Engineer


Internal Contributor/submitter
[email protected]

Internal Eng Responsible Engineer
[email protected]

Internal Services Knowledge Engineer
[email protected]

Internal Eng Business Unit Group
SSG WGS (Workgroup Systems)

Internal Escalation ID
1-24303970, 1-24865240, 1-25122354, 1-24110501, 1-24146902, 1-23946501 1-24171305, 1-24864989, 1-25084069, 1-24435495

Internal Sun Alert & FAB Admin Info
WF 23-Mar-2009, jfolla: Updated with requested changes and sent to Nitin for review.
WF 07-Jan-2009, jfolla: This Sun Alert was placed on hold per request. Received release approval. Sent for release.
WF 18-Dec-2008, jfolla: sent for 24 hr review
WF 18-Dec-2008, jfolla: sent to submitter review along with a few questions regarding the submitted draft
WF 18-Dec-2008, jfolla: created


Attachments

This solution has no attachment