Sun SPARC Enterprise M9000-32 Server
 
Sun SPARC Enterprise M8000 Server
 
Sun SPARC Enterprise M9000-64 Server
 
Sun SPARC Enterprise M5000 Server
 
Sun SPARC Enterprise M4000 Server
 
Sun SPARC Enterprise M3000 Server
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: Mx000
 
.Old GCS Categories>Sun Microsystems>Servers>OPL Servers
 

Applies to:

Goal

The implementation of the Fault Management Architecture on the Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers is complex. The goal of this document is not to describe how FMA behaves on the Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers but to help to identify and display the faults reported on the components of these platforms, how/when these faults can be cleared and how/when the fault LEDs are turned ON or OFF.

Fix

Faults on Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers :

FMA is obviously available on the Sun SPARC Enterprise domains running Solaris[TM] 10. But FMA has also been ported to run on the Service Processor and the fmd daemon is running as part of the XSCF software. Via the "Event Transport Module" (ETM) on top of the "Domain to Service Processor Communications Protocol" (DSCP), XSCF on the Service Processor and Solaris on the domain, we can communicate ereports and faults.

When a fault is diagnosed, the system is able to identify one or several suspects, depending on the nature of the fault. The suspect or the list of the suspects can be displayed using the 'fmdump' command on XSCF.

Example where a list of suspects has been identified :

Example where only one suspect has been identified :

The Knowledge Articles available at https://support.oracle.com request to collect the result for a 'fmdump -m'.  The "-m" option is available only on the XSCF (not on the Solaris domain) and displays the Fault Manager syslog message contents for the event(s).

Example :

Only users with the platop, platadm, or fieldeng privileges can run the 'fmdump' command.

The information about the faulty status of the components is available in the CMEM database on XSCF.
Based on the certainty of the fault, the following flags are set in the CMEM :

• CFF => Certainly Faulty Flag

• UFF => Uncertainly Faulty Flag

Every FRU that is considered as suspect in the list has the uncertain_secondary_status set. But only the primary suspect may also have the CFF or UFF bit set.

As a consequence for a fault detected and depending on the CFF or UFF bit set, the primary suspects in a suspect list are reported as "faulted" (completely broken/not working) or "degraded" (should be replaced, but is still working with some limitations) in the 'showhardconf' and 'showstatus' outputs.
The secondary suspects show up as "degraded". Some components may be reported as "deconfigured" if these components are victims of a fault detected on another component.

As part of FMA ported on XSCF, the 'fmadm faulty' command is only available in Escalation mode. The resource cache has less info about platform faults than the "CMEM" database. CMEM is the database that has the real info about faulty FRUs for OPL, not the resource cache.  Therefore, the 'showstatus' command is the preferred method to be used by the customer and field, as it provides a truly accurate picture of what is going on regarding faults.

The 'showstatus' and 'showhardconf' commands are available to the users with the following privileges :
useradm, platadm, platop, fieldeng

Example for the showstatus and showhardconf commands reporting a chip on a CPU Module as "faulted".

Example from a M9000 system where a CMU is reported as degraded due to some DIMMs deconfigured as a consequence for a fault detected on the Memory Address Controller.

For some components considered as primary suspect and certainly faulty (CFF), the Maintenance Action Required bit is set.  This information is available in a 'fmdump -V' output.

Example :

Further OPL FMA information can be found at:

<Document 1386385.1> M3000/M4000/M5000/M8000/M9000 Server: How to Use FMA With OPL Servers

Steps to Follow

Clearing Faults on Sun SPARC [TM] Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers.

Clearing Faults (Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers running XCP 1050 or later) :

1. Usual process :

The right course of action when the system has identified faulty components on a platform is to replace the primary suspect.

In order to repair a fault with single or multiple FRUs, the typical repair action will be:

• Replace the first FRU indicted in the suspect list; if the FRU is a CMU or sub-FRU of a CMU (on M8000/M9000), IOU (on M8000/M9000), FAN, PSU, DDC_A (on M8000), XSCFU (on M8000/M9000), or XSCFU_C (M9000 plus expansion cabinet) use the 'replacefru' command to do so; Otherwise, use cold replacement;
  The fieldeng privileges are required to run the 'replacefru' command.

• For all the other FRUs on the suspect list (secondary suspects), use the 'clearfault' command.
  The 'clearfault' command will return with an error message when trying to clear a FRU that is the first suspect on a suspect list :
  "clearfault: Fault cannot be cleared for this FRU"
  You must be in Service or Escalation mode to run the 'clearfault' command. 

• Sun Shared Shell
  The Service Engineer should offer the use of Shared Shell to accomplish the service mode and clearfault.
  Before the Shared shell session is terminated the "disablemodes" command should be run.

2. Complex cases :

In some more complex cases, the context might lead to the decision to not replace the first FRU in the suspect list. This might be due to the diagnosis engine being wrong or the FRU in a single-FRU indictment seems to be wrong or first FRU has already been replaced and this is a repeat fault with an identical list etc...

The decision to not replace the first FRU in the suspect list MUST be done by Service and the whole process of clearing a fault without replacing the suspect component must be done under the supervision of TSC. The following section describe how to handle the complex cases.

2.1 - Power cycle (NFB Off/On) :

For the systems running XCP1050, it's possible to clear the fault status for primary suspects by power-cycling the platform and the keyswitch is in the service position.  For the systems running XCP1060 and later, faults are not be cleared on NFB-on, no matter what the position of the keyswitch.

2.2 - Commands available to clear the faults :

2.2.1 - clearfru / clearstatus :

These 2 commands can be used to clear the fault information of all the FRUs (clearfru) or the fault information of FRUs that have been detected as faulty units (clearstatus).

You must be in Escalation mode to run the clearfru / clearstatus commands.

The domains must be down and an immediate platform power cycle is required. The component is reported as faulted as long as the power cycle hasn't occured.

Using clearfru and clearstatus must be done only on directions from TSC and/or Engineering.

Example :

2.2.2 - clearfault :

The 'clearfault' command provides a much clever way to manage faults for primary suspects and can be used in a way to :

• Immediately clear the fault status for other FRUs than the first suspect in a multi-suspect list,

• Clear the fault for FRUs reported as the first suspect in a suspect list in the following cases :

2.2.2.1 - During the next power-cycle by using 'clearfault [-l]' (no need for the keyswitch to be in the service position).

As a consequence, the next power-cycle clears the faults for the components.

Notes :

• If the user invokes 'clearfault -l', there is no attempt to clear the fault status now. The user is prompted with the information that the  "FRU will be marked to clear fault on next circuit breaker off and on."

Example :

• If the user invokes 'clearfault' (without -l), the system will attempt to clear the status now.
  If the FRU is a primary suspect and clearfault cannot clear the fault status, it will prompt the user requesting whether or not the FRU should be marked to clear fault on next circuit breaker off and on; behaving the same manner as invoking 'clearfault -l' directly.

Example :

• If clearfault is invoked when another maintenance command is running, it will behave the same manner as invoking 'clearfault -l' directly.

Example :

2.2.2.2 - M3000/M4000/M5000 :

• Immediately clear the fault for the first suspect in a suspect list if the FRU is a PSU or a FAN,

Example :

• For the other FRUs, a power cycle is required to clear the fault.

Example :

2.2.2.3 - M8000/M9000 :

• Immediately clear the fault for the first suspect in a suspect list if the FRU is a PSU or FAN or DDC_A .

Example :

• Immediately clear fault if the FRU is a CMU or IOU, including sub-FRUs like CPUM, DIMM ... and non-FRU like DDC, SSM ..., without NFB off/on when :

  • the CMU, IOU (and sub-FRU and non-FRU) is not part of a running domain. See the Examples section below,

  • If the FRU is a CMU (and sub-FRU and non-FRU) or an IOU and is part of a running domain, Dynamic Reconfiguration (deleteboard) must be invoked prior to run the clearfault command in order to detach the FRU from the domain. After clearing the fault, the component can safely be added back to the domain. See the Examples section below.

  • If the FRU is a CMU (and sub-FRU and non-FRU) or an IOU and is part of a running domain but DR cannot be used then the domain must be powered off in order to immediately clear the fault status. Otherwise, the fault status will be cleared on next circuit breaker off and on. See the Examples section below.

    • Note : if the XSB containing the CMU/IOU is in quad-mode, all of the domains using the quads must be powered off.

• if the FRU is an XSCFU, a platform power cycle is required or replacefru can be invoked from the XSCFU to dummy replace and power cycle the XSCFU.

2.3 - As a summary, to clear fault on a FRU :

M3000/M4000 / M5000 :

M8000 / M9000 :

For more detail on accessing escalation or service mode see <Document 1002928.1>

LED behaviour :

Each M3000/M4000/M5000/M8000/M9000 system has an Operator Panel (OPNL) with 3 LEDs :

• the Power LED,

• the XSCF Standby LED,

• the Check LED.

When turned ON, the Check LED, aka the System Check LED, indicates a fault on the system. See below.

Most of the FRUs on the SPARC Enterprise servers have a FRU check LED which reports that the unit contains an error. However, some FRUs like DIMMs or CPUMs do not have LEDs.  Refer to the SPARC Enterprise Mx000/Mx000 Servers Service Manuals for more information about LEDs.

For Sun SPARC Enterprise servers running a version of XCP later than 1050, the check LEDs will be set and reset as below :

• the FRU check LED is set if the FRU is the sole FRU in a suspect list; including sub-FRU (CPUM, DIMM ...) and non-FRU (DDC, SSM ...).

• the system check LED is set if there are any FRUs which is considered as the primary suspect (CFF / UFF) or secondary suspect; which means when 'showstatus' reports any FRUs as faulty or degraded.
  Including IO Box FRUs reported as suspect.

Check LEDs behaviour after clearfault, clearstatus, replacefru :

• replacefru :

  • the FRU's check LED is :

    • ON until the maintenance,

• blinking during the maintenance,

• OFF as soon as the replacefru as completed successfully.

  • the System check LED is OFF :

    • as soon as the replacefru as completed successfully,

• and there is no other suspect component in the system left,

• clearfault :

  • the FRU's check LED is turned off as soon as the clearfault command has succeeded successfully in clearing the fault for the FRU.

• the System's check LED is turned off as soon as the fault status for the latest suspect component is cleared.
  This implies that the LED will turn off after the subsequent power cycle in certain cases as applicable.

• clearstatus/ clearfru : the FRU and System check LEDs remain ON until the next power cycle,

Faults on IOBox :

Faults detected on IOBox are stored in the CMEM and in the FRUID of the IOBox (Status_CurrentR).
This information is reported in the showstatus output on the XSCF.
Example :

When a fault is reported on the IOBox or its components, the Service LED on the IOBox or PSU is lit.
When an iobox fru is discovered, dfrud reads the Status_CurrentR. If it contains fault info, the fault info is added to CMEM, and the Service led is turned on. This can be checked via the ioxadm command :

XSCF> ioxadm env -v Location Sensor Min Min Alarm Value Max Alarm Max Units [...] IOX@X156/IOB1 SERVICE - - On - - LED

Even if a fault is reported on IOBox and Service LED is lit, the OPNL System Ckeck LED is not lit .
The clearfault command can be used to clear the fault status for primary and secondary suspect on the IOBox and its components; similarly to any other components in the platform chassis (CMU, DIMM, IOU etc ...) for XCP > 1050.
Example :

Clearing the LINK to the IOBox:

Example:

As soon as there is no more fault status reported in the showstatus output then all the Service LEDs are cleared. There is no condition requiring to power cycle the IOBox to clear a  fault status (similar to clearfault -l).

Hierarchical fault clearing :

In certain cases, the faulted resources appear to be hierarchical.
In the following example, after clearing the fault on CMU#0, we need to clear the fault on the subordinates.

CMU#0 remains in the output, although not marked faulted, until the subordinates are cleared:

1. M3000/M4000 / M5000 :

1.1 - clearing a fault on a PSU :

1.2 - clearing a fault on a DIMM :

1.3 - clearing a fault on a CPUM :

2. M8000 / M9000 :

2.1 - clearing a fault on a PSU :

2.2 - clearing a fault on the OPNL :

2.3 - clearing a fault on an IOU not part of a running domain :

2.4 - clearing a fault on a CMU not part of a running domain

2.5 - clearing a fault on a CMU which is part of a running domain :

We can use DR to detach the XSB and clear the fault.

2.6 - clearing a fault on a CMU which is part of a running domain but DR cannot be used :

Since DR cannot be used for whatever reason, the domain must be powered off prior to using clearfault :

Keywords: opl, fault, clear, clearfault, clearstatus, fmdump, showstatus

References

This solution has no attachment