Sun Fire E25K Server
 
Sun Fire E20K Server
 
Sun Fire 12K Server
 
Sun Fire 15K Server
 

GCS>Sun Microsystems>Servers>High-End Servers
 

Domains on Sun Fire [TM] 12k/15k/E20k/E25k may dstop due to clock issues when SMS is simultaneously started on both SCs.

The system controller (SC) in Sun Fire high-end systems is a multifunction, CP1500- or CP2140-based printed circuit board (PCB) that provides critical services and resources required for the operation and control of the Sun Fire system. The SCs provides several services for the Sun Fire system via the System Management Services (SMS). Among these services, the SCs provides:

The SC that controls the platform is referred to as the main SC, while the other SC acts as a backup and is called the spare SC.

When SMS is simultaneously started on both SCs, they will both start their daemons in the "main" mode. During the startup period, the main SC will reprogram the boards to select its clock as input. As both SCs believe that it is the main SC, the synchronization of the clocks is not guaranteed (the spare SC is supposed to "follow" the main SC's clock) and they will both try to reprogram the boards.

The SMS software check for the synchronization of SCs'clocks before reprogramming the boards but the programming process may take a couple of seconds and the clocks may get out of synchronization during this small window. In this case, the domains using the reprogrammed boards may dstop due to clock issues.

After a short period, the SMS software will detect that both SCs are trying to become the main SC and SC0 will reset SC1 to prevent what is called a split brain situation. But the detection occurs too late to prevent the potential issue of getting the clocks out of synchronization during the reprogramming of the boards.

As a result, we do always recommend to start SMS on the previous main SC first, wait for the completion of the sms startup process and then start SMS on the spare SC.

For example, from the README file of the SMS patches:

Special Install Instructions:
-----------------------------
Follow these steps when installing on the SC:
1. Record which SC is the main SC.
2. Disable failover on MAIN SC (setfailover off).
3. Stop the SMS processes on both SC's simultaneously.
   /etc/init.d/sms stop
4. Install the patch on both SC's.
5. Start the SMS processes on the previous main SC first.
   /etc/init.d/sms start
6. After all the sms processes have started (i.e. you're able to run the
   showenvironment command and get all the system's status),
   start the SMS processes on the Spare SC next.
7. Enable failover on MAIN SC (setfailover on).

During normal operation, the SCs use the 75Mhz voltage controlled crystal oscillator (VCXO) to source clocking for the entire platform.
The control voltage circuit of the VCXO either uses the output of a phase-frequency comparator or a fixed voltage reference to drive the VCXO.

The selection of the reference is done via a software write to a register (REF_SEL) in the Gchip.

When REF_SEL=1 (this is called 'leader mode'), the SC use the fixed voltage reference to drive the VCXO.

When REF_SEL=0 (this is called 'follower mode'), the SC compares its clock against the clock coming from the other SC and uses the error signal resulting from this comparison to drive the VCXO. This cause the SC clock to follow the other SC clock.

By setting the MAIN SC REF_SEL to 1 and the SPARE SC REF_SEL to 0, the two SCs will generate identical clocks signals. This is called phase lock and this is necessary to ensure a safe clock failover.

When the SCs are both in 'leader mode' or both in 'follower mode', the phase lock is not guaranteed.
2)Failover Management Daemon (fomd) Background

When fomd is started, its state is UNKNOWN and its initial task is to determine the role of the SC.

In order to determine it's own role, fomd must determine the state of the opposite SC. This is accomplished by checking if the opposite SC is responding to a Remote Procedure Call (RPC) via the I2 network and/or is generating a hardware heartbeat
(by checking the interrupt status register in its SBBC).

If the opposite SC is running as MAIN, the starting fomd becomes SPARE.

If the opposite SC is not producing a heartbeat, the starting fomd will assume the main role. It will change its state to BECOMING_MAIN, will start generating a heartbeat and will instruct the Hardware Access Daemon (hwad) to change its state to MAIN.

When SMS is started simultaneously on both SCs, the check for a heartbeat will fail and both SCs will proceed to BECOMING_MAIN (and both SCs will instruct hwad to become MAIN).

To prevent a split brain (both SCs are in the MAIN role), SC0 will try again to determine the role of SC1 after entering the BECOMING_MAIN state.

If SC1 is not Spare at this time, SC0 will reset SC1 and will instruct hwad to restart the MAIN initialization to make sure it undoes anything the spare might have done before it got reset.

When hwad is becoming MAIN, it first configures the SC in 'follower mode' (setting REF_SEL to 0), then it reads the device presence registers to decide what boards are present and register these boards.

If the SC is able to register all boards and if the clocks are phase locked, the SC will program the Smart Phased Lock Loop (SPLL) on each boards to select its clock as input.

4) Potential clock issue when SMS is started simultaneously on both SCs
========================================================================

When both SCs are started simultaneously, the check by fomd for a heartbeat will fail and both SCs will instruct hwad to become MAIN.
Both SCs will go into 'follower mode' and the phase lock is not guaranteed anymore.

Hwad will check if the clocks are phase lock before reprogramming the SPLL on the boards but it may take a couple of seconds to reprogram the SPLLs.

There is a small chance to loose the phase lock during the reprogramming of the SPLLs. In this case, if the SPLL was previously programmed to use the clock from the other SC, the domain using the reprogrammed board may dstop due to clock issue.

This solution has no attachment