---------------------------------------------------------------------
- Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------  
                            FIELD INFORMATION NOTICE
                  (For Authorized Distribution by SunService)

FIN #: I0753-1

Create Date: Jan/02/02

Keywords: 

Sun Fire 15K installations running SMS 1.1 may experience deadlock condition

SunAlert: No

Top FIN/FCO Report: No

Products Reference: Sun Fire 15000 Server

Product Category: Server / Service

Product Affected: 

Mkt_ID   Platform   Model   Description            Serial Number
------   --------   -----   -----------            -------------
  -        F15K      ALL    Sun Fire 15K Server          -


X-Options Affected
------------------
Mkt_ID   Platform   Model   Description   Serial Number
------   --------   -----   -----------   -------------
  -         -         -          -              -

Parts Affected: 

Part Number      Description                          Model
-----------      --------------------                 -----
501-5121-10      System Control Board, Starcat          -

References: 

BugId: 4504754 - tmd throttle should be 1 until all lock issues are 
                 resolved.

Issue Description: 

Sun Fire 15K installations running System Management Software 1.1 (SMS)
might not complete HPOST and will not boot Solaris if the
'setkeyswitch' command is run at the same time on multiple domains.
This may lead to a power recovery or a global DStop where recovery of
all domains is attempted. 

Due to locking issues between SMS 1.1 and HPOST, running multiple
simultaneous setkeyswitch operations could result in a deadlock
situation, causing the setkeyswitch/HPOST process will hang.  The
operation eventually times out, at which time the setkeyswitch/HPOST
operation fails.  Solaris will not boot up on the affected domains.

The problem is seen when multiple setkeyswitch commands are run in
parallel via 'setkeyswitch on' for one domain and then immediately
running it on other domains before the first domain completes HPOST.
A possible scenario would be multiple administrators running 
'setkeyswitch on' for different domains at the same time.  This
results in more than one 'setkeyswitch' process running concurrently
which can be seen using 'ps -ef' from the SMS console.

A typical hang condition would look like:

     % setkeyswitch -d a on
     ......
	___          <--- POST messages cease, don't return to prompt
	
This deadlock condition occurs because HPOST fails to follow the proper
lock ordering policy.  It could occur with any command that invokes
HPOST such as setkeyswitch, addboard, moveboard, rcfgadm or cfgadm.

SMS version 1.2 fixes this deadlock situation and is now in testing.
Until this new version is released, please follow the workaround 
recommendation given below.

Implementation: 

---
        |   |   MANDATORY (Fully Pro-Active)
         ---    
         
  
         ---
        |   |   CONTROLLED PRO-ACTIVE (per Sun Geo Plan) 
         --- 
         
                                
         ---
        | X |   REACTIVE (As Required)
         ---

Corrective Action: 

An Authorized Enterprise Services Field Representative may avoid the
above mentioned problems by following the recommendations as shown
below.

Please adhere to the following guidelines:    

Ensure that setkeyswitch operations are serialized and not run in
parallel.  This can be done by modifying the throttle value for
the 'tmd' daemon in the ssd_start file. 

  Change the 2nd field of the tmd entry in 

   /etc/opt/SUNWSMS/startup/ssd_start from '-t 4' to '-t 1'. 
   
   The final entry reads:

         tmd:-t 1:0:1:0:1:4:4:sms-tmd

Then, as root, stop and start SMS for the changes to take effect.

	 # /etc/init.d/sms stop
	 # /etc/init.d/sms start


With this workaround, setkeyswitch operations can be issued in
parallel, but SMS internally serializes the operations.  This means
that all but the first parallel setkeyswitch operation will appear to
hang until the first operation completes.  The operations are in fact
not hanging, but awaiting their turn to execute.

There is a penalty to this serialization.  Testing has revealed that
booting 15 domains at the same time takes twice as long when the above 
workaround is implemented.

Comments: 

None.

=========================================================================

Implementation Footnote: 

i)   In case of MANDATORY FINs, Enterprise Services will attempt to    
     contact all affected customers to recommend implementation of 
     the FIN. 
   
ii)  For CONTROLLED PROACTIVE FINs, Enterprise Services mission
critical    
     support teams will recommend implementation of the FIN  (to their  
     respective accounts), at the convenience of the customer. 

iii) For REACTIVE FINs, Enterprise Services will implement the FIN as
the   
     need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network 
browser as follows:
 
SunWeb Access:
-------------- 
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/

* From there, select the appropriate link to query or browse the FIN and
  FCO Homepage collections.
 
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/

* From there, select the appropriate link to browse the FIN or FCO
index.

Supporting Documents:
---------------------
* Supporting documents for FIN/FCOs can be found on Edist.  Edist can be 
  accessed internally at the following URL: http://edist.corp/.
  
* From there, follow the hyperlink path of "Enterprise Services
Documenta- 
  tion" and click on "FIN & FCO attachments", then choose the
appropriate   
  folder, FIN or FCO.  This will display supporting directories/files
for 
  FINs or FCOs.
   
Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to [email protected]
---------------------------------------------------------------------------