How to Replace the Motherboard or Service Processor in a Logical Domain (LDOM) Environment

Asset ID:	1-71-1019720.1
Update Date:	2012-08-22
Keywords:

Solution Type Technical Instruction Sure

Solution 1019720.1 : How to Replace the Motherboard or Service Processor in a Logical Domain (LDOM) Environment

Applies to:

Sun Blade T6340 Server Module - Version Not Applicable and later
Sun Fire T1000 Server - Version Not Applicable and later
Solaris SPARC Operating System - Version 10 9/10 U9 and later
Sun SPARC Enterprise T5240 Server - Version Not Applicable and later
Sun SPARC Enterprise T5440 Server - Version Not Applicable and later
All Platforms

Goal

This document details how to backup and recover the Oracle VM for SPARC (aka LDOM) environment following the replacement of the service processor in a CMT capable system.

The LDOM configuration is stored on the service processor in CMT systems. The service processor may be integral to the motherboard or an individual FRU. When this part is replaced the LDOM configuration will be lost.

This procedure explains the steps needed to recover the Logical Domains in a basic LDOM configuration. It does not include procedures for recovery of Solaris ZFS or Veritas Storage management.

Fix

The method used to backup and recover your Oracle VM for SPARC (aka LDOM) configuration depends on which version of Oracle VM for SPARC you are running:

Oracle VM for SPARC 2.0 and later only:

The backup and recovery process for Oracle VM for SPARC 2.0 and later is documented in the Oracle VM for SPARC administration guide:

Oracle VM for SPARC 2.0 Administration Guide > Saving Domain Configurations for Future Rebuilding

Oracle VM for SPARC 2.1 Administration Guide > Saving Domain Configurations for Future Rebuilding

Oracle VM for SPARC 2.2 Administration Guide > Saving Domain Configurations for Future Rebuilding

The procedure is the same for Oracle VM for SPARC 2.0, 2.1 and 2.2. The links provided above are just for convenience.

Oracle VM for SPARC 1.3 and earlier only:

Only continue beyond this point if you are running Oracle VM for SPARC 1.3 and lower.

This section provides a basic example of recovering your Oracle VM for SPARC (aka LDOMs) 1.3 and lower configuration as per the documentation at Logical Domains 1.3 Administration Guide > Saving Logical Domain Configurations for Future Rebuilding

Backup of the ldom environment before hardware replacement.
Identify the ldom names using theldm list-domaincommand:

Example:

# ldm list-domain NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv SP 8 4G 0.1% 1d 4h 52m guest1 active ----- 8 4G
Capture the LDOM environment
Log in as an authorized user or superuser
Use each of the following methods and save the output to the root-disk of the primary domain to ensure the required information is captured
Run the command:# ldm ls -e -p > ldm_ls-e-p.out
Run the command:# ldm list-constraints -x <ldom> > <ldom>.xml for each of the ldom names identified earlier
Create an explorer output of the primary domain (use explorer rev 6.7 or higher) with the command:# /opt/SUNWexplo/bin/explorer -w default,ldom
If this is not available then the system configuration can be obtained from the file/var/opt/SUNWldm/ldom-db.xml

*See the Appendix to this document for examples of the outputs from these commands. These examples are used in this procedure.
Shutdown all the domains and power off the system
Replace the motherboard or SC following the procedures shown in the appropriate service guide. See http://www.oracle.com/technetwork/documentation/sparc-tseries-servers-252697.html and http://docs.oracle.com/cd/E19076-01/index.html
Boot the operating system.
The system will display an errors similar to the following:

Oct 14 12:15:18 svc.startd[7]: svc:/network/physical:default: Method "/lib/svc/method/net-physical" failed with exit status 96. Oct 14 12:15:18 svc.startd[7]: network/physical:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details) Oct 14 12:15:34 svc.startd[7]: svc:/ldoms/vntsd:default: Method "/lib/svc/method/svc-vntsd" failed with exit status 1. Oct 14 12:15:34 svc.startd[7]: svc:/ldoms/vntsd:default: Method "/lib/svc/method/svc-vntsd" failed with exit status 1. Oct 14 12:15:35 svc.startd[7]: svc:/ldoms/vntsd:default: Method "/lib/svc/method/svc-vntsd" failed with exit status 1. Oct 14 12:15:35 svc.startd[7]: ldoms/vntsd:default failed: transitioned to maintenance (see 'svcs -xv' for details)

It will still be possible to login to the system, however there will be no network access.
Login to the system as root
Move the virtual network interface from /etc/hostname.vsw0 to the physical network interface. In this example the physical interface is e1000g0

# mv /etc/hostname.vsw0 /etc/hostname.e1000g0
Configure the physical network interface

# ifconfig e1000g0 plumb # ifconfig e1000g0 ip_of_e1000g0 netmask netmask_of_e1000g0 broadcast + up
Reboot the system and then login again.
Verify that the firmware on the system is on the same level or higher. Please check this URL to get information about system firmware: http://www.oracle.com/technetwork/systems/patches/firmware/release-history-jsp-138416.html

In some instances the previous svc-vntsd failure messages may still be displayed. However, it will still be possible to login.
Manually re-create the Primary Domain.

The following examples of the commands use the parameters from the example files in the Appendix. Substitute the true values from the saved files.
1. Recreate the Virtual Disk Server (vds)
  
  <ldom> <ldom_name>primary</ldom_name> - - - <vds> <vds_instance> <service_name>primary-vds0</service_name> <vds_volumes> <vds_volume> # ldm add-vds primary-vds0 primary
2. Re-create the Virtual Console Concentrator service (vcc)
  
  <vcc> <vcc_instance> <service_name>primary-vcc0</service_name> <min_port>5000</min_port> <max_port>5100</max_port> </vcc_instance> </vcc> # ldm add-vcc port-range=5000-5100 primary-vcc0 primary
3. Re-create the cryptographic resource
  
  <mau> <number>1</number> </mau> # ldm set-mau 1 primary
4. Re-create the Virtual CPUs
  
  <cpu> <number>8</number> </cpu> # ldm set-vcpu 8 primary
5. Re-create the memory
  
  <memory> <size>4294967296</size> </memory> # ldm set-memory 4G primary
6. Save the configuration to the service controller
  
  # ldm add-config initial
7. List the saved configs
  
  # ldm list-config factory-default [current] initial [next]
8. Perform a re-configuration boot of the system
  
  # touch /reconfigure # shutdown -y -g0 -i6
Power off and then power on the platform and allow the system to boot
Restore the virtual network

# ifconfig -a # ifconfig vsw0 plumb # ifconfig e1000g0 down unplumb # ifconfig vsw0 <ip_of_e1000g0> netmask <netamsk_of_e1000g0> broadcast + up # mv /etc/hostname.e1000g0 /etc/hostname.vsw0
Check the virtual network terminal server daemon is enabled.

# svcs -l vntsd fmri svc:/ldoms/vntsd:default name virtual network terminal server enabled true state online next_state none state_time Wed Oct 15 16:03:54 2008 logfile /var/svc/log/ldoms-vntsd:default.log restarter svc:/system/svc/restarter:default contract_id 57 dependency optional_all/error svc:/milestone/network (online) dependency optional_all/none svc:/system/system-log (online)

If this is not enabled use the following command

# svcadm enable vntsd
Restore the virtual disk server to the guest domains

<vds> <vds_instance> <service_name>primary-vds0</service_name> <vds_volumes> <vds_volume> <block_dev>/dev/dsk/c1t1d0s2</block_dev> <vol_name>vol1</vol_name> <vol_opts/> </vds_volume> </vds_volumes> </vds_instance> </vds> # ldm add-vdsdev /dev/dsk/c1t1d0s2 vol1@primary-vds0
Run the following command for each guest domain’s XML file you created in Step 7.

# ldm add-domain -i ldom.xml
Finally Bind and Start the remaining domains.

# ldm bind-domain guest1 # ldm start-domain guest1

Appendix

Output from command:ldm ls -e -p

VERSION 1.1

DOMAIN|name=primary|state=active|flags=normal,control,vio-service|cons=SP|ncpu=8|mem=4294967296|util=12.8|uptime=684412|softstate=Solaris running

VCPU

|vid=0|pid=0|util= 17|strand=100

|vid=1|pid=1|util=0.4|strand=100

|vid=2|pid=2|util=0.4|strand=100

vid=3|pid=3|util=0.4|strand=100

|vid=4|pid=4|util= 81|strand=100

|vid=5|pid=5|util=1.5|strand=100

|vid=6|pid=6|util=2.1|strand=100

|vid=7|pid=7|util=0.0|strand=100

MAU

|id=0|cpuset=0,1,2,3,4,5,6,7

MEMORY

|ra=0x8000000|pa=0x8000000|size=4294967296

IO

|dev=pci@0|alias=pci

|dev=niu@80|alias=niu

VCC|name=primary-vcc0|nclients=1|port-range=5000-5100

VSW|name=primary-vsw0|nclients=1|mac-addr=00:14:4f:fb:6f:f9|net-dev=e1000g0|dev=switch@0|mode=

VDS|name=primary-vds0|nclients=1

|vol=vol1|opts=|dev=/dev/dsk/c1t1d0s2

VLDC|name=primary-vldc3|nclients=7

VLDC|name=primary-vldc0|nclients=4

VLDCC|name=ds|service=primary-vldc0@primary|desc=domain-services

VLDCC|name=vldcc2|service=SP|desc=spfma

VLDCC|name=vldcc1|service=primary-vldc0@primary|desc=ldmfma

VCONS|type=SP

DOMAIN|name=guest1|state=active|flags=normal|cons=5000|ncpu=8|mem=4294967296|util=0.1|uptime=512444|softstate=Solaris running

VCPU

|vid=0|pid=8|util= 75|strand=100

|vid=1|pid=9|util= 49|strand=100

|vid=2|pid=10|util= 11|strand=100

|vid=3|pid=11|util=0.0|strand=100

|vid=4|pid=12|util=100|strand=100

|vid=5|pid=13|util=0.0|strand=100

|vid=6|pid=14|util=0.0|strand=100

|vid=7|pid=15|util=0.0|strand=100

MEMORY

|ra=0x8000000|pa=0x108000000|size=4294967296

VARIABLES

|auto-boot?=true

|boot-device=vdisk1

|keyboard-layout=US-English

VNET|name=vnet1|dev=network@0|service=primary-vsw0@primary|mac-addr=00:14:4f:f9:15:9e

VDISK|name=vdisk1|vol=vol1@primary-vds0|dev=disk@0|server=primary

VLDCC|name=ds|service=primary-vldc0@primary|desc=domain-services

VCONS|group=guest1|service=primary-vcc0@primary|port=5000

Output fromldm list-constraintscommand for control/primary domain

<?xml version="1.0"?>

<LDM_interface version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="./schemas/combined-v3.xsd" xmlns:ovf="./schemas/envelope" xmlns:rasd="./schemas/CIM_ResourceAllocationSettingData" xmlns:vssd="./schemas/CIM_VirtualSystemSettingData" xmlns:gprop="./schemas/GenericProperty" xmlns:bind="./schemas/Binding">

<data version="3.0">

<Envelope>

<References/>

<Content xsi:type="ovf:VirtualSystem_Type" ovf:id="primary">

<Section xsi:type="ovf:ResourceAllocationSection_Type">

<Item>

<rasd:OtherResourceType>ldom_info</rasd:OtherResourceType>

<rasd:Address>00:14:4f:ab:e3:16</rasd:Address>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>cpu</rasd:OtherResourceType>

<rasd:AllocationUnits>8</rasd:AllocationUnits>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>mau</rasd:OtherResourceType>

<rasd:AllocationUnits>1</rasd:AllocationUnits>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>memory</rasd:OtherResourceType>

<rasd:AllocationUnits>4G</rasd:AllocationUnits>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>physio_device</rasd:OtherResourceType>

<gprop:GenericProperty key="name">pci@0</gprop:GenericProperty>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>physio_device</rasd:OtherResourceType>

<gprop:GenericProperty key="name">niu@80</gprop:GenericProperty>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>vsw</rasd:OtherResourceType>

<rasd:Address>auto-allocated</rasd:Address>

<gprop:GenericProperty key="service_name">primary-vsw0</gprop:GenericProperty>

<gprop:GenericProperty key="dev_path">e1000g0</gprop:GenericProperty>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>vcc</rasd:OtherResourceType>

<gprop:GenericProperty key="service_name">primary-vcc0</gprop:GenericProperty>

<gprop:GenericProperty key="min_port">5000</gprop:GenericProperty>

<gprop:GenericProperty key="max_port">5100</gprop:GenericProperty>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>vds</rasd:OtherResourceType>

<gprop:GenericProperty key="service_name">primary-vds0</gprop:GenericProperty>

</Item>

</Section>

<Section xsi:type="ovf:VirtualHardwareSection_Type">

<Item>

<rasd:OtherResourceType>vds_volume</rasd:OtherResourceType>

<gprop:GenericProperty key="vol_name">vol1</gprop:GenericProperty>

<gprop:GenericProperty key="block_dev">/dev/dsk/c1t1d0s2</gprop:GenericProperty>

<gprop:GenericProperty key="service_name">primary-vds0</gprop:GenericProperty>

</Item>

</Section>

</Content>

</Envelope>

</data>

</LDM_interface>

Contents of the file /var/opt/SUNWldm/ldom-db.xml

<?xml version="1.0"?>

<ldom_database version="3.0">

<ldom>

<ldom_name>primary</ldom_name>

<cpu>

<number>8</number>

</cpu>

<mau>

<number>1</number>

</mau>

<memory>

<size>4294967296</size>

</memory>

<io>

<io_instance>

<iodevice>pci@0</iodevice>

<bypass_mode>0</bypass_mode>

</io_instance>

<io_instance>

<iodevice>niu@80</iodevice>

<bypass_mode>0</bypass_mode>

</io_instance>

</io>

<mac_address>0x00144fabe316</mac_address>

<vsw>

<vsw_instance>

<service_name>primary-vsw0</service_name>

<mac_address>0x00144ffb6ff9</mac_address>

<dev_path>e1000g0</dev_path>

<mode/>

</vsw_instance>

</vsw>

<network/>

<vds>

<vds_instance>

<service_name>primary-vds0</service_name>

<vds_volumes>

<vds_volume>

<block_dev>/dev/dsk/c1t1d0s2</block_dev>

<vol_name>vol1</vol_name>

<vol_opts/>

</vds_volume>

</vds_volumes>

</vds_instance>

</vds>

<disk/>

<vcc>

<vcc_instance>

<service_name>primary-vcc0</service_name>

<min_port>5000</min_port>

<max_port>5100</max_port>

</vcc_instance>

</vcc>

<console/>

<vdpcs/>

<vdpcc/>

<variables/>

<options/>

</ldom>

<ldom>

<ldom_name>guest1</ldom_name>

<cpu>

<number>8</number>

</cpu>

<mau><number>0</number>

</mau>

<memory>

<size>4294967296</size>

</memory>

<io/>

<mac_address>0x00144ffab8f0</mac_address>

<vsw/>

<network>

<network_instance>

<vnet_name>vnet1</vnet_name>

<service_name>primary-vsw0</service_name>

<mac_address>0x00144ff9159e</mac_address>

<mtu>1500</mtu>

</network_instance>

</network>

<vds/>

<disk>

<disk_instance>

<vdisk_name>vdisk1</vdisk_name>

<service_name>primary-vds0</service_name>

<vol_name>vol1</vol_name>

<timeout>-1</timeout>

</disk_instance>

</disk>

<vcc/>

<console>

<console_instance>

<service_name>primary-vcc0</service_name>

<group>guest1</group>

<port>5000</port>

</console_instance>

</console>

<vdpcs/>

<vdpcc/>

<variables>

<var>

<name>auto-boot?</name>

<value>true</value>

</var>

<var>

<name>boot-device</name>

<value>vdisk1</value>

</var>

<var>

<name>keyboard-layout</name>

<value>US-English</value>

</var>

</variables>

<options/>

</ldom>

</ldom_database>

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community, Oracle Solaris Installation, Booting and Patching Community.

Internal Comments
comments

This document contains normalized content and is managed by the the Domain Lead(s) of the respective domains. To notify content owners of a knowledge gap contained in this document, and/or prior to updating this document, please contact the domain engineers that are managing this document via the "Document Feedback" alias(es) listed below:

Domain Lead: [email protected]

LDOM, ldom, CMT, recovery, "logical domain"

Attachments

This solution has no attachment