Solaris Volume Manager (SVM): How to Replace Internal FC-AL Disks in 280R, V480, V490, V880, V890, and E3500 Servers

Asset ID:	1-71-1010753.1
Update Date:	2012-07-31
Keywords:

Solution Type Technical Instruction Sure

Solution 1010753.1 : Solaris Volume Manager (SVM): How to Replace Internal FC-AL Disks in 280R, V480, V490, V880, V890, and E3500 Servers

Applies to:

Sun Fire V880 Server - Version Not Applicable and later
Sun Fire 280R Server - Version Not Applicable and later
Sun Fire V480 Server - Version Not Applicable and later
Sun Fire V490 Server - Version Not Applicable and later
Sun Enterprise 3500 Server - Version Not Applicable and later
All Platforms

Goal

This document provides a working example of how to replace a system's failed internal Fibre Channel Arbitrated Loop (FC-AL) disk when it is under Solaris Volume Manager (SVM) control. The most typical scenario involves a failed boot disk mirrored with SVM.

Fix

Let's start with a working mirrored pair of boot disks in a V490. In this example, the root and swap partitions are mirrored. Notice the optimal outputs from metastat, metadb and format.

# metastat



 d1: Mirror

     Submirror 0: d11

       State: Okay

     Submirror 1: d21

       State: Okay

     Size: 1052163 blocks (513 MB)



     d11: Submirror of d1

         State: Okay

         Size: 4194828 blocks (2.0 GB)

             Device     Start Block  Dbase        State Reloc Hot Spare

             c4t1d0s1          0     No            Okay   Yes

     d21: Submirror of d1

         State: Okay

         Size: 4194828 blocks (2.0 GB)

             Device     Start Block  Dbase        State Reloc Hot Spare

             c4t0d0s1          0     No            Okay   Yes



 d0: Mirror

     Submirror 0: d10

       State: Okay

     Submirror 1: d20

       State: Okay

     Size: 16629921 blocks (7.9 GB)



     d10: Submirror of d0

         State: Okay

         Size: 25166079 blocks (12 GB)

             Device     Start Block  Dbase        State Reloc Hot Spare

             c4t1d0s0          0     No            Okay   Yes

     d20: Submirror of d0

         State: Okay

         Size: 25166079 blocks (12 GB)

             Device     Start Block  Dbase        State Reloc Hot Spare

             c4t0d0s0          0     No            Okay   Yes



     Device Relocation Information:

     Device   Reloc  Device ID

     c4t0d0   Yes    id1,ssd@n20000004cf7fe655

     c4t1d0   Yes    id1,ssd@n20000004cf8f57c1



# format

      28. c4t0d0 <SEAGATE-ST336605FSUN36G-0438 cyl 24620 alt 2 hd 27 sec 107>

          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf7fe655,0

      29. c4t1d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>  

          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf8f57c1,0



# metadb

        flags           first blk       block count

     a m  p  luo        16              8192            /dev/dsk/c4t0d0s7

     a    p  luo        8208            8192            /dev/dsk/c4t0d0s7

     a    p  luo        16400           8192            /dev/dsk/c4t0d0s7

     a    p  luo        16              8192            /dev/dsk/c4t1d0s7

     a    p  luo        8208            8192            /dev/dsk/c4t1d0s7

     a    p  luo        16400           8192            /dev/dsk/c4t1d0s7

Disk c4t1d0 fails. Notice how the outputs have changed. Submirrors are in "Needs Maintenance", state database replicas have write errors and format output shows "drive type unknown". Other logs such as /var/adm/messages should be reviewed as well for evidence of failure.

# metastat 



 d1: Mirror

     Submirror 0: d11

       State: Needs maintenance

     Submirror 1: d21

       State: Okay

     Size: 1052163 blocks (513 MB)



     d11: Submirror of d1

         State: Needs maintenance

         Size: 4194828 blocks (2.0 GB)

             Device     Start Block  Dbase   State       Reloc Hot Spare

             c4t1d0s1          0     No     Maintenance   Yes

     d21: Submirror of d1

         State: Okay

         Size: 4194828 blocks (2.0 GB)

             Device     Start Block  Dbase   State Reloc  Hot Spare

             c4t0d0s1          0     No      Okay   Yes



 d0: Mirror

     Submirror 0: d10

       State: Needs maintenance

     Submirror 1: d20

       State: Okay

     Size: 16629921 blocks (7.9 GB)



     d10: Submirror of d0

         State: Needs maintenance

         Size: 25166079 blocks (12 GB)

             Device     Start Block  Dbase     State      Reloc Hot Spare

             c4t1d0s0          0     No      Maintenance   Yes

     d20: Submirror of d0

         State: Okay

         Size: 25166079 blocks (12 GB)

             Device     Start Block  Dbase        State Reloc Hot Spare

             c4t0d0s0          0     No            Okay   Yes



     Device Relocation Information:

     Device   Reloc  Device ID

     c4t0d0   Yes    id1,ssd@n20000004cf7fe655

     c4t1d0   Yes    id1,ssd@n20000004cf8f57c1



# format

      28. c4t0d0 <SEAGATE-ST336605FSUN36G-0438 cyl 24620 alt 2 hd 27 sec 107>

          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf7fe655,0

      29. c4t1d0 <drive type unknown>

          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf8f57c1,0



# metadb 

        flags           first blk       block count

     a m  p  luo        16              8192            /dev/dsk/c4t0d0s7

     a    p  luo        8208            8192            /dev/dsk/c4t0d0s7

     a    p  luo        16400           8192            /dev/dsk/c4t0d0s7

      W   p  l          16              8192            /dev/dsk/c4t1d0s7

      W   p  l          8208            8192            /dev/dsk/c4t1d0s7

      W   p  l          16400           8192            /dev/dsk/c4t1d0s7

If any submirror in the failed disk is still reporting as "Okay", that submirror should be detached. For example if d10 was still reporting "Okay":

# metadetach d0 d10

We can now proceed with the hardware replacement of the disk.

# luxadm remove_device -F /dev/rdsk/c4t1d0s2



   WARNING!!! Please ensure that no filesystems are mounted on these device(s).

   All data on these devices should have been backed up.

   The list of devices which will be removed is:

     1: Device name: /dev/rdsk/c4t1d0s2

        Node WWN:    20000004cf8f57c1

        Device Type:Disk device

        Device Paths:

        /dev/rdsk/c4t1d0s2

   Please verify the above list of devices and

   then enter 'c' or <CR> to Continue or 'q' to Quit. [Default: c]:

   stopping:  /dev/rdsk/c4t1d0s2....Done

   offlining: /dev/rdsk/c4t1d0s2....Done

   Hit <Return> after removing the device(s).

   ............

   Device: /dev/rdsk/c4t1d0s2 Removed.

For the v880 only, you may remove the device by enclosure name and slot number. For example:

# luxadm remove_device FCloop,s1

Physically replace the disk.

The next command will create the devices for the new drive.

# luxadm insert_device 

     Please hit <RETURN> when you have finished adding Fibre Channel Enclosure(s)/Device(s):

     Waiting for Loop Initialization to complete...

     New Logical Nodes under /dev/dsk and /dev/rdsk :

        c4t1d0s0

        c4t1d0s1

        c4t1d0s2

        c4t1d0s3

        c4t1d0s4

        c4t1d0s5

        c4t1d0s6

        c4t1d0s7

For the v880 only, you may add the device by enclosure name and slot number. For example:

# luxadm insert_device FCloop,s1

Alternatively, you can use the command devfsadm -Cv to create the devices. Depending on your version of Solaris, the picld daemon may make them for you as well. If all these fail, a reconfiguration reboot usually solves the problem.

Once the disk is labeled and partitioned in Solaris, we can finish up the repairs in SVM.

Prepare the partition table on the disk to match that of the disk it is mirrored with.

# prtvtoc /dev/rdsk/c4t0d0s2 | fmthard -s - /dev/rdsk/c4t1d0s2

    fmthard:  New volume table of contents now in place.

Restore the state database replicas.

# metadb -d c4t1d0s7

# metadb -a -c 3  c4t1d0s7

Remirror and monitor the resynch using the metastat command. Note that any submirror which was manually detached needs to be reattached. For example, if metadetach was used on d10, reattach it:

# metattach d0 d10

Otherwise, remirror with metareplace.

# metareplace -e d0  c4t1d0s0

   d0: device c4t1d0s0 is enabled

# metareplace -e d1  c4t1d0s1

   d1: device c4t1d0s1 is enabled

Update the replica database with the new device id of the disk. (Only if Device ID's are in use).

# metadevadm -u c4t1d0

Updating Solaris Volume Manager device relocation information for c4t1d0

Old device reloc information:

        id1,ssd@n20000004cf8f57c1

New device reloc information:

        id1,ssd@n20000004cf8c26a8

For further information, please refer to the following articles.

<Document 1007367.1> Removing and Replacing 280R, V480, V490, V880, V880z or V890 Hot-Pluggable Internal Disk Drives
<Document 1006196.1> Solaris Volume Manager (SVM): How To Use metadevadm To Maintain Device Relocation Information After Disk Replacement
<Document 1386408.1> Solaris Volume Manager (SVM): How To Copy The Partition Table From One Disk To Another

Want to know more?

You can find further details on this topic in the Solaris Volume Manager Administration Guide. View the HTML version in your browser, download the PDF version for later reference, or view the relevant section in the window below.

References

<NOTE:1006196.1> - Solaris Volume Manager (SVM): How To Use metadevadm To Maintain Device Relocation Information After Disk Replacement
<NOTE:1007367.1> - Removing and Replacing the Sun Fire[TM] 280R , Sun Fire[TM] V480 ,Sun Fire[TM] V490 ,Sun Fire[TM] V880 ,Sun Fire[TM] V880z or Sun Fire[TM] V890 Hot-Pluggable Internal Disk Drives.

Attachments

This solution has no attachment