Sun Microsystems, Inc.  Sun System Handbook - ISO 3.4 June 2011 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1007782.1
Update Date:2010-05-27
Keywords:

Solution Type  Problem Resolution Sure

Solution  1007782.1 :   Dynamic Reconfiguration or hot plug problems on a QGE HBA (4444A).  


Related Items
  • Sun Fire E6900 Server
  •  
  • Sun Fire E25K Server
  •  
  • Sun Fire E20K Server
  •  
  • Sun Fire 3800 Server
  •  
  • Sun Fire 6800 Server
  •  
  • Sun Fire E4900 Server
  •  
  • Sun Fire 12K Server
  •  
  • Sun Fire V880 Server
  •  
  • Sun Fire 4800 Server
  •  
  • Sun Fire 15K Server
  •  
  • Sun Fire 4810 Server
  •  
Related Categories
  • GCS>Sun Microsystems>Servers>Entry-Level Servers
  •  
  • GCS>Sun Microsystems>Servers>High-End Servers
  •  
  • GCS>Sun Microsystems>Servers>Midrange Servers
  •  

PreviouslyPublishedAs
210767


Symptoms
After unconfiguring a Quad Gigabit Ethernet (QGE) Host Bus Adaptor (HBA) and then configuring it back from within a domain using the cfgadm command, the NIC ports on the QGE HBA do not work anymore (i.e. plumb up) unless the domain is rebooted.
Execute the "devfsadm" command does not help, and "cfgadm" command shows everything works fine. However "prtdiag -v" command shows some noticeable differences on the slot when comparing the output of a functioning QGE HBA and a non-functioning QGE HBA after it has been unconfigured/configured.

Sample output of the encounter:

The QGE HBA is at IO16/Slot 0 and before performing DR:

 root@dm1c1h # cfgadm |grep pcisch5:e16b1slot0
pcisch5:e16b1slot0             pci-pci/hp   connected    configured   ok
 root@dm1c1h # prtdiag -v
[..]
========================= IO Cards =========================
                            Bus  Max
IO   Port Bus  Freq Bus  Dev,
Slot ID     Type  ID  Side MHz  Freq Func State Name                              Model
----------  ---- ---- ---- ---- ---- ---- ----- --------------------------------  ----------------------
......
/IO16/C5V0  PCI  540   B    33   33  1,0  ok    pci-pci8086,b154.0/bootbus-contr+ pci-bridge
/IO16/C5V0  PCI  540   B    33   33  2,0  ok    bootbus-controller-pci108e,c416/+ SUNW,sbbc
/IO16/C5V0  PCI  540   B    33   33  3,0  ok    pci108e,1100-pci108e,1100.1
/IO16/C5V0  PCI  540   B    33   33  3,1  ok    network-pci108e,1101.1            SUNW,pci-eri
/IO16/C5V0  PCI  540   B    33   33  3,2  ok    firewire-pci108e,1102.1001
/IO16/C5V0  PCI  540   B    33   33  3,3  ok    usb-pci108e,1103.1001
/IO16/C3V0  PCI  540   A    66   66  1,0  ok    pci-pci8086,b154.0/pci (pci)      pci-bridge
/IO16       PCI  540   A    66   66  0,0  ok    pci-pci8086,b154.0/network (netw+ pci-bridge
/IO16       PCI  540   A    66   66  0,0  ok    network-pci100b,35.30             SUNW,pci-qge
/IO16       PCI  540   A    66   66  1,0  ok    network-pci100b,35.30             SUNW,pci-qge
/IO16       PCI  540   A    66   66  4,0  ok    pci-pci8086,b154.0/network (netw+ pci-bridge
/IO16       PCI  540   A    66   66  2,0  ok    network-pci100b,35.30             SUNW,pci-qge
/IO16       PCI  540   A    66   66  3,0  ok    network-pci100b,35.30             SUNW,pci-qge
/IO16/C3V2  PCI  541   B    66   66  1,0  ok    pci-pci8086,b154.0/network (netw+ pci-bridge
/IO16/C3V2  PCI  541   B    66   66  0,0  ok    network-pci100b,35.30             SUNW,pci-ce
/IO16/C3V2  PCI  541   B    66   66  1,0  ok    network-pci100b,35.30             SUNW,pci-ce
/IO16/C3V2  PCI  541   B    66   66  2,0  ok    scsi-pci1000,b.7/disk (block)
/IO16/C3V2  PCI  541   B    66   66  2,1  ok    scsi-pci1000,b.7/disk (block)
/IO16/C3V1  PCI  541   A    66   66  1,0  ok    SUNW,qlc-pci1077,2312.1077.10a.2+
/IO16/C3V1  PCI  541   A    66   66  1,1  ok    SUNW,qlc-pci1077,2312.1077.10a.2+                       

Unconfigure IO16/Slot 0 (C3V0, a QGE HBA):

 root@dm1c1h # cfgadm -c unconfigure pcisch5:e16b1slot0
root@dm1c1h # cfgadm |grep pcisch5:e16b1slot0
pcisch5:e16b1slot0             unknown      connected    unconfigured unknown

After unconfiguring the QGE HBA, prtdiag does not show C3V0 entry anymore:

 root@dm1c1h # prtdiag -v
[..]
========================= IO Cards =========================
                            Bus  Max
IO   Port Bus  Freq Bus  Dev,
Slot ID     Type  ID  Side MHz  Freq Func State Name                              Model
----------  ---- ---- ---- ---- ---- ---- ----- --------------------------------  ----------------------
.....
/IO16/C5V0  PCI  540   B    33   33  1,0  ok    pci-pci8086,b154.0/bootbus-contr+ pci-bridge
/IO16/C5V0  PCI  540   B    33   33  2,0  ok    bootbus-controller-pci108e,c416/+ SUNW,sbbc
/IO16/C5V0  PCI  540   B    33   33  3,0  ok    pci108e,1100-pci108e,1100.1
/IO16/C5V0  PCI  540   B    33   33  3,1  ok    network-pci108e,1101.1            SUNW,pci-eri
/IO16/C5V0  PCI  540   B    33   33  3,2  ok    firewire-pci108e,1102.1001
/IO16/C5V0  PCI  540   B    33   33  3,3  ok    usb-pci108e,1103.1001
/IO16/C3V2  PCI  541   B    66   66  1,0  ok    pci-pci8086,b154.0/network (netw+ pci-bridge
/IO16/C3V2  PCI  541   B    66   66  0,0  ok    network-pci100b,35.30             SUNW,pci-ce
/IO16/C3V2  PCI  541   B    66   66  1,0  ok    network-pci100b,35.30             SUNW,pci-ce
/IO16/C3V2  PCI  541   B    66   66  2,0  ok    scsi-pci1000,b.7/disk (block)
/IO16/C3V2  PCI  541   B    66   66  2,1  ok    scsi-pci1000,b.7/disk (block)
/IO16/C3V1  PCI  541   A    66   66  1,0  ok    SUNW,qlc-pci1077,2312.1077.10a.2+
/IO16/C3V1  PCI  541   A    66   66  1,1  ok    SUNW,qlc-pci1077,2312.1077.10a.2+                       
 root@dm1c1h # ifconfig ce11 plumb
Sep 15 17:06:16 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_ATTACH_REQ(11), errno 8, unix 0
ifconfig: SIOCSLIFNAME for ip: ce11: no such interface
root@dm1c1h # Sep 15 17:06:16 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_BIND_REQ(1), errno 3, unix 0
Sep 15 17:06:16 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_PHYS_ADDR_REQ(49), errno 3, unix 0
Sep 15 17:06:16 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_UNBIND_REQ(2), errno 3, unix 0
Sep 15 17:06:16 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_DETACH_REQ(12), errno 3, unix 0

Configure the IO16/Slot 0 QGE HBA back into domain:

 root@dm1c1h # cfgadm -c configure pcisch5:e16b1slot0
root@dm1c1h # cfgadm |grep pcisch5:e16b1slot0
pcisch5:e16b1slot0             pci-pci/hp   connected    configured   ok

prtdiag now shows missing QGE entries for IO16/C3V0:

 root@dm1c1h # prtdiag -v
[..]
========================= IO Cards =========================
                            Bus  Max
IO   Port Bus  Freq Bus  Dev,
Slot ID     Type  ID  Side MHz  Freq Func State Name                              Model
----------  ---- ---- ---- ---- ---- ---- ----- --------------------------------  ----------------------
......
/IO16/C5V0  PCI  540   B    33   33  1,0  ok    pci-pci8086,b154.0/bootbus-contr+ pci-bridge
/IO16/C5V0  PCI  540   B    33   33  2,0  ok    bootbus-controller-pci108e,c416/+ SUNW,sbbc
/IO16/C5V0  PCI  540   B    33   33  3,0  ok    pci108e,1100-pci108e,1100.1
/IO16/C5V0  PCI  540   B    33   33  3,1  ok    network-pci108e,1101.1            SUNW,pci-eri
/IO16/C5V0  PCI  540   B    33   33  3,2  ok    firewire-pci108e,1102.1001
/IO16/C5V0  PCI  540   B    33   33  3,3  ok    usb-pci108e,1103.1001                                   
 /IO16/C3V0  PCI  540   A    66   66  1,0  ok    pci-pci8086,b154/pci (pci)        pci-bridge
/IO16       PCI  540   A    66   66  0,0  ok    pci-pci8086,b154                  pci-bridge
/IO16       PCI  540   A    66   66  4,0  ok    pci-pci8086,b154                  pci-bridge
/IO16/C3V2  PCI  541   B    66   66  1,0  ok    pci-pci8086,b154.0/network (netw+ pci-bridge
/IO16/C3V2  PCI  541   B    66   66  0,0  ok    network-pci100b,35.30             SUNW,pci-ce
/IO16/C3V2  PCI  541   B    66   66  1,0  ok    network-pci100b,35.30             SUNW,pci-ce
/IO16/C3V2  PCI  541   B    66   66  2,0  ok    scsi-pci1000,b.7/disk (block)
/IO16/C3V2  PCI  541   B    66   66  2,1  ok    scsi-pci1000,b.7/disk (block)
/IO16/C3V1  PCI  541   A    66   66  1,0  ok    SUNW,qlc-pci1077,2312.1077.10a.2+
/IO16/C3V1  PCI  541   A    66   66  1,1  ok    SUNW,qlc-pci1077,2312.1077.10a.2+                       
 root@dm1c1h # ifconfig ce11 plumb
Sep 15 17:09:22 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_ATTACH_REQ(11), errno 8, unix 0
ifconfig: SIOCSLIFNAME for ip: ce11: no such interface
root@dm1c1h # Sep 15 17:09:22 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_BIND_REQ(1), errno 3, unix 0
Sep 15 17:09:22 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_PHYS_ADDR_REQ(49), errno 3, unix 0
Sep 15 17:09:22 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_UNBIND_REQ(2), errno 3, unix 0
Sep 15 17:09:22 dm1c1h ip: ip_rput_dlpi(ce11): DL_ERROR_ACK for DL_DETACH_REQ(12), errno 3, unix 0

root@dm1c1h # devfsadm -C

Executing "devfsadm" command does not help and the interface "ce11"
still can not be plumbed up.



Resolution
The fix for QGE HBA hotplug patches are available on sunsolve:
Solaris[TM] 8 Patch ID: 110900-13 -- Post to sunsolve on 5/25/04

Solaris[TM] 9 Patch ID: 112838-13 -- Post to sunsolve on 5/25/04



Additional Information
The QGE HBA can not be used with the V1280/Netra[TM] 1280/E2900. This is
because the QGE is a long PCI card and will not mechanically fit into
the I/O cage for these servers.


Product
Sun Fire V880 Server
Operating Environments
Sun Fire E6900 Server
Sun Fire E4900 Server
Sun Fire 6800 Server
Sun Fire 4810 Server
Sun Fire 4800 Server
Sun Fire 3800 Server
Sun Fire E25K Server
Sun Fire E20K Server
Sun Fire 15K Server
Sun Fire 12K Server

Internal Comments
The following is strictly for Sun employees:

The problem is due to a known bug-  Bug ID: 4879904 .This bug applies
to all High-End servers Sun Fire[TM] 12K, E20K, SF15K and E20K servers
regardless of type of I/O assembly.
DR, Dynamic Reconfiguration, cfgadm, GQE, 4444A, Quad Gigabit Ethernet UTP
Previously Published As
78350

Change History
Date: 2004-09-28
User Name: 7058
Action: Approved
Comment: Trademarks in place.
Acronyms expanded as needed.
Checked patches and the patches do not describe the nature of the problem with examples as well as this doc so I think it should be published.

Attachments
This solution has no attachment
  Copyright © 2011 Sun Microsystems, Inc.  All rights reserved.
 Feedback