| Asset ID: |
1-75-1435063.1 |
| Update Date: | 2012-10-05 |
| Keywords: | |
Solution Type
Troubleshooting Sure
Solution
1435063.1
:
Sun Storage 7000 Unified Storage System: How to Troubleshoot Infiniband issues
| Related Items |
- Sun Storage 7410 Unified Storage System
- Sun Storage 7310 Unified Storage System
- Sun ZFS Storage 7120
- Sun Storage 7110 Unified Storage System
- Sun ZFS Storage 7320
- Sun ZFS Storage 7420
- Sun Storage 7210 Unified Storage System
|
| Related Categories |
- PLA-Support>Sun Systems>DISK>NAS>SN-DK: 7xxx NAS
- .Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
|
Applies to:
Sun ZFS Storage 7420 - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)
Purpose
To enable the ZFS Storage Appliance 7000 series administrator to troubleshoot problems with Inifiniband networking.
Troubleshooting Steps
Preliminary system checks
Before going through further steps, consider the following possible causes of issues. This may provide some easy clues.
- Check for (and investigate) any 'error/fault' system LEDs
- Check for (and resolve) any reported hardware faults/alerts
- Check for (and resolve) any reported system alerts
- Check for (and resolve) infiniband cables. Reseat them to avoid that they not seated properly
- 375-3696 [C] X4242A Sun InfiniBand Dual Port 4x QDR PCIe Low Profile Host Channel Adapter M2 User's Guide
- 375-3606 [C] X4237A-N [S] Sun Dual Port 40Gb/sec 4x Infiniband QDR Host Channel Adapter PCI Express User Guide
Symptoms
The System Log on the Appliance may show errors like:
- Network connectivity via datalink pffff_ibp0 has been lost. Major alert
- IP connectivity via interface ipmp1 has become degraded
- IP connectivity via interface pffff_ibp0 has been lost due to link-based failure
Framing the problem
Execute the following commands from the storage node to verify all data links and interfaces are up.
Check for IB hardware properties
s7420-ftlauder-a:maintenance hardware> select chassis-000
s7420-ftlauder-a:maintenance chassis-000> select slot
s7420-ftlauder-a:maintenance chassis-000> list
s7420-ftlauder-a:maintenance chassis-000> select slot-003 ( e.g. slot from QDR IB HCA )
s7420-ftlauder-a:maintenance chassis-000 slot-003> ls
Properties:
label = PCIe 4
present = true
faulted = false
manufacturer = Sun Microsystems, Inc.
model = Dual Port QDR IB HCA
part = 375-3606
revision = 2.6.0
type = hca
guid = 0x212800013f2416
Ports:
STATE GUID SPEED
Port 1 active 0x212800013f2417 QDR
Port 2 active 0x212800013f2418 QDR
s7420-ftlauder-a:maintenance chassis-000 slot-003>
Check if the device link status is "up"
s7420-ftlauder-a:> configuration net devices ls
Devices:
DEVICE UP SPEED MAC
ibp0 true 32000 Mbit/s not available
ibp1 true 32000 Mbit/s not available
igb0 true 1000 Mbit/s 0:21:28:3e:12:f8
igb1 true 100 Mbit/s 0:21:28:3e:12:f9
igb2 false 0 Mbit/s 0:21:28:3e:12:fa
igb3 true 100 Mbit/s 0:21:28:3e:12:fb
ixgbe0 false 0 Mbit/s 0:1b:21:81:4f:e4
ixgbe1 true 10000 Mbit/s 0:1b:21:81:4f:e5
s7420-ftlauder-a:>
s7420-ftlauder-a:> configuration net datalinks show
Datalinks:
DATALINK CLASS LINKS STATE LABEL
aggr1 aggregation igb2 up netxxx.168.10.130datalink
igb3
igb0 device igb0 up igb0-Datalink
ixgbe1 device ixgbe1 up net xxx.168.20.130datalinkpffff_ibp0 partition ibp0 up ib0_part
pffff_ibp1 partition ibp1 up ib1_part
s7420-ftlauder-a:>
If the "LINKS" shows status "down" please review the IB Switch Ports and the IB Cable + LED status.
Check physical links in IPMP group
s7420-ftlauder-a:> configuration net interfaces
s7420-ftlauder-a:configuration net interfaces> show
Interfaces:
INTERFACE STATE CLASS LINKS ADDRS LABEL
aggr1 up ip aggr1 xxx.168.10.130/24 netxxx.168.10.130interface
igb0 up ip igb0 xx.145.229.130/22 admin-igb0
ipmp1 up ipmp pffff_ibp0 xxx.168.2.200/24 ipmp_ib-data
pffff_ibp1
ixgbe1 up ip ixgbe1 xxx.168.20.130/24 netxxx.168.20.130interface
pffff_ibp0 up ip pffff_ibp0 0.0.0.0/8 ibp0-ipmp
pffff_ibp1 up ip pffff_ibp1 0.0.0.0/8 ibp1-ipmp
s7420-ftlauder-a:configuration net interfaces>
Check for IB interface properties
s7420-ftlauder-a:configuration net interfaces> select pffff_ibp0
s7420-ftlauder-a:configuration net interfaces pffff_ibp0> show
Properties:
state = up
curaddrs = 0.0.0.0/8
class = ip
label = ibp0-ipmp
enable = true
admin = true
links = pffff_ibp0
v4addrs = 0.0.0.0/8
v4dhcp = false
v6addrs =
v6dhcp = false
s7420-ftlauder-a:configuration net interfaces pffff_ibp0>
Check datalink properties
s7420-ftlauder-a:configuration net datalinks> select pffff_ibp1
s7420-ftlauder-a:configuration net datalinks pffff_ibp1> show
Properties:
class = partition
label = ib1_part
mac = 80:0:0:4a:fe:80:0:0:0:0:0:0:0:21:28:0:1:3f:24:1 8
links = ibp1
pkey = ffff
linkmode = cm
s7420-ftlauder-a:configuration net datalinks pffff_ibp1>
Manage routing configuration
s7420-ftlauder-a:configuration net routing> ls
Properties:
multihoming = adaptive
Routes:
ROUTE DESTINATION GATEWAY INTERFACE TYPE
route-000 0.0.0.0/0 xxx.168.2.1 ipmp1 static
route-001 0.0.0.0/0 xx.145.228.2 igb0 static
route-002 xx.145.228.0/22 xx.145.229.130 igb0 system
route-003 xxx.168.2.0/24 xxx.168.2.200 ipmp1 system
route-004 xxx.168.10.0/24 xxx.168.10.130 aggr1 system
route-005 xxx.168.20.0/24 xxx.168.20.130 ixgbe1 system
s7420-ftlauder-a:configuration net routing>
Check for provider using Infiniband
- SRP (SCSI Remote protocol or SCSI RDMA protocol)
s7420-ftlauder-a:configuration services srp> ls
Properties:
<status> = online
s7420-ftlauder-a:configuration services srp>
Sun ZFS Storage Appliances Troubleshooting Resource Center <Document 1416406.1>
Information Center: Sun ZFS Storage Appliances Overview Advisor <Document 1432269.2>
Introduction to troubleshhoting Infiniband issues from the OS shell or support bundle
One physical data link is created by default per port per HCA. The physical links can be used as administrative and observability data points. You can create IB partition links over the physical data links, similar to a creating VNICs over a NIC. Keep in mind that the physical data links are not used for data transfers, so plumbing and assigning an IB address is not supported on these links. Data is transferred on the partition data links.
Use the dladm show-phys command to display physical data link information on your system. The physical link state directly corresponds to the IB HCA port state.
Framing the problem at IPoIB Devices with :
- dladm
- netstat, arp -a
- ifconfig
- dladm, ipmpstat -i
- snoop
- kstat
- ping, ipmpstat -a, ipadm show-addr,
- and the raw cmd "datalink.listDevices" at the the appliance shell
- ls -l /dev/ib*
- cfgadm -a
- stmfadm list-target -v -> check for SRP SCSI Remote protocol or SCSI RDMA protocol that allows the use of SCSI devices across infiniband
- detailed instructions for data gathering ( diagnostic instructions for CR 7162888 )
- Infiniband IPMP Setup
- How do I set up Active/Active IPMP using the IB interface ? Doc ID 1354980.1
1. Display physical data link information
At the OS shell:
s7420-ftlauder-a#dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
igb2 Ethernet down 0 half igb2
ixgbe0 Ethernet down 0 unknown ixgbe0
igb0 Ethernet up 1000 full igb0
igb3 Ethernet up 100 full igb3
igb1 Ethernet up 100 full igb1
ixgbe1 Ethernet up 10000 full ixgbe1
ibp0 Infiniband up 32000 unknown ibp0
ibp1 Infiniband up 32000 unknown ibp1
s7420-ftlauder-a#
s7420-ftlauder-a#
LINK HCAGUID PORTGUID PORT STATE PKEYS
ibp0 212800013F2416 212800013F2417 1 up FFFF
ibp1 212800013F2416 212800013F2418 2 up FFFF
s7420-ftlauder-a#
At the appliance shell:
s7420-ftlauder-a:> datalink.listDevices()
result = ['igb2', 'ixgbe0', 'igb0', 'igb3', 'igb1', 'ixgbe1', 'ibp0', 'ibp1']
s7420-ftlauder-a:>
2. Check Interface Properties
At the OS shell:
s7420-ftlauder-a#ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
aggr1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet xxx.168.10.130 netmask ffffff00 broadcast xxx.168.10.255
ether 0:21:28:3e:12:fa
pffff_ibp0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 65520 index 3
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
groupname ipmp1
ipib 80:0:0:4b:fe:80:0:0:0:0:0:0:0:21:28:0:1:3f:24:17
pffff_ibp1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 65520 index 4
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
groupname ipmp1
ipib 80:0:0:4a:fe:80:0:0:0:0:0:0:0:21:28:0:1:3f:24:18
ixgbe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 index 5
inet xxx.168.20.130 netmask ffffff00 broadcast xxx.168.20.255
ether 0:1b:21:81:4f:e5
igb0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
inet xx.145.229.130 netmask fffffc00 broadcast xx.145.231.255
ether 0:21:28:3e:12:f8
ipmp1: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 65520 index 7
inet xxx.168.2.200 netmask ffffff00 broadcast xxx.168.2.255
groupname ipmp1
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
s7420-ftlauder-a#
3. Display IB Partition Link Information
At the OS shell:
s7420-ftlauder-a#dladm show-part
LINK PKEY OVER STATE FLAGS
pffff_ibp1 FFFF ibp1 up f---
pffff_ibp0 FFFF ibp0 up f---
s7420-ftlauder-a#
s7420-ftlauder-a#dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
igb2 phys 1500 down -- --
ixgbe0 phys 1500 down -- --
igb0 phys 1500 up -- --
igb3 phys 1500 up -- --
igb1 phys 1500 up -- --
ixgbe1 phys 9000 up -- --ibp0 phys 65520 up -- --
ibp1 phys 65520 up -- --
pffff_ibp1 part 65520 up -- ibp1
aggr1 aggr 1500 up -- igb2 igb3pffff_ibp0 part 65520 up -- ibp0
You have mail in /var/mail/root
s7420-ftlauder-a#
s7420-ftlauder-a# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
pffff_ibp1 yes ipmp1 ------- up disabled ok
pffff_ibp0 yes ipmp1 --mb--- up disabled ok
4. Check data links properties
At the appliance shell:
s7420-ftlauder-a:> datalink.getDeviceState('ibp0')
result = {
speed: 32000,
duplex: 0,
active: false,
up: true
}
s7420-ftlauder-a:>
5. Check Network status and packets
At the OS shell:
s7420-ftlauder-a# netstat -m -I ibp1
streams allocation:
cumulative allocation
current maximum total failures
streams 202 380 210639 0
queues 455 642 452524 0
mblk 62619 70618 5582704 0
dblk 62621 72573 61275041 0
linkblk 17 83 22 0
syncq 13 50 45 0
qband 2 62 2 0
15499 Kbytes allocated for streams data
s7420-ftlauder-a#
s7420-ftlauder-a# arp -a
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- -------- ---------------
pffff_ibp0 all-routers.mcast.net 255.255.255.255 S 00:ff:ff:ff:ff:10:40:1b:00:00:00:00:00:00:00:00:00:00:00:02
pffff_ibp0 igmp.mcast.net 255.255.255.255 S 00:ff:ff:ff:ff:10:40:1b:00:00:00:00:00:00:00:00:00:00:00:16
6. Check Network IPoIB
At the OS shell:
s7420-ftlauder-a# ping xx.145.231.255
xx.145.231.255 is alive
s7420-ftlauder-a#
s7420-ftlauder-a# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
aggr1/_a static ok 192.168.10.130/24
ixgbe1/_a static ok 192.168.20.130/24
igb0/_a static ok 10.145.229.130/22
ipmp1/? static ok 192.168.2.200/24
lo0/v6 static ok ::1/128
s7420-ftlauder-a# ipmpstat -a
ADDRESS STATE GROUP INBOUND OUTBOUND
192.168.2.200 up ipmp1 pffff_ibp0 pffff_ibp1 pffff_ibp0
s7420-ftlauder-a#
7. kstat displays kernel statistics
At the OS shell:
s7420-ftlauder-a# kstat ibp:1
module: ibp instance: 1
name: mac class: net
brdcstrcv 0
brdcstxmt 0
collisions 0
crtime 187.912473943
ierrors 0
ifspeed 32000000000
ipackets 0
ipackets64 0
link_state 1
link_up 1
multircv 0
multixmt 0
norcvbuf 0
noxmtbuf 0
obytes 0
obytes64 0
oerrors 0
oflo 0
opackets 0
opackets64 0
promisc 0
rbytes 0
rbytes64 0
snaptime 414417.102784163
uflo 0
unknowns 0
s7420-ftlauder-a#
8. The snoop program captures and inspects network packets:
At the OS shell:
s7420-ftlauder-a# snoop -d ibp1
9. Administering IPoIB Devices (dladm)
Check the following documentation from the Solaris 11 System Administration manual IB configuration
10. Check if device entry's are available
At the OS shell:
s7420-ftlauder-a# ls -l /dev/ib*
lrwxrwxrwx 1 root root 29 Aug 1 2011 /dev/ibp -> ../devices/pseudo/clone@0:ibp
lrwxrwxrwx 1 root root 69 Aug 1 2011 /dev/ibp0 -> ../devices/pci@0,0/pci8086,3410@9/pci15b3,673c@ 0/ibport@1,0,ipib:ibp0
lrwxrwxrwx 1 root root 69 Aug 1 2011 /dev/ibp1 -> ../devices/pci@0,0/pci8086,3410@9/pci15b3,673c@ 0/ibport@2,0,ipib:ibp1
s7420-ftlauder-a#
11. Check with cfgadm -a
At the OS shell:
s7420-ftlauder-a# cfgadm -a
Ap_Id Type Receptacle Occupant Condition
ib IB-Fabric connected configured ok
ib::212800013F2417,0,ipib IB-PORT connected configured ok
ib::212800013F2418,0,ipib IB-PORT connected configured ok
ib::iser,0 IB-PSEUDO connected configured ok
ib::rdsib,0 IB-PSEUDO connected configured ok
ib::rpcib,0 IB-PSEUDO connected configured ok
ib::sdpib,0 IB-PSEUDO connected configured ok
ib::srpt,0 IB-PSEUDO connected configured ok
12. Check for SRP (SCSI Remote protocol or SCSI RDMA protocol)
At the OS shell:
s7420-ftlauder-a# svcs | grep srp
online Apr_13 svc:/system/ibsrp/target:default
s7420-ftlauder-a# stmfadm list-target -v
Target: eui.00212800013F2416
Operational Status: Online
Provider Name : srpt
Alias : -
Protocol : SRP
Sessions : 4
Initiator: eui.0021280001A1318A
Alias: 81212800013f2418:0021280001a1318a
Logged in since: Fri Apr 13 20:36:08 2012
Initiator: eui.0021280001A1318A
Alias: 81212800013f2417:0021280001a1318a
Logged in since: Fri Apr 13 20:36:08 2012
Initiator: eui.0021280001A13189
Alias: 81212800013f2418:0021280001a13189
Logged in since: Fri Apr 13 20:36:08 2012
Initiator: eui.0021280001A13189
Alias: 81212800013f2417:0021280001a13189
Logged in since: Fri Apr 13 20:36:08 2012
Target: iqn.1986-03.com.sun:02:6e6a0308-11e3-ce9f-bad2-8e8023beccc1
Operational Status: Online
Provider Name : iscsit
Alias : -
Protocol : iSCSI
Sessions : 0
Target: iqn.1986-03.com.sun:02:521c6534-add3-4840-84f4-bca6ff9359a8
Operational Status: Offline
Provider Name : iscsit
Alias : -
Protocol : iSCSI
Sessions : 0
Christian Rasmussen highly recommended that if you get customers using the ZFSSA and IB to install the latest IDR for 2011.1.3
You can find the currently available (or soon to be available IDRs here) http://rpe.us.oracle.com/bin/view/Tech/AK_IDR_Information
Just an amendment here.
Most of the serious IB problems have been seen on 7420s, due to their reliance on the apix/apic driver
to deliver interrupts. Although the IDR contains fixes that would benefit customers on the 7320 as well, a proactive
installation of the IDR is probably not warranted on this platform.
7162888 IB infiniband interface stop communicating on both heads is now a duplicate of
7013410 apix: MSI interrupt is not delivered when interrupt remapping is active on X4470 server ( 12th-07-2012 )
IB Disconnects Exadata/7420 systems
- 7420 seems to be not hang
- Ethernet interfaces can still access the export.
- hosts will loos connectivity
- df commands hang on the NFS mount.
- connectivity problem appears to be happenning when the 7420 is idle
- when hung, the Exadata cannot ping the 7420 InfiniBand interface.
Verify the problem with
diagnostic instructions
Workaround:
- failover/reboot.
- RMAN backups appear then to run correctly
Fix:
Follow process described here to gain
access to IDR
Maintaining the InfiniBand Network
The InfiniBand network connects the database servers and Exadata Storage Servers through the BONDIB0 interface to the InfiniBand switches in the rack. This section describes how to perform maintenance on the InfiniBand switches.
Support information
Support for new InfiniBand adapter: 4242A and x4242A
As part of typical technology transition, the current card supported with ZFS Storage appliance (4237A-N, x4237A-N) has already been replaced with the new option and is already shipping with other Oracle products. The SW release 2010.Q3.3.1 release enables the use of the new adapter with the ZFS Storage Appliance.
Please note the following points:
- The last order date for 4237A-N (x-option: x4237A-N) option was 5/31/2011. 375-3606 [C]
- The replacement card is the 4242A (x-option: x4242A) 375-3696 [C]
- The SW release 2010.Q3.3.1 is required to support the new InfiniBand adapter
- The ATO-option 4242A will be added to the Configurator on June 21.
- For the period between 6/1 and 6/21, customers who require InfiniBand with a new Storage 7000 appliance will need to order the X-option part number x4242A
* What is the used MTU size of the ZFSSA on Infiniband ?
The MTU is 2044 in Unreliable Datagram mode (UD) and 65520 in Connected Mode (RC).
References
<BUG:7162888> - IB INFINIBAND INTERFACE STOP COMMUNICATING ON BOTH HEADS
<NOTE:1354980.1> - Sun Storage 7000 Unified Storage System: FAQ: RMAN backup from Exadata to Sun ZFS Backup Appliance
@ <BUG:6984798> - MODIFICATIONS MADE IN 7.2 ON FIELDS IN THE AUDIT RULES THAT WERE CREATED IN 7.1
@ <BUG:7023335> - RFI BACKPORT OF BUG 6320018 FOR INCLUSION IN 11.1.0.7.0 (RFI #82723)
<NOTE:1163758.1> - Gathering troubleshooting information for the Infiniband Switch (36 port, 72 port, 648 port Magnum, Exadata V2 Rack Network Switch models)
SOLARIS 11 ADMINISTERING IPOIB DEVICES: HTTP://DOCS.ORACLE.COM/CD/E19963-01/HTML/821-1459/GJWWE.HTML#GJWWD
<BUG:7173093> - 7410 IB NETWORK PERFORMANCE ISSUE (KERNEL MEMORY ISSUE?)
<BUG:7087415> - INFINIBAND NETWORK INTERFACE "HUNG" PRODUCING LOSS OF ACCESS FOR ALL CLIENTS
@ <BUG:6978400> - PLS BOUNCE THE MIDDLE TIER FOR CZ4MS204
@ <BUG:7146107> - PSE FOR BASE BUG 6944036 ON TOP OF 10.2.0.2.0 FOR HP-UX ITAN(197) (PSE #86250)
<BUG:6987187> - UNDER I/O LOAD, DATALINK:PFFFF_IBP0 DROP INTO MAINTENANCE WHEN TRYING TO CREATE ANOTHER IB DATALINK
@ <BUG:7066594> - ANALYTICS NEEDED FOR IB PORT
<BUG:6987187> - UNDER I/O LOAD, DATALINK:PFFFF_IBP0 DROP INTO MAINTENANCE WHEN TRYING TO CREATE ANOTHER IB DATALINK
@<NOTE:1489262.1> - Sun ZFS Storage Appliance: Infiniband clients report "NFS server not responding" messages (7420 only)
Attachments
This solution has no attachment
