Sun Blade 6000 System
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Interfaces
 
.Old GCS Categories>Sun Microsystems>Operating Systems>Solaris Network
 

In this Document

Purpose

Details

Product Support Team

Alerts

Description

Abbreviations/Acronyms

Versions

Hardware Versions

Software Versions

Compatibility/Patches

Hardware Compatibility Matrix

Software Compatibility Matrix

Configuration

Pathes/Packages

Configuration Files Location

Important configuration commands

FAQ

Can we use Link Aggregation with sxge ?

Information Gathering

Installation

De-installation

Troubleshooting

Performance

Lab

Contacts

Proactive

Further Reading

References

Oracle Confidential (PARTNER). Do not distribute to customers.
Reason: Solaris and Network Domain (SaND) Product Page, internal content

Applies to:

Purpose

This document contains the Virtualized 40 GbE Network Express Module (sxge) Product Page

Details

Product Support Team

PLA: SN-SND: Sun Network Interfaces (L2)

Alerts

Description

• The Sun Blade 6000 Virtualized 40 GbE Network Express Module (NEM) is a multi-purpose connectivity module for the Sun Blade 6000 modular system.
• The NEM supports connection to external devices through four 10 Gigabit Ethernet small form-factor pluggable (SFP)+ ports and connects server modules (blades) in a Sun Blade 6000 modular system chassis.
• The SFP+ ports can be used in 1x 10GBE, 2x10GBE or 1x 40GBE configuration
• The NEM's interface is visible on the blade system via the sxge driver.
• The NEM is also known as "SOL NEM" (project name).
• sxge is a multithreaded, loadable, clonable, GLDv3-based STREAMS driver.
• GLD is a multithreaded, clonable Generic LAN Driver support module which resides in a loadable kernel module, /kernel/misc/gld. It implements most of the STREAMS functions and DLPI(7P) functionality required of a LAN driver. Several Solaris network drivers are implemented using GLD.
• The NEM also provides the magnetics and RJ-45 connectors for the ten 10/100/1000 BASE-T Ethernet interfaces that come from server modules through the midplane. There is no active circuitry on the NEM for these GbE ports.
• The total number of Ethernet ports is 14.

Abbreviations/Acronyms

• 10GBE - 10 Gigabit Ethernet (according to IEEE 802.3ae)
• 40GBE - 40 Gigabit Ethernet (according to IEEE 802.3ba)
• FDX - Full Duplex: Capability to send and receive frames at the same time.
• HDX - Half Duplex: Either send or receive frames at given time.
• IEEE - Institute of Electrical and Electronics Engineers, Inc., standardizes Ethernet
• Jumbo Frame - An Ethernet frame which is larger than 1518 bytes (typical 9000 bytes).
• NIC - Network Interface Card
• VLAN - Virtual LAN, the ability to have several independent net segments within one hardware.
• LAN - Local Area Network, network which is geographically limited (typically to a 1 km radius, e.g. within a building)
• IEEE 802.3ae - 10 Gigabit Ethernet Standard

Versions

Hardware Versions

Software Versions

• unbundled sxge driver for older Solaris, Linux, Windows - obtain via MOS Patches and Updates
• Solaris bundled sxge driver startin with Solaris 10 U11

Note - do not overwrite the Solaris bundled version with downloaded unbundled version

Compatibility/Patches

Hardware Compatibility Matrix

• See System Handbook Blade 6000 Full Components Lists, Ethernet Section
• Check the NEM User's Guide and Release Notes for hardware requirements and restrictions.

Software Compatibility Matrix

Configuration

• Pathes/Packages

  • SUNWcarx - Core Architecture Root (64-bit), contains gld (SPARC)
  • SUNWcsr - Core Solaris Root, contains gld (x86)
  • SUNWdsxge - Contains the sxge driver
  • 5.11 pkg : driver/network/ethernet/sxge
• Path:
• Manual search path: /usr/man

• Configuration Files Location

  • /etc/hostname.sxge<instance>
  • sxge.conf, e.g. /kernel/drv/sxge.conf

• Important configuration commands

  • Configure Jumbo Frames
    Edit /kernel/drv/sxge.conf, remove comment before accept-jumbo = 1; and reboot :
    
    

    ...
    # To enable jumbo support,
    accept-jumbo = 1;
    #
    # To disable jumbo support,
    # accept-jumbo = 0;
    #
    # Default is 0.
    ...

  • Configure VLAN tagging (802.1q)
    • VLAN tags get enabled by using "magic instance numbers", consisting of the original instance and the VLAN tag.
    • boot persistant config: create /etc/hostname.hcge<vlan-tag><instance number>
    • The hostname in this file should be found in /etc/hosts
    • The VLAN tag must be the same on the switch the Hydra NEM is connected to

                                            instance # (last 3 digits, value=0 -> uses sxge0 interface)
                                            |||
                       /etc/hostname.sxge123000
                                         |||
    VLAN tag (digit 4 and higher, value=123)
    
    # cat /etc/hostname.sxge123000
    va64-x6440b-gmp03
    # grep va64-x6440b-gmp03 /etc/hosts
    129.156.213.163 va64-x6440b-gmp03
    # ifconfig sxge123000
    sxge123000: flags=sxge123000 mtu 1500 index 2
                inet 129.156.213.163 netmask fffff800 broadcast 129.156.215.255
                ether 0:21:28:29:90:69
    

FAQ

Can we use Link Aggregation with sxge ?

Answer:

As the switch port of the external connection will not accept several Aggregations per port, Link Aggregation can not be used.

Information Gathering

• Information needed to assign to an other support group or level:
  • Short failure description including error or warning messages
  • Short customer situation including timeframes and business impact
  • Platform, part# and revision of NIC and PCI slot (if obtainable)
  • Explorer output (or path to it's location)
  • Network overview (detailed sketch, cable lengths, networking hardware)
  • Where is a test system connected (e.g. a "slow" client)
  • Short description of your measures to find and solve the problem before the handover

Installation

• See User's Guide- it contains a section per supported OS.

De-installation

•  See User's Guide.

Troubleshooting

• get a problem object and defect (what is wrong with what)
• did it work once ?
• if so, what did change ?
• collect information as described at Information Gathering
• check ifconfig -a output if each interface has an unique MAC address
• in most cases eeprom local-mac-address\? should be true (SPARC)
• if several interfaces in same network are available on the system (e.g. an IPMP group): try to swap OK with not-OK cable connection
• try to connect an OK system directly for an back-to-back test
• try a different, tested cable
• check switch link settings, compare with the system's.
• check if packet counter increases via e.g.
  
  # kstat -p sxge:0 | grep packets64
  repeat twice
• ping an adjacent system, repeat the above
• snoop the interface,  inspect the output e.g
  
  # snoop -d sxge0 -s 128
• collect a binary snoop for later inspection from problem time, e.g.
  
  # snoop -d sxge0 -o /tmp/snoop.out -s 128
• check that all systems in the subnet have the same MTU size (e.g. all 1500 or all 9000)

Performance

• See Relase Notes of Chassis, NEM and Blade Version for Hardware Requirements.
• Ensure CPU capacity (cycles and cores) for sending and receiving frames. Recommendation: 2 CPUs per 10GBE NIC.
• Use latest Solaris version for improved TCP/IP stack.
• Use the Uperf traffic generator to measure throughput, start with a low thread number and increase as long as a performance gain is observed.

Lab

Use Global Lab / jsc

Contacts

• [email protected] mail alias - use for technical research 
• support_ethernet Mail Archive

Proactive

Further Reading

• Sun Blade 6000 Virtualized 40 GbE Network Express Module Documentation (SOL NEM) Document Library
• sxge(7D) man page
• External Marketing Page : http://www.oracle.com/goto/networking
• Virtualized 40 GbE NEM Marketing Page : http://www.oracle.com/us/products/servers-storage/servers/netra/6000-40gbe-nem/overview/
• TOI slides are attached, see below.

References

This solution has no attachment