Sun Blade 6000 System
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Interfaces
 
.Old GCS Categories>Sun Microsystems>Boards>Ethernet 10GbE
 
.Old GCS Categories>Sun Microsystems>Operating Systems>Solaris Network
 

In this Document

Purpose

Details

Product Support Team

Alerts

Description

Abbreviations/Acronyms

Versions

Hardware Versions

Software Versions

Compatibility/Patches

Hardware Compatibility Matrix

Software Compatibility Matrix

Configuration

Pathes/Packages

Configuration Files Location

Important configuration commands

FAQ

Can we use Link Aggregation with hxge/Hydra ?

Information Gathering

Installation

De-installation

Troubleshooting

Performance

Lab

Contacts

Proactive

Further Reading

Oracle Confidential (PARTNER). Do not distribute to customers.
Reason: Solaris and Network Domain (SaND) Product Page, internal content

Applies to:

Purpose

This document contains the Virtualized Multi-Fabric 10GbE Network Express Module (hxge) Product Page

Details

Product Support Team

PLA: SN-SND: Sun Network Interfaces (L2)

Alerts

FAB <Document 1222158.1> NEMHydra's Main Power Shuts Down Unexpectedly
<Document 1381298.1> hxge traffic intermittently and partially stops flowing, resulting in access issues to blades

Description

• The Sun Blade 6000 Virtualized Multi-Fabric 10GbE Network Express Module (NEM) is a multi-purpose connectivity module for the Sun Blade 6000 modular system.
• The NEM supports connection to external devices through two 10 Gigabit Ethernet small form-factor pluggable (SFP)+ ports and connects server modules (blades) in a Sun Blade 6000 modular system chassis.
• It also offers 10 10/100/1000 twisted-pair Ethernet (TPE) ports (pass-through) and 4 storage SAS ports (not supported).
• Total number of external ports is 16.
• The NEM's interface is visible on the blade system via the hxge driver.
• The NEM is also known as "Smart Link Virtualized NEM"
• hxge is a multithreaded, loadable, clonable, GLDv3-based STREAMS driver.
• GLD is a multithreaded, clonable Generic LAN Driver support module which resides in a loadable kernel module, /kernel/misc/gld. It implements most of the STREAMS functions and DLPI(7P) functionality required of a LAN driver. Several Solaris network drivers are implemented using GLD.

Abbreviations/Acronyms

• 10GBE - 10 Gigabit Ethernet (according to IEEE 802.3ae)
• FDX - Full Duplex: Capability to send and receive frames at the same time.
• HDX - Half Duplex: Either send or receive frames at given time.
• IEEE - Institute of Electrical and Electronics Engineers, Inc., standardizes Ethernet
• Jumbo Frame - An Ethernet frame which is larger than 1518 bytes (typical 9000 bytes).
• NIC - Network Interface Card
• VLAN - Virtual LAN, the ability to have several independent net segments within one hardware.
• LAN - Local Area Network, network which is geographically limited (typically to a 1 km radius, e.g. within a building)
• IEEE 802.3ae - 10 Gigabit Ethernet Standard

Versions

Hardware Versions

Software Versions

• unbundled hxge driver for older Solaris, Linux, Windows - obtain via MOS Patches and Updates
• Solaris bundled hxge driver

Note - do not overwrite the Solaris bundled version with downloaded unbundled version

Compatibility/Patches

Hardware Compatibility Matrix

• See System Handbook Blade 6000 Full Components Lists, Ethernet Section
• Check the NEM User's Guide and Release Notes for hardware restrictions

Software Compatibility Matrix

Configuration

• Pathes/Packages

  • SUNWcarx - Core Architecture Root (64-bit), contains gld (SPARC)
  • SUNWcsr - Core Solaris Root, contains gld (x86)
  • SUNWhxge - Contains the hxge driver
• Path:
• Manual search path: /usr/man

• Configuration Files Location

  • /etc/hostname.hxge<instance>
  • hxge.conf, e.g. /kernel/drv/hxge.conf

• Important configuration commands

  • Configure Jumbo Frames
    Edit /kernel/drv/hxge.conf, remove comment before accept-jumbo = 1; and reboot :
    
    

    ...
    # To enable jumbo support,
    accept-jumbo = 1;
    #
    # To disable jumbo support,
    # accept-jumbo = 0;
    #
    # Default is 0.
    ...

  • Configure VLAN tagging (802.1q)
    • VLAN tags get enabled by using "magic instance numbers", consisting of the original instance and the VLAN tag.
    • boot persistant config: create /etc/hostname.hcge<vlan-tag><instance number>
    • The hostname in this file should be found in /etc/hosts
    • The VLAN tag must be the same on the switch the Hydra NEM is connected to

                                            instance # (last 3 digits, value=0 -> uses hxge0 interface)
                                            |||
                       /etc/hostname.hxge123000
                                         |||
    VLAN tag (digit 4 and higher, value=123)
    
    # cat /etc/hostname.hxge123000
    va64-x6440b-gmp03
    # grep va64-x6440b-gmp03 /etc/hosts
    129.156.213.163 va64-x6440b-gmp03
    # ifconfig hxge123000
    hxge123000: flags=hxge123000 mtu 1500 index 2
                inet 129.156.213.163 netmask fffff800 broadcast 129.156.215.255
                ether 0:21:28:29:90:69
    

FAQ

Can we use Link Aggregation with hxge/Hydra ?

Answer:

As the switch port of the external connection will not accept several Aggregations per port, Link Aggregation can not be used.

Information Gathering

• Information needed to assign to an other support group or level:
  • Short failure description including error or warning messages
  • Short customer situation including timeframes and business impact
  • Platform, part# and revision of NIC and PCI slot (if obtainable)
  • Explorer output (or path to it's location)
  • Network overview (detailed sketch, cable lengths, networking hardware)
  • Where is a test system connected (e.g. a "slow" client)
  • Short description of your measures to find and solve the problem before the handover

Installation

De-installation

Troubleshooting

• get a problem object and defect (what is wrong with what)
• did it work once ?
• if so, what did change ?
• collect information as described at Information Gathering
• check ifconfig -a output if each interface has an unique MAC address
• in most cases eeprom local-mac-address\? should be true (SPARC)
• if several interfaces in same network are available on the system (e.g. an IPMP group): try to swap OK with not-OK cable connection
• try to connect an OK system directly for an back-to-back test
• try a different, tested cable
• check switch link settings, compare with the system's.
• check if packet counter increases via e.g.
  
  # kstat -p hxge:0 | grep packets64
  repeat twice
• ping an adjacent system, repeat the above
• snoop the interface,  inspect the output e.g
  
  # snoop -d hxge0 -s 128
• collect a binary snoop for later inspection from problem time, e.g.
  
  # snoop -d hxge0 -o /tmp/snoop.out -s 128
• check that all systems in the subnet have the same MTU size (e.g. all 1500 or all 9000)

Performance

• See Relase Notes of Chassis, NEM and Blade Version for Hardware Requirements.
• Ensure CPU capacity (cycles and cores) for sending and receiving frames. Recommendation: 2 CPUs per 10GBE NIC.
• Use latest Solaris version for improved TCP/IP stack.
• Use the Uperf traffic generator to measure throughput, start with a low thread number and increase as long as a performance gain is observed.

Lab

Use Global Lab / jsc

Contacts

Proactive

Further Reading

• 16-Port Virtualized Multi-Fabric Express Module (Hydra)  Document Library
• 16-Port Virtualized Multi-Fabric M2 Express Module (Hydra2)  Document Library 
• hxge(7D) man page
• External Marketing Page : http://www.oracle.com/goto/networking
• TOI slides :
• Blade 6000 Modules Overview (internal, pdf)
• Hydra slides (internal, pdf)
• Hydra2 slides (internal, pdf)

This solution has no attachment