Sun Fire[TM] 12K/15K/E20K/E25K: System Controller /var/adm/messages shows many eri "link up" (and sometimes "link down") messages

Asset ID:	1-72-1006491.1
Update Date:	2012-07-30
Keywords:

Solution Type Problem Resolution Sure

Solution 1006491.1 : Sun Fire[TM] 12K/15K/E20K/E25K: System Controller /var/adm/messages shows many eri "link up" (and sometimes "link down") messages

Applies to:

Sun Fire 12K Server
Sun Fire 15K Server
Sun Fire E20K Server
Sun Fire E25K Server
All Platforms

Symptoms

On high end Sun Fire[TM] servers, such as the Sun Fire[TM] 15K, many messages indicating that eri interfaces are "up" may be noticed in the /var/adm/messages file of the Main SC (System Controller).

Example:

Oct 13 16:03:32 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri7 : 100 Mbps half duplex link up
Oct 13 16:04:14 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri9 : 100 Mbps half duplex link up
Oct 13 16:04:55 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri8 : 100 Mbps half duplex link up
Oct 13 16:05:38 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri7 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri9 : 100 Mbps half duplex link up
Oct 13 16:07:00 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri8 : 100 Mbps half duplex link up
Oct 13 16:07:41 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri7 : 100 Mbps half duplex link up
Oct 13 16:08:22 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri9 : 100 Mbps half duplex link up
Oct 13 16:09:03 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri8 : 100 Mbps half duplex link up

Cause

This issue can have a number of potential causes, including bad hardware. However, experience has shown that the most likely cause of this issue is in fact a network configuration error.

Solution

The System controller's SMS (Systems Management Software) configuration includes a file that holds all of the *expected* IP addresses of the domain's I1 (Internal SC to Domain) network.

On the SC:

$ cat /etc/opt/SUNWSMS/config/MAN.cf

f15ka
C       SC-FLOATER-C1   f15ka-ssc       10.15.2.36
C       SC-TEST-hme0-C1 f15ka-sc1-hme0
C       SC-TEST-eri1-C1 f15ka-sc1-eri1
I2      NM-I2   netmask-i2      255.255.255.252
I2      SC0-I2  f15ka-sc0-i2    30.1.1.1
I2      SC1-I2  f15ka-sc1-i2    30.1.1.2
I1      NM-I1   netmask-i1      255.255.255.224
I1      SC-I1   f15ka-sc-i1      20.1.1.1
I1      DA-I1   f15ka-a 20.1.1.2    <------------------**NOTE!**
I1      DB-I1   f15ka-b 20.1.1.3
I1      DC-I1   f15ka-c 20.1.1.4
I1      DD-I1   f15ka-d 20.1.1.5

As shown above, it is expected that domain A should use the IP address 20.1.1.2 for its dman0 (I1 network) interface.

Moving to the domain, the configuration can be checked in a few ways:

f15ka-dom-a# ndd -get /dev/dman man_get_hostinfo
manc_magic = 0x4d414e43
manc_version = 01
manc_csum = 0x0
manc_ip_type = AF_INET
manc_dom_ipaddr = 20.1.1.2     <--------------**NOTE**
manc_dom_ip_netmask = 255.255.255.224
manc_dom_ip_netnum = 20.1.1.0
manc_sc_ipaddr = 20.1.1.1
manc_dom_eaddr = 0:0:be:a8:9:aa
manc_sc_eaddr = 8:0:20:f9:f7:40
manc_iob_bitmap = 0x38000       io boards = 15.1, 16.1, 17.1,
manc_golden_iob = 15

This command retrieves information from the IOSRAM (Input Output SRAM) which is the SC's CURRENT configuration for the domain's IP address.

Compare this against the file /etc/hostname.dman0 and the output of running the command ifconfig dman0.

f15ka-dom-a# ifconfig dman0
dman0: flags=1008843 mtu 1500 index 3
inet 20.1.1.3 netmask ffffffe0 broadcast 20.255.255.255
ether 0:0:be:a8:9:aa

f15ka-dom-a# cat /etc/hostname.dman0
20.1.1.3 netmask + broadcast + private up

It can seen above that the configuration does NOT agree. In this case, set the IP address to 20.1.1.2, as configured on the SC. This should be also be changed in the /etc/hostname.dman0 file.

Once the change has been made, reboot to see the change take effect (a good idea to check that the change was correctly made).

Alternatively, changes can be made on the fly with the following command:

f15ka-dom-a# ifconfig dman0 20.1.1.2 netmask 0xffffffe0 broadcast + up

In the event that it is demonstrated that the network configuration is correct, and all files, ndd and ifconfig have matching configuration, the possibility of missing patches and bad hardware should be considered.

Additional Information
Included all variants of eri2 through eri19 in the message to help this document be found in relevant cases.

Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri2 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri3 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri4 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri5 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri6 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri7 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri8 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri9 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri10 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri11 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri12 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri13 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri14 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri15 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri16 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri17 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri18 : 100 Mbps half duplex link up
Oct 13 16:06:19 f15ka-dom-a eri: [ID 517527 kern.info] SUNW,eri19 : 100 Mbps half duplex link up