Sun Storage 7000 Unified Storage System: How to generate NMI to collect a system core dump

Asset ID:	1-71-1173064.1
Update Date:	2012-10-03
Keywords:

Solution Type Technical Instruction Sure

Solution 1173064.1 : Sun Storage 7000 Unified Storage System: How to generate NMI to collect a system core dump

Applies to:

Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable and later
Sun ZFS Storage 7320 - Version Not Applicable and later
Sun ZFS Storage 7420 - Version Not Applicable and later
Information in this document applies to any platform.

Goal

How to generate a system core dump in case of system hang (BUI and CLI fails to respond) using NMI

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

Fix

Before collecting a system crash dump, try to retrieve some akd information as per Doc 1401288.1 : Storage 7000 Unified Storage System: Data collection for akd hang issue

The following will stop a hung system by generating a Non Mask-able Interrupt (NMI). It should force a core dump and reboot the node.

Soft NMI

Have 2 ssh sessions running, one to the console and the other to the Service Processor (SP).

From the ILOM 2.x revision :

-> cd /SP/diag

-> set generate_host_nmi=true

From the ILOM 3.x revision :

-> cd /HOST/

-> set generate_host_nmi=true

The console session should report something similar to the following:

panic[cpu2]/thread=ffffff001eccbc60: NMI received

 ffffff001eccbac0 pcplusmp:apic_nmi_intr+7c ()

 ffffff001eccbaf0 unix:av_dispatch_nmivect+30 ()

 ffffff001eccbb00 unix:nmiint+154 ()

 ffffff001eccbbf0 unix:mach_cpu_idle+b ()

 ffffff001eccbc20 unix:cpu_idle+c2 ()

 ffffff001eccbc40 unix:idle+114 ()

 ffffff001eccbc50 unix:thread_start+8 ()

 syncing file systems... done

 dumping to /dev/zvol/dsk/system/dump, offset 65536, content: kernel + curproc

 100% done: 356267 pages dumped, compression ratio 3.84, dump succeeded

Generate a bundle after the reboot and the core should be in the cores section of the bundle.

Refer to Sun Storage 7000 Unified Storage System: How to collect supportfile bundle using the BUI or CLI (Doc ID 1019887.1)

Hardware NMI

If Soft NMI is not even possible from the Service Processor (SP), you can press the NMI button located on the SP as shown below.

NMI switch location for for 7110, 7310 and 7410:
The Reset switch on the motherboard sends a reset order to the CPUs, resetting the main system, but not the service processor. The button for this switch is one of the 3 hidden (recessed) buttons on the back of the motherboard located between the NET MGT and NET0 connectors and closest to NET0. It can be pushed by sticking a paper clip or similar object through the hole provided on the rear of the chassis.

NMI switch location for 7210:
As written in the rear side of the 7210 (button in the middle).

NMI switch location for 7120 and 7320:
Face the rear of the 7120 or 7320. There are three recessed switch buttons (holes) between the "NET MGT" port to the left and the "NET 0" port of the 1GB NIC to the right. The middle button is the NMI reset switch. This button can be depressed using a straightened out paper clip.

NMI switch location for 7420:
Face the rear of the 7420. There are three recessed switch buttons (holes) between the "CLUSTER CARD" slot to the left and the "NET 0" port of the 1GB NIC to the right. The rightmost button nearest "NET 0" is the NMI reset switch. This button can be depressed using a straightened out paper clip.

Attachments

This solution has no attachment