Sun Fire X2100 Server
 
Sun Ultra 20 M2 Workstation
 
Sun Fire X4140 Server
 
Sun Netra CT900 Server
 
Sun Java Workstation W2100z
 
Sun Fire X2100 M2 Server
 
Sun Fire X4240 Server
 
Sun Fire V20z Compute Grid Rack System
 
Sun Fire X4150 Server
 
Sun Fire V20z Server
 
Sun Netra X4200 M2 Server
 
Sun Ultra 40 Workstation
 
Sun Ultra 24 Workstation
 
Sun Fire X4450 Server
 
Sun Fire X2200 M2 Server
 
Sun Fire V40z Server
 
Sun Netra X4250 Server
 
Sun Ultra 20 Workstation
 
Sun Ultra 27 Workstation
 
Sun Ultra 40 M2 Workstation
 
Sun Fire X4540 Server
 
Sun Java Workstation W1100z
 
Sun Fire X4440 Server
 
Sun Fire X2250 Server
 
Sun Fire X4250 Server
 

PLA-Support>Sun Systems>x64>Server>SN-x64: MISC-SERVER
 

Applies to:

Purpose

Description

This document provides a high-level guide to troubleshooting documents for Oracle's Sun x64/x86 product line.

Contents
ILOM password reset
Sun System Handbook, Documentation, Downloads
Kernel Analysis
Fatal Reset
OS Panic
Hangs
OS Troubleshooting
Disk and Redundant Array of Independent/Inexpensive Disks (RAID) Troubleshooting
General Troubleshooting
IPMItool

Troubleshooting Steps

The product links above contain general information about the specific product. The Sun system handbook links from above contain system specifications, parts lists, documentation, and the list of minimum supported operating systems. System firmware, drivers, and BIOS can be downloaded via the Download link.

ILOM password reset

The following document shows how to reset the ILOM password back to its default    How to reset the ILOM root password back to the default 'changeme' using ipmitool. (Doc ID 1328316.1)

Kernel Analysis

A system becomes unresponsive for one of three reasons:

• Fatal reset (hardware detected)
• Operating system panic (software detected)
• Operating system/application hang (not detected)

The following document provides some information about the necessary data to gather:
DocID: 1010911.1 What to send to Sun[TM] after a system panic and/or unexpected reboot

Fatal Reset

Fatal Resets are hardware detected problems and are caused when the central processing unit (CPU) performs a trap which immediately drops to the BIOS.
One reason for this is due to a watchdog reset which is caused when the operating system fails to access the watchdog circuitry within its time out period.
This is really due to an operating system hang detected by the watchdog timer, so see the hang section below for techniques to diagnose.

Other reasons for fatal resets are due to hardware failure like loss of input voltage, or other major hardware related issues. No core file is saved and the messages file shows normal operation followed by an abrupt system restart (no shutdown messages).

The most important diagnosis information to retrieve is the following which his mostly gained through the service processor (SP):

• Console output. This typically contains a reason for the reset for example "sync flood" (or nothing for total power loss).
• SP events. This could contain sensor related events like under voltage conditions on one rail or OEM specific events like 0x12's.
• SP sensor data. This contains information if a sensor has a consistent problem like a voltage regulator or fan failure.
• SP field replaceable unit (FRU) data. This describes the hardware inventory configuration to assist with hardware replacement. Collect this to determine if the system has the proper configuration (eg. partially installed memory bank). A good item to check is the system board page in the Sun System Handbook.
• Explorer or other operating system data collector that contains the messages files and other data.

If the cause of the reset cannot be quickly determined, its important to perform hardware diagnostics such as a full power on self test (POST), the bundled PCcheck or SunVTS to determine if the hardware is stable.

PCcheck and other diagnostic tools can typically be downloaded via the Download link above or already be available as part of the BIOS boot menu.

OS Panic

OS Panics are software detected problems and caused when the operating system detects that the integrity of data is suspect or in danger of being corrupted. The panic routine will create a core dump if properly configured and place panic strings into the messages file to assist in fault isolation.

Panics can be caused by either operating system coding errors which are typically fixed by patches, or caused by hardware related problems like memory Uncorrectable Errors (UE's).

If software related, collect the core dump and pass to Sun's kernel group for analysis.
If hardware related then collect the following data so the problem can be isolated:

• Explorer or other operating system data collector that contains the messages files and other data. This typically contains panic messages and a stack trace related to the panic.
• SP events. This could contain sensor related events like under voltage conditions on one rail or OEM specific events like 0x12's.
• SP sensor data. This contains information if a sensor has a consistent problem like a voltage regulator or fan failure.
• SP FRU data. This describes the hardware inventory configuration to assist with hardware replacement. Collect this to determine if the system has the proper configuration (eg. partially installed memory bank). A good item to check is the system board page in the Sun System Handbook.

If the cause of the reset cannot be quickly determined, its important to perform hardware diagnostics such as a full power on self test (POST), the bundled PCcheck or SunVTS to determine if the hardware is stable. PCcheck and other diagnostic tools can typically be downloaded via the Download link above or already be available as part of the BIOS boot menu.

Hangs

A Hang is when some applications may operate properly, and others appear dead, but the hardware and operating system do not detect a problem. Hangs are caused by resource deadlocks due to operating system race conditions or resource deprivation due to one or more applications that are too needy. Sometimes console messages may indicate the source of the hang, but typically a core should be forced so that Sun's kernel group can analyze the data. There is a small possibility that hangs can be caused by hardware, but please contact the kernel group first for isolation.

DocID: 1012991.1 How to check if your x64 platform "system hang" actually is a system hang.

This document can be referenced to assist with possible hang situations.

The following operating system diagnostic section should be read to determine how to configure and force core dumps, but forcing a core dump from a hung system is not always possible.

OS Troubleshooting

Sun x86/x64 systems typically support Solaris[TM], Red Hat Enterprise Linux, SuSE Enterprise Linux and the Windows operating system.
Please check the Sun Systems Handbook to ensure that the operating system in question is supported on that platform.
A good overall operating system document to review is:

DocID: 1019144.1 Data Requirements reference: What data is needed in order to troubleshoot my software or hardware problem?

Solaris:
Six important Solaris documents that discuss procedures and configuration for Solaris panics and hangs are as follows:

DocID: 1012913.1 Troubleshooting Panics, dumps, hangs or crashes in the Solaris[TM] Operating System
DocID: 1001950.1 Troubleshooting Suspected Solaris Operating System Hangs
DocID: 1004506.1 How to force a crash when my machine is hung
DocID: 1001950.1 When to Force a Solaris[TM] System Core File
DocID: 1004530.1 KERNEL: How to enable deadman kernel code
DocID: 1003085.1 Solaris[TM] Operating System: Forcing a kernel core dump on an x86 or x64 system

Red Hat Linux:
Three important Red Hat documents that discuss procedures and configuration for Red Hat panics & hangs are as follows:

DocID: 1005528.1 How to configure Kdump on Red Hat Enterprise Linux 5 systems
DocID: 1006577.1 Red Hat Linux: Diskdump Pre-requisites, install and settings
DocID: 1007699.1 Crash Dump capturing for Red Hat Linux

SuSE Linux:
Two important SuSE documents that discuss procedures and configuration for SuSE panics & hangs are as follows:
DocID: 1108937.1 How to configure Kdump on SuSE Linux Enterprise System 10
DocID: 1010059.1 How to configure LKCD on SuSE Linux Enterprise Systems 8 and 9

Windows:
An important Windows document that discusses procedures and configuration for panics is:
DocID: 1007054.1 How to handle Microsoft Windows panics on x64 platforms

Additional documents that assist in Windows troubleshooting are:
DocID: 1011590.1 How to check for Windows platform disk errors and online/offline status
DocID: 1010936.1 Microsoft Windows and Linux operating systems: How to obtain troubleshooting information

Disk and Redundant Array of Independent/Inexpensive Disks (RAID) Troubleshooting

Disk and RAID problems are sometimes related to the disk/RAID controller firmware and boot configuration.

A good overall document to determine the firmware revision from systems with a supported operating system and how to search for known issues is:
DocID: 1008396.1 How to Identify Optical and Hard Disk Firmware Revisions for Checking of Known Issues

A good document on boot related issues is:
DocID: 1005506.1 How to verify your boot media exists and is bootable on a Sun Fire[TM] X4100/X4200/X4600 and M2 models Server

Once the version is known, the following document can be used to provide information of how to list, create, or delete RAID volumes:
DocID: 1005358.1 Hardware RAID usage on X64 based systems with the LSI SAS1064

The LSI RAID controller firmware requires 64MB unpartitioned disk space at the end of the disk for volume management. Thus, data backup prior any RAID creation should be performed.
LSI related RAID status can be obtained via the BIOS as shown in the following:

DocID: 1013107.1 How to Identify BIOS and Solaris[TM] Hardware RAID Status

Disks placed into a RAID volume should be of identical size to avoid problems.

RAID levels are:
RAID-0: Stripe of 2 or more disks to form a virtual larger disk. No redundancy so data lost on failure, but higher performance due to access to multiple disks for a file.
RAID-1: Mirrors of 2 or more disks to provide redundant data copies to prevent data loss on disk failure. Write performance decreases due to 2 or more writes per single file update but read performance increases due to access to file access from multiple disks.
RAID-01: Mirror of striped disks, but disk failure will offline its associated stripe.
RAID-10: Stripes of mirrored disks which can tolerate loss of two disks depending on configuration.
RAID-5: Stripes 3 or more disks with distributed parity so data loss is prevented if a disk fails. Medium performance is sustained since two writes are performed for each file update, but access is striped across multiple disks.

The Solaris raidctl command provides RAID status and provides RAID creation & deletion information as described in the following:

DocID: 1013107.1 How to Identify BIOS and Solaris[TM] Hardware RAID Status

Solaris commands that are helpful in disk troubleshooting, are as follows:

LINUX disk issues can be isolated using the following :
DocID: 1013003.1 How to Identify if a Linux Operating Environment is Installed on a Hardware RAID Controller

The following document indicates how to determine if a LINUX disk is under RAID control.
Software RAID is configured using mdadm as discussed in:
DocID: 1011427.1 How to setup software RAID in Linux

LINUX commands that are helpful in disk troubleshooting, are as follows: 

Windows disk status can be checked using information from the following:

DocID: 1011590.1 How to check for Windows platform disk errors and online/offline status

An example of a Windows RAID installation is obtained from:
DocID: 1009559.1 Installing Windows 2003 Server with RAID enabled on Sun Fire[TM] x2100

General Troubleshooting

For problems not covered by the prior two sections, collect the following information:

• Obtain SP related data in all cases. This can be done via ipmitool (see below), or via the SP's GUI or command line interfaces (if functionality exists; see SP link above).
• Ensure that the installed operating system is supported per the Sun System Handbook link above.
• When possible, obtain operating system data collectors such as explorer or other output that records the state of the operating system and file system (including messages files).
• PCcheck & other diagnostic tools can typically be downloaded via the Download link above or already be available as part of the BIOS boot menu.

IPMItool

IPMItool is a very useful tool that can gather information from the ILOM and other Service Processors (SP's).

Example commands to collect are as follows replacing the "ipaddress" with the address of the service processor, not the main platform:

@ Previously Published As 88276

References

This solution has no attachment