VMware ESX Reports Native Multipath Errors "nmp_CompleteCommandForPath:" and "Valid sense data: 0x5 0x94 0x1" After Array Controller Upgrade from Sun Storage 6140 to Sun Storage 6180

Asset ID:	1-75-1268023.1
Update Date:	2011-09-08
Keywords:

Solution Type Troubleshooting Sure

Solution 1268023.1 : VMware ESX Reports Native Multipath Errors "nmp_CompleteCommandForPath:" and "Valid sense data: 0x5 0x94 0x1" After Array Controller Upgrade from Sun Storage 6140 to Sun Storage 6180

Applies to:

Sun Storage 6180 Array - Version: Not Applicable to Not Applicable - Release: N/A to N/A
Information in this document applies to any platform.
Prerequisites:
Problem reported on ESX 4.0.0 version 261974, but may occur on other versions.
Sun Storage 6140 upgraded to 6180, or where upgrade has occurred where the SCSI identity string may change for existing LUNS in use by VMWare ESX servers.

Purpose

After upgrading a Sun Storage[TM] 6140 to 6180 there may be an issue with VMWare ESX hosts where the host may attempt to use the wrong path to access a LUN on the upgraded array.
The host may report messages such as:

Nov 2 15:04:46 host-esx4 vmkernel: 32:22:36:53.077 cpu2:5826)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x410001129b00) to NMP device
"naa.600a0b8000336f1200000a0849c9195d" failed on physical path "vmhba1:C0:T6:L14" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x94 0x1.

Which can be decoded at this external VMWare location which in summary for this instance is:

2/0 0x5 0x94 0x1 - ILLEGAL REQUEST - SCSI_ASC_INVALID_REQ_DUE_TO_CURRENT_LU_OWNERSHIP

This issue is related to the fact that the SCSI identity string for the array will change during the upgrade from CSM200_R to SUN_6180.
The ESX Host may not be aware of this new identity string, and so default to a multipath policy of Active/Active which is incompatible with this array, being Active/Passive.

Note: Other SCSI read errors may also be generated at the time due to failed IO's down the wrong path to a LUN, but the above error message is the one significant for this issue.

Last Review Date

November 19, 2010

Instructions for the Reader

A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.

Troubleshooting Details

The ESX hypervisor relies on the identity string to decide which storage array is connected. This is done within NMP (Native Multipathing) which consists of 2 entities:

SATP (Storage Array Type Plugin).
PSP (Path Selection Plugin).

As the names states, SATP is the part that looks at the identity string and decides which array type is connected and PSP decides the multipathing policy.
These 2 are interlinked, meaning that the storage array type will automatically set a default multipathing policy. For the Sun Storage[TM] 6180, the SATP should decide upon VMW_SATP_LSI, which has multipath policy MRU (Most Recently Used). Unfortunately, the new identity string was not recognized by the SATP and it decided to set its default type, which is VMW_SATP_DEFAULT_AA (active/active) and has multipath policy
FIXED. SCSI reads were performed on paths that were leading to the passive controller, resulting in the many SCSI read errors.

A possible workaround is to add the identity string to the VMW_SATP_LSI type, so that the storage is identified as an active/passive storage.

Below are the SATP types and their default PSP.

Name	Default	Description
VMW_SATP_ALUA_CX	VMW_PSP_FIXED	Supports EMC CX that use the ALUA protocol
VMW_SATP_SVC	VMW_PSP_FIXED	Supports IBM SVC
VMW_SATP_MSA	VMW_PSP_MRU	Supports HP MSA
VMW_SATP_EQL	VMW_PSP_FIXED	Supports EqualLogic arrays
VMW_SATP_INV	VMW_PSP_FIXED	Supports EMC Invista
VMW_SATP_SYMM	VMW_PSP_FIXED	Supports EMC Symmetrix
VMW_SATP_LSI	VMW_PSP_MRU	Supports LSI and other arrays compatible with the SIS 6.10 in non-AVT mode
VMW_SATP_EVA	VMW_PSP_FIXED	Supports HP EVA
VMW_SATP_DEFAULT_AP	VMW_PSP_MRU	Supports non-specific active/passive arrays
VMW_SATP_CX	VMW_PSP_MRU	Supports EMC CX that do not use the ALUA protocol
VMW_SATP_ALUA	VMW_PSP_MRU	Supports non-specific arrays that use the ALUA protocol
VMW_SATP_DEFAULT_AA	VMW_PSP_FIXED	Supports non-specific active/active arrays
VMW_SATP_LOCAL	VMW_PSP_FIXED	Supports direct attached devices

Note: This table is understood to be accurate at the time of writing though it may be subject to change and for authoritative information the customer should approach their VMWare support representative.

Attachments

This solution has no attachment