Sun Microsystems, Inc.  Sun System Handbook - ISO 4.1 October 2012 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-75-1402579.1
Update Date:2012-06-27
Keywords:

Solution Type  Troubleshooting Sure

Solution  1402579.1 :   Sun Storage 7000 Unified Storage System: How to Troubleshoot NFS Problems  


Related Items
  • Sun Storage 7310 Unified Storage System
  •  
  • Sun Storage 7410 Unified Storage System
  •  
  • Sun ZFS Storage 7120
  •  
  • Sun ZFS Storage 7320
  •  
  • Sun ZFS Storage 7420
  •  
  • Sun Storage 7110 Unified Storage System
  •  
  • Sun Storage 7210 Unified Storage System
  •  
Related Categories
  • PLA-Support>Sun Systems>DISK>NAS>SN-DK: 7xxx NAS
  •  
  • .Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
  •  




In this Document
Purpose
Troubleshooting Steps
 Framing the problem
 Gather useful data
 Common NFS problems
 Performance Problems:
 No Lock Available:
 NFSv4 Domain must match on appliance and clients:
 Shares accessed via NFSv4 hang whilst the same share accessed via NFSv3 or SMB is responsive
 Dropped Packets on Fast Networks:
 RHEL Clients using NFSv3 see errors like "bad nfs status return value: 88":
 Further help required:
References


Applies to:

Sun ZFS Storage 7320 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7420 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Purpose

This document is intended to provide a troubleshooting path for problems that involve loss of access to shares that are accessed by clients using the NFS protocol.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

Troubleshooting Steps

Problems with client access to the data shares via the NFS protocol can have many causes, many problems that at first glance may appear to be network connectivity problems can be due to other issues.  Links to other troubleshooting resolution path documents will be provided below where applicable.

Framing the problem

First it must be decided whether the problem is likely to be due to issues with the NFS protocol or elsewhere.
Some basic questions must be asked.

Is the problem with loss of connectivity to shares?
If the problem is due to an inability to read or write to a particular file or directory, but the share itself is mounted and other subdirectories or files are accessible, then this is likely to be a permissions issue that will be better covered in
<Document 1439378.1> "Sun Storage 7000 Unified Storage System: How to Troubleshoot UNIX/NFS File and Directory Permission Issues".

Does the problem involve clients that access the shares via other protocols than NFS?
If so check the General Network troubleshooting resolution path
<Document 1392086.1> "Sun Storage 7000 Unified Storage System: How to Troubleshoot Network Problems".

Gather useful data

Useful information to gather will include:

  • The NFS version(s) that the client(s) use.
  • The Directory Service(s) used
  • Can the appliance ping the client IP address and vice versa
  • Can the appliance resolve the client hostname and vice versa
  • Any errors generated from the client - the files to check will depend on the client operating system, examples would be
    • Linux - /var/log/messages
    • Solaris - /var/adm/messages
    • Windows - The "system log" viewable through the "Event Viewer".

At the very least a support bundle will be useful to examine the network configuration of the appliance. See
<Document 1019887.1> "Sun Storage 7000 Unified Storage System: How to collect supportfile bundle using the BUI or CLI".

It is more likely for complex issues that a network trace will need to be gathered to effectively troubleshoot the problem. A network trace will need to be gathered from the client side and the appliance side. For more details see
<Document 1398376.1> "Sun Storage 7000 Unified Storage System: How to get a network trace to assist in troubleshooting network problems".

If the problem is more of a performance type issue - e.g. I/O to the share takes longer and longer until the timeout is reached and the share becomes unavailable - then it would be useful to collect the recommended analytics datasets from
<Document 1230143.1> "Sun Storage 7000 Unified Storage System: Collecting Analytics data for NFS performance issues".

Common NFS problems

Performance Problems:

Many times a loss of connection to NFS shares may result from performance issues that may be due to a poor choice of configuration, or either an appliance-side or client-side tuning issue.
These scenarios are covered in detail in the following documents. To find recommendations on how best to initially configure the pool and hardware to maximize NFS performance see
<Document 1213725.1> "Sun Storage 7000 Unified Storage System: Configuration and tuning for NFS performance".

To find out about known NFS performance limitations, and what may be tuned on the appliance-side - such as the number of server threads, or on the client-side for different types of client or workload then check
<Document 1213739.1> "Sun Storage 7000 Unified Storage System: Known NFS performance issues".

No Lock Available:

If an error similar to "No locks available" is noticed when trying to mount a NFS share on a client, then the default number of NFS locks available on the appliance may need to be increased. This is controlled by the parameter LOCKD_SERVERS, for details on how to increase this see
<Document 1375378.1> "Sun Storage 7000 Unified Storage System: NFS Lock Issue".

If the problem is that a non-Solaris client cannot clear the NFS locks after a crash, or other problem that leaves a stale NFS lock then see <Bug 6858883> "Need way to clear NFS locks on behalf of non-Solaris NFS clients" for a workflow to do this

NFSv4 Domain must match on appliance and clients:

If NFSv4 is used then special attention must be paid to the concept of the NFS domain. This must match on both appliance and clients for the clients to be able to mount appliance shares over NFSv4. See
<Document 1409693.1> "Sun Storage 7000 Unified Storage System: NFSv4 clients cannot mount shares if NFSv4 identity domains do not match"

Shares accessed via NFSv4 hang whilst the same share accessed via NFSv3 or SMB is responsive

In an environment where clients mount filesystems from the ZFS Storage Appliance utilizing NFSv4 protocol it is seen that the requests for file access were hanging, and additionally commands such as ls and df were also unresponsive.  The same filesystems mounted via NFS v3 or SMB/CIFS are responsive. This is because the limit of entries for the OpenOwner table is exhausted. See
<Document 1385644.1> "Sun Storage 7000 Unified Storage System: OpenOwner Entries Count Increases on ZFS Storage Appliance".

Dropped Packets on Fast Networks:

If when using a fast network technology such as 10 Gigabit Ethernet or IPoIB it is found that NFS performance suffers with dropped packets and TCP transmission problems, then this may be due to the TCP buffer size on the appliance being set to too small a default value to cope with these fast technologies. More details on this problem and how to increase the TCP buffer size can be found here:
<Document 1401076.1> "Sun Storage 7000 Unified Storage System: How to increase the TCP buffer size to improve NFS performance in fast 10Gb Ethernet or IPoIB networks".

RHEL Clients using NFSv3 see errors like "bad nfs status return value: 88":

This is to do with NFSv4 using UTF-8 encoding but the clients mounting the share may not.  See:
<Document 1405993.1> "Sun Storage 7000 Unified Storage System: Nfs_stat_to_errno: Bad Nfs Status Return Value: 88".

Further help required:

At this point if the problem still exists and further troubleshooting is required please contact Oracle Support Services to raise a new Service Request (SR) via the My Oracle Support portal (https://support.oracle.com) or by telephoning your Oracle Global Support telephone number.

Please provide all data collected in the steps above to assist in the troubleshooting process.

 

Back to <Document 14164061.1> Sun ZFS Storage Appliances Troubleshooting Resource Center.

References

<NOTE:1375378.1> - Sun Storage 7000 Unified Storage System: NFS Lock Issue
<NOTE:1392086.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Network Problems
<NOTE:1019887.1> - Sun Storage 7000 Unified Storage System: How to collect a supportbundle using the BUI or CLI
<NOTE:1230143.1> - Sun Storage 7000 Unified Storage System: Collecting analytics data for NFS performance issues
<NOTE:1398376.1> - Sun Storage 7000 Unified Storage System: How to get a network trace to assist in troubleshooting network problems
<NOTE:1401076.1> - Sun Storage 7000 Unified Storage System: How to increase the TCP buffer size to improve NFS performance in fast 10Gb EThernet or IPoIB networks
<NOTE:1409693.1> - Sun Storage 7000 Unified Storage System: NFSv4 clients cannot mount shares if NFSv4 identity domains do not match
<NOTE:1213725.1> - Sun Storage 7000 Unified Storage System: Configuration and tuning for NFS performance
<NOTE:1213739.1> - Sun Storage 7000 Unified Storage System: Known NFS performance issues
<NOTE:1439378.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot UNIX/NFS file and directory permission issues
<NOTE:1416406.1> - Sun ZFS Storage Appliances Troubleshooting Resource Center
<NOTE:1385644.1> - Sun STorage 7000 Unifed Storage System: OpenOwner Entries count increases on ZFS Storage Appliance.
<NOTE:1405993.1> - Sun Storage 7000 Unified Storage System: Nfs_stat_to_errno: Bad Nfs Status Return Value: 88

Attachments
This solution has no attachment
  Copyright © 2012 Sun Microsystems, Inc.  All rights reserved.
 Feedback