Sun Microsystems, Inc.  Sun System Handbook - ISO 4.1 October 2012 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-75-1331769.1
Update Date:2012-06-15
Keywords:

Solution Type  Troubleshooting Sure

Solution  1331769.1 :   Sun Storage 7000 Unified Storage System: How to Troubleshoot Performance Issues  


Related Items
  • Sun Storage 7410 Unified Storage System
  •  
  • Sun ZFS Storage 7320
  •  
  • Sun Storage 7210 Unified Storage System
  •  
  • Sun Storage 7310 Unified Storage System
  •  
  • Sun ZFS Storage 7120
  •  
  • Sun ZFS Storage 7420
  •  
  • Sun Storage 7110 Unified Storage System
  •  
Related Categories
  • PLA-Support>Sun Systems>DISK>NAS>SN-DK: 7xxx NAS
  •  
  • .Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
  •  




In this Document
Purpose
Troubleshooting Steps
 Preliminary system checks
 Step 1 - Framing the problem
 Step 2 - Expected performance for various configurations.
 Step 3 - Configuration and tuning
 Step 4 - Known performance issues.
 Step 5 - Collecting analytics data for performance issues.
 Step 6 - other data collection.
References


Applies to:

Sun Storage 7410 Unified Storage System - Version Not Applicable and later
Sun ZFS Storage 7120 - Version Not Applicable and later
Sun ZFS Storage 7320 - Version Not Applicable and later
Sun ZFS Storage 7420 - Version Not Applicable and later
Sun Storage 7210 Unified Storage System - Version Not Applicable and later
7000 Appliance OS (Fishworks)
NAS head revision : [not dependent]
BIOS revision : [not dependent]
ILOM revision : [not dependent]
JBODs Model : [not dependent]
CLUSTER related : [not dependent]


Purpose

This document is provided to assist in troubleshooting performance issues.
It will help to frame the problem, determine what can be expected for various configurations, identifies some known issues and provide some guidelines to get the best possible performance for iSCSI, CIFS, NFS and FC protocols. Finally, it provides some information to collect analytics in order to isolate the issue and provide required information to ORACLE support team, if needed.
This document has been written as a resolution path, each step giving links to other specific documents. It should be kept as a reference for any performance issue on ZFS storage appliance.
In some situations when analytics is not enough, some dtrace scripts can be used to narrow down the issue. Performance and Applications Engineering team (PAE) has written a tool to help. See Step 6 : Other data collection.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

Troubleshooting Steps

Preliminary system checks

Before going through further steps, consider the following possible causes of performance issues. This may provide some easy clues.
  • Check for (and investigate) any 'error/fault' system LEDs
  • Check for (and resolve) any reported hardware faults/alerts
  • Check for (and resolve) any reported system alerts
  • Check for 'nearly' full (80%) filesystems
  • Check for 'nearly' full (80%) pool. When the capacity crosses the 80% threshold, we start using a "better fit" algorithm for writes, so it is a little slower. When we cross 96% full we use a "best fit" algorithm and writes are a lot slower. All storage systems get slower as they get full.
    Please refer to <Document 1392492.1> (Performance issue when pool is almost Full)

Step 1 - Framing the problem

In order to provide a 'complete picture' of the perceived performance problem, we should collect information about the 'nature' of the issue - including a measure of the performance we are currently seeing, details of the system workload, how the issue is manifesting itself and when the performance is being seen.
Please refer to <Document 1213705.1> (Sun Storage 7000 Unified Storage System: Performance issues - Framing the problem)

This information will be used in subsequent steps in this troubleshooting document and will also be very useful to engineers should the issue need escalating into Oracle Support.

Possible outcomes :
If the system performance has never been good/acceptable, jump to step 2 : Expected performance for various configurations.
If system performance was good/acceptable and has now fallen, jump to step 4 : known performance issues.

Step 2 - Expected performance for various configurations.

To determine if there MAY be a performance issue on your system, it will be useful to compare the 'measured' performance metric for your current workload/system configuration with an 'expected performance' metric for an 'equivalent' reference workload/configuration.

Please refer to <Document 1213714.1> (Sun Storage 7000 Unified Storage System: Performance clues and considerations)

Step 3 - Configuration and tuning

The following documents provide details of the system configuration choices that affect performance. It provides guidance on how to configure the system (and possibly, clients) to maximise performance for either 'latency' or 'throughput'.
  • NFS   : Please refer to <Document 1213725.1> (Sun Storage 7000 Unified Storage System: Configuration and tuning for NFS performance)
  • CIFS  : Please refer to <Document 1213723.1> (Sun Storage 7000 Unified Storage System: Configuration and tuning for CIFS performance)
  • iSCSI : Please refer to <Document 1175573.1> (Sun Storage 7000 Unified Storage System: Configuration and tuning for iSCSI performance)

Step 4 - Known performance issues.

The following documents provides details of any 'known' issues regarding performance.
These will include information on any product limitations (features not yet supported/implemented), references to any 'outstanding' bugs (bugs not yet fixed in the latest appliance firmware release), and known workarounds for some issues.

The following NAS wiki pages can be of interest :
memory issues : https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Possible+memory+issues
fragmentation issues : https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Spacemaps+or+How+to+check+Fragmentation
dedup usage : https://stbeehive.oracle.com/teamcollab/wiki/Best+Practise+Guide:Performance+issue+-+DDT+table+usage
  • NFS     : Please refer to <Document 1213739.1> (Sun Storage 7000 Unified Storage System: Known NFS performance issues)
  • CIFS    : Please refer to <Document 1213735.1> (Sun Storage 7000 Unified Storage System: Known CIFS performance issues)
  • iSCSI   : Please refer to <Document 1374509.1> (Sun Storage 7000 Unified Storage System: Known iSCSI performance issues)

Step 5 - Collecting analytics data for performance issues.

The 7000 Series NAS appliance range provides a very powerful performance analysis tool called 'Analytics'.

Analytics is an advanced facility to graph a variety of statistics in real-time and record this data for later viewing. It has been designed for both long term monitoring and short term analysis. When needed, it makes use of DTrace to dynamically create custom statistics, which allows different layers of the operating system stack to be analyzed in detail.
The Analytics User Guide is available here : http://docs.oracle.com/cd/E26765_01/html/E26398/index.html
The embedded appliance wiki help pages are also a useful reference to get started : https://<yournasipaddress>:215/wiki/index.php/Analytics
Note that since 2011.1.3, a retention period can be set for analytics, avoiding too much space usage in system pool.
@ The following NAS wiki page can be of interest :
Assess performance using analytics : https://stbeehive.oracle.com/teamcollab/wiki/Best+Practise+Guide:Best+Practice+-+assess+performance+using+analytics
  • NFS   : Please refer to <Document 1230143.1> (Sun Storage 7000 Unified Storage System: Collecting analytics data for NFS performance issues)
  • CIFS  : Please refer to <Document 1230145.1> (Sun Storage 7000 Unified Storage System: Collecting analytics data for CIFS performance issues)
  • iSCSI : Please refer to <Document 1229193.1> (Sun Storage 7000 Unified Storage System: Collecting analytics data for iSCSI performance issues)
  • FC    : Please refer to <Document 1375010.1> (Sun Storage 7000 Unified Storage System: Collecting analytics data for Fiber Channel performance issues)

Step 6 - other data collection.

In some scenarios, some other data collection have to be done from inside or outside the NAS head.
Some network trace might be needed to narrow down the performance issue. A workflow can be used to collect snoop trace from inside the NAS head. A network trace may have to be collected from the client side as well. Some times, some specific dtrace scripts might be run. In such case, please contact the support services.
  • Solaris hosts : Please refer to <Document 1285485.1> (GUDS - A Script for Gathering Solaris Performance Data)

Back to <Document 1416406.1> ZFS Storage Appliance Product Information Center

References

PAE DTRACE PERFORMANCE SCRIPTS: HTTPS://PAE.US.ORACLE.COM/TWIKI/BIN/VIEW/PUBLIC/ZFSPERFORMANCEDIAGNOSIS
<NOTE:1230145.1> - Sun Storage 7000 Unified Storage System: Collecting analytics data for CIFS performance issues
<NOTE:1213723.1> - Sun Storage 7000 Unified Storage System: Configuration and tuning for CIFS performance
<NOTE:1175573.1> - Sun Storage 7000 Unified Storage System: Configuration and tuning for iSCSI performance
<NOTE:1213739.1> - Sun Storage 7000 Unified Storage System: Known NFS performance issues
<NOTE:1229193.1> - Sun Storage 7000 Unified Storage System: Collecting analytics data for iSCSI performance issues
<NOTE:1285485.1> - GUDS - A Script for Gathering Solaris Performance Data
ANALYTICS USER GUIDE: HTTP://DOCS.ORACLE.COM/CD/E26765_01/HTML/E26398/INDEX.HTML
<NOTE:1375010.1> - Sun Storage 7000 Unified Storage System: Collecting analytics data for Fibre Channel performance issues
<NOTE:1213735.1> - Sun Storage 7000 Unified Storage System: Known CIFS performance issues
<NOTE:1374509.1> - Sun Storage 7000 Unified Storage System: Known iSCSI performance limitations
<NOTE:1230143.1> - Sun Storage 7000 Unified Storage System: Collecting analytics data for NFS performance issues
AMBER ROAD WIKI PAGE ON PAE DTRACE SCRIPTS: HTTPS://STBEEHIVE.ORACLE.COM/TEAMCOLLAB/WIKI/AMBERROADSUPPORT:PAE+-+ZFS+PERF+DIAG
<NOTE:1392492.1> - Sun Storage 7000 Unified Storage System: Performance issue when pool is almost Full
<NOTE:1213705.1> - Sun Storage 7000 Unified Storage System: Performance issues - Framing the problem
<NOTE:1213725.1> - Sun Storage 7000 Unified Storage System: Configuration and tuning for NFS performance
<NOTE:1213714.1> - Sun ZFS Storage Appliance: Performance clues and considerations

Attachments
This solution has no attachment
  Copyright © 2012 Sun Microsystems, Inc.  All rights reserved.
 Feedback