Sun Storage 9990V System
 
Sun Storage 9970 System
 
Sun Storage 9985V System
 
Sun Storage 9990 System
 
Sun Storage 9960 System
 
Sun Storage 9985 System
 
Sun Storage 9980 System
 

GCS>Sun Microsystems>Storage - Disk>Datacenter Disk
 

Sun StorEdge 9900 series storage arrays can accommodate up to 1024  queued commands per fibre port, and up to 32 queued commands per Logical Unit Number(LUN). However, default settings for  Solaris[TM] Operating System (OS), permit these queue depths to be exceeded, potentially resulting in SCSI transport errors, and/or poor performance. The most common method for eliminating this problem, is to adjust the ssd_max_throttle setting of the Solaris target driver, also known as "global" throttling.

Overview of Queue Depth

This section provides an overview of queue depth.  Later sections provide information about the specifics of how to throttle.

If the Sun StorEdge 9900 series storage is appropriately configured, it is extremely difficult to overrun the queues.  Queue overrun is often associated with one or more of the following:

The most common method of throttling on the Solaris OS, involves the adjustment of a global variable(ssd_max_throttle), that limits the number of I/Os that can be queued to an individual LUN.  As noted in the ssd_max_throttle formula section of this document, the ssd_max_throttle value is multiplied by the total number of LUNs on the port, to determine the maximum number of I/Os that can be queued up for that port.  However, all LUNs on the port would need to be simultaneously busy, to reach the maximum given by(ssd_max_throttle * #LUNs). So, the administrator can tune ssd_max_throttle higher than the formula dictates, if the administrator is willing to risk that an unexpected workload spike could cause queue overrun

Ssd_max_throttle Formula

Table 1

To calculate ssd _max_throttle for a host, divide queue depth per port from Table 1 above, by the maximum number of LUNs presented to that host on a port. For example, if 182 LUNs are presented to a host on port 1A of a 9960, and 45 LUNs are presented on 1B, ssd_max_throttle is 256 / 182 = 1 (rounding to the lower integer).

To avoid overrunning the per-LUN queue depth, ssd_max_throttle should be set to a maximum of 32.  Because the Solaris ssd_max_throttle default is 256, always set ssd_max_throttle no higher than 32 with Sun StorEdge 9900 series storage, to avoid the possibility of overrunning the per-LUN queue.

Also note that ssd_max_throttle settings less than 8 may have a negative performance impact for some types of workloads that benefit from queueing. Generally, you should avoid the requirement for low throttle settings by limiting the number of LUNs configured on a port.

Global Method for Setting ssd_max_throttle

To set ssd_max_throttle, add the following line to the /etc/system file:

set ssd:ssd_max_throttle=x

where x is given by queue depth  / #LUNs (refer to Table 1).  A reboot is required to make the changes effective.

JNI (AMCC) and other third party HBAs sometimes bind with the parallel SCSI ( sd ) driver.  In this case, the variable to tune is "sd_max_throttle." Otherwise, the method for calculating and setting sd_max_throttle is exactly the same as shown previously.

To summarize:

[s]sd_max_throttle = queue depth / #LUNs, where [s]sd_max_throttle <= 32

SAN Considerations

When a single HBA connects to several Sun StorEdge 9900 series ports through a Storage Area Network (SAN), the port with the largest number of LUNs should dictate the host's ssd_max_throttle setting.

When multiple hosts connect to individual Sun StorEdge 9900  ports in a SAN, all attached hosts should set ssd_max_throttle according to the total number of LUNs presented on the port, even if LUN masking is in use so that some of the LUNs are not visible to certain hosts.

For example:

Sun StorEdge[TM] 9910 port 1A presents 64 LUNs.  Host "Fred" sees 32 and host "Barney" sees 32.  If ssd_max_throttle is set according to the number of LUNs visible on each host, instead of to the total number of LUNs present on the port, then Fred and Barney will each set ssd_max_throttle to 256 / 32 = 8.  However this setting means that Fred can initiate 32*8 = 256 I/Os, and Barney can initiate 32*8 = 256 I/Os, thus, easily overrunning the port's queue depth of 256.  Depending upon I/O pattern, the queue overrun may only manifest itself on one of the attached hosts.

Shared LUNs

If Sun StorEdge 9900 series LUNs are shared by active/active clusters, or by system software such as QFS, in which more than one host can initiate I/O to a LUN, divide the value of [s]sd_max_throttle by the number of hosts that share LUNs.  For example, with the  Sun StorEdge[TM] 9980 System port 1A presenting 32 LUNs, ssd_max_throttle would be:

9980 queue depth (512) / 32 = 16

Now assume that 2 hosts share all 32 LUNs.

Host A can initiate 16 * 32 I/Os = 512

Host B can initiate 16 * 32 I/Os = 512 (1024 total)

With shared LUNs there could be up to 1024 I/Os enqueued, which would exceed port capacity. To avoid this possibility, set ssd_max_throttle to queue depth / #LUNs / #hosts, or to continue with the SE9980 example:

512 / 32 / 2 = 8

To prevent calculations from becoming overly complex, avoid presenting both shared and unshared LUNs on the same port.

A Note on LUSE

The Logical Unit Size Expansion (LUSE) facility can be used to concatenate multiple LDEVs into a single large LUN.  Note that the resulting LUN is still subject to the same maximum queue depth of 32 on the array, as well as the host-based limit imposed by ssd_max_throttle, as any ordinary LUN.  If a larger queue depth is needed for a big volume, it can be obtained by using a host-based volume manager to stripe or concatenate multiple LUNs into a single large volume.  The queue depth for the resulting volume is effectively the sum of the queue depths for each component LUN (assuming workload is evenly distributed across the volume).

LUN Throttling in Driver Configuration Files

The global throttling method has the advantage of simplicity and ease of use, but also has the disadvantage of setting a global throttle value for all devices controlled by the target driver, perhaps unnecessarily restricting performance on devices that the administrator did not intend to throttle.

Many third party drivers have parameters (e.g. JNI's target_throttle) that allow more granular throttling by target, or groups of LUNs. This capability is very useful for SE9900 storage, which can present many different types of LUNs in large quantity.  Unfortunately the Solaris administrator using Sun HBAs is forced to prevent target queue overrun by globally limiting the number of I/Os that can be queued to an individual LUN. It is possible to throttle different types of SE9900 LUNs differently by setting bit flags in ssd.conf.  For example, OPEN-Es could be throttled with one setting, and OPEN-Vs could be throttled with another.  However this method has limited practical value in most situations, and the specifics are beyond the scope of this article.

This solution has no attachment