Sun Storage 7000 Unified Storage System: SMB operations per second is very high

Asset ID:	1-72-1401557.1
Update Date:	2012-06-27
Keywords:

Solution Type Problem Resolution Sure

Solution 1401557.1 : Sun Storage 7000 Unified Storage System: SMB operations per second is very high

Applies to:

Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7320 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)
NAS head revision : [before 2010.Q3.3.1]
BIOS revision : [not dependent]
ILOM revision : [not dependent]
JBODs Model : [not dependent]
CLUSTER related : [not dependent]

Symptoms

A very high amount of SMB operations per sec which are caused be a storm of smb_nt_transact_notify_change messages.

See <Bug 6978261>

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

Cause

Background info on the smb_nt_transact_notify_change messages

The smb_nt_transact_notify_change messages are generated to notify the client that something has changed in a directory. For example the windows explorer application will use it to display and update automatically the content of an smb share.
The Windows client will send a request and the storage will notify only when something in the directory has changed (maybe 5 min later).
The bug is that the storage will respond DIRECTLY to the client (something changed) even if NOTHING changed. Windows will send then an other request (each time Windows receives a notify change response, it needs to send an other request for the next change in the directory) ... and the storage will respond directly ... causing a torrent or storm until you close the Windows explorer.

Verification of the high amount of smb_nt_transact_notify_change messages

dknco-stor2> status activity show

Activity:

CPU 23 %util Partly Cloudy

FTP 0 bytes/sec Sunny

Disk 2009 ops/sec Cloudy

iSCSI 31 ops/sec Partly Cloudy

NFSv3 90 ops/sec Sunny

NFSv4 0 ops/sec Sunny

Network 12.4M bytes/sec Partly Cloudy

SMB 27697 ops/sec Rainy

This shows 27697 SMB ops/sec for only 2009 Disk ops/sec.

The verification that the system is hitting this bug should be done by Oracle TSC by creating a service request and referencing this document.

The below dtrace script counts the number of notify_change requests done in 10sec. Having > 300000 (so 30000/s) is crazy. How can something change so quickly in a directory !
Normal value is not easy to find but is more likely that notify_change should no be more than other regular SMB requests. If nothing has changed, the storage should not respond to a notify change.

dknco-stor2# dtrace -n 'smb_nt_transact_notify_change:entry{ @[stack(60)] = count() }' -n 'tick-10sec{ exit(0) }'

dtrace: description 'smb_nt_transact_notify_change:entry' matched 1 probe

dtrace: description 'tick-10sec' matched 1 probe

CPU ID FUNCTION:NAME

1 78654 :tick-10sec 





smbsrv`smb_nt_trans_dispatch+0x13a

smbsrv`smb_com_nt_transact+0x172

smbsrv`smb_dispatch_request+0x472

smbsrv`smb_session_worker+0x4a

genunix`taskq_d_thread+0xb1

unix`thread_start+0x8

327531

There are 327531 messages per 10 sec generated, 32753 per/sec
This verifies we are hitting bug 6978261

Solution

Upgrade to at least AK-version 2010.08.17.3.1 (2010Q3.3.1).

See <Bug 6978261> for further details.

See http://wikis.sun.com/display/FishWorks/ak-2010.08.17.3.1+Release+Notes for further information.

Back to <Document 1408716.1> Sun Storage 7000 Unified Storage System: How to Troubleshoot SMB (CIFS) Problems.

References

@ <BUG:6978261> - NOTIFY CHANGE BUFFERING MECHANISM CAN CAUSE A TORRENT OF NOTIFY CHANGE REQUESTS/RESPONSES
<NOTE:1408716.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot SMB (CIFS) problems

Attachments

This solution has no attachment