Sun Storage 7000 Unified Storage System: BUI/CLI hang due to 'excessive' amount of 'old' log files

Asset ID:	1-72-1408493.1
Update Date:	2012-07-09
Keywords:

Solution Type Problem Resolution Sure

Solution 1408493.1 : Sun Storage 7000 Unified Storage System: BUI/CLI hang due to 'excessive' amount of 'old' log files

Applies to:

Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

Symptoms observable by the customer:

Cannot login to BUI/CLI
Logging in via SP/console, appliance drops to 'failsafe' shell:

appliance/kit/akd: default:method "exec /usr/lib/ak/akd" failed with exit status 1.

Failure to configure storage pool - error message is "failed to initialize libzfs".
Replication failures

Symptoms observable by the Oracle Support engineer:

aklog may have the following message logged:

akd: failed to start appliance kit: failed to locate metadata schema fma: Too many open files

Replication failures may log similar messages in rm.ak.txt:

      Tue Aug 17 04:21:12 2010: failed to remote insert ak:/replication/action-7fef609f-5e95-c76a-fbf2-a04fd7615f7d: rpc com.sun.ak.stash.teleportFini:

                1 failed on remote peer: cannot access object com/sun/ak/replication_action/7fef609f-5e95-c76a-fbf2-a04fd7615f7d: No such file or directory

      Tue Aug 17 04:21:12 2010: import of ak:/replication/action-7fef609f-5e95-c76a-fbf2-a04fd7615f7d succeeded in 0.000s

The /var/ak/logs directory contains very many 'old' log files:

# cd /var/ak/logs

      # ls -ltr | wc -l

        496                            => 496 files



      # du -hs .

        41M   .                        => total size of logs directory = 41Mb



      # ls

      < snipped >

      alert.ak.40                httpd.txt.3                   scrk_curl.txt.16.gz

      alert.ak.41                httpd.txt.4                   scrk_curl.txt.2

      alert.ak.42                httpd.txt.5                   scrk_curl.txt.3

      alert.ak.43                httpd.txt.6.gz                scrk_curl.txt.4

      alert.ak.44                httpd.txt.7.gz                scrk_curl.txt.5

      alert.ak.45                httpd.txt.8.gz                scrk_curl.txt.6.gz

      alert.ak.46                httpd.txt.9.gz                scrk_curl.txt.7.gz

      alert.ak.47                jobs.txt.1                    scrk_curl.txt.8.gz

      alert.ak.48                jobs.txt.10.gz                scrk_curl.txt.9.gz

      < snipped >

Cause

This is a known issue. The root cause of this problem is too many open files by akd.
A few libraries (including libzfs) used by akd have a 256 file descriptor limit.

Known issue - <SunBug 6914407> - akd should globally enable extended FILE stdio (Fixed in 2010.Q1.0.0)

Solution

Recommended action for the customer:

For the fix for this issue, the Appliance Firmware Release should be upgraded to 2010.Q1.0.0 or later.
Oracle Support always recommend that you upgrade to the latest Appliance Firmware Release.

To remove the (excessive) 'old' log files, you will need to engage Oracle Support, by opening a Service Request, so that Oracle Support Services can provide confirmation of this issue and then carry out the appropriate activities to resolve the issue.

Recommended actions for the Oracle Support engineer:

Delete the 'old' log files from /var/ak/logs - see the following wiki document:

https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Solaris+shell+procedure+to+%27clean+up%27+%28and+rotate%29+log+files
If you cannot access this document engage NAS Storage-TSC for assistance

Back to <Document 1401282.1> Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface.

References

@ <BUG:6914407> - DUPLICATED DEPLOYABLE CHECKBOXES
@ <BUG:6966620> - QUICK OUTLINE:SELECT METHODS ICON, FIELDS DISPLAYED
SUPPORT WIKI - ROTATE LOG FILES: HTTPS://STBEEHIVE.ORACLE.COM/TEAMCOLLAB/WIKI/AMBERROADSUPPORT:SOLARIS+SHELL+PROCEDURE+TO+%27CLEAN+UP%27+%28AND+ROTATE%29+LOG+FILES
<NOTE:1401282.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang)

Attachments

This solution has no attachment