Sun Storage 7000 Unified Storage System: aksh fatal error: no memory

Asset ID:	1-72-1325025.1
Update Date:	2012-07-22
Keywords:

Solution Type Problem Resolution Sure

Solution 1325025.1 : Sun Storage 7000 Unified Storage System: aksh fatal error: no memory

Applies to:

Sun ZFS Storage 7320 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7420 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

The message 'aksh fatal error: no memory' is seen when customer tried to login to CLI (Admin command Line interface). But all the data services are running fine and shares are accessible.

CLI login fails with error:

#ssh [email protected] Password: Last login: Mon May 9 02:00:31 2011 from 172.22.20.2 aksh fatal error: no memory

Also the BUI is not accessible, and fails with errors.

Exception type: coXmlrpcFault
Native message: malformed XML at line 1, column 0: out of memory
Mapped file: https://192.168.181.61:215/lib/crazyolait/coError.js line 37
Mapped stack trace:
:0 @Error()

Native file: https://192.168.181.61:215/lib/crazyolait/index.js line 364
Native stack trace:
Error()@:0
.@https://192.168.181.61:215/lib/crazyolait/index.js:364

Additional native members:
faultCode: 16
faultString: malformed XML at line 1, column 0: out of memory
coStack: coXmlrpcProxy.unmarshallDoc(doc:<object> "[object XMLDocument]", xml:"<methodResponse><fault>
<value><struct><member>
<name>faultCode</name>
<value><int>16</int></value>
</member>
<member>
<name>faultString</name>
<value><string>malformed XML at line 1, column 0: out of memory</string></value>

Cause

This error occurs when the appliance kit service akd runs out of memory.

Solution

Please vefify that the appliance is running the latest Appliance system software, as this issue is not seen in latest releases ( visit Software Updates page )

If already at the latest firmware then please contact Oracle Support to help resolve the issue.

Verify with customer if they are dropping into Solaris shell/Failsafe mode after logging into Appliance CLI.

If not even able to login to the shell and getting this error then the NMI dump should be collected and the system restarted as mentioned in <Document 1173064.1>.
After reboot the Appliance BUI/CLI should start working fine, later a support bundle should be run so that Oracle Support can analyze the dump.

Incase it is dropping to Failsafe/Solaris shell then verify the issue as follows.
1. Follow steps mentioned in <Document 1401288.1> to collect akd related logs.
2. Verify the physical memory and used memory :

# echo "::memstat" | mdb -k > /var/ak/dropbox/memstat.txt

3. Check the stash size

# df -h | sed -ne 1p -e /stash/p

4. Check the akd process size, if akd > 2.5 then it could be a concern.

# prstat -n 150 1 1 | sed -ne 1p -e /akd/p

5. Check if the analytics datasets are taking too much space

Bugs that may be of interest but not be directly related :

<SunBug 6914407> - akd should globally enable extended FILE stdio

<SunBug 6993881> - accrued analytics data can completely wedge akd

<SunBug 7157268> - large number of oversize allocations in akd fragments heap and akd runs out of memory

Back to <Document 1401282.1> Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI Hang).

References

SOFTWARE UPDATE PAGE: HTTPS://WIKIS.ORACLE.COM/DISPLAY/FISHWORKS/SOFTWARE+UPDATES
<NOTE:1173064.1> - Sun Storage 7000 Unified Storage System: How to generate NMI to collect a system core dump
@<NOTE:1401288.1> - Sun Storage 7000 Unified Storage System: Data collection for akd hang issues

Attachments

This solution has no attachment