Sun Storage 7000 Unified Storage System: SP console access always gives "in use" message even after 'stop/(re)start'

Asset ID:	1-72-1495018.1
Update Date:	2012-09-30
Keywords:

Solution Type Problem Resolution Sure

Solution 1495018.1 : Sun Storage 7000 Unified Storage System: SP console access always gives "in use" message even after 'stop/(re)start'

Applies to:

Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Symptoms

After logging in to the Service Processor (SP) and attempting to start a console session the message that the console is in use is returned.

Attempting to stop and restart the console session does not free it up:


-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y

Serial console is in use.
-> stop /SP/console

Are you sure you want to stop /SP/console (y/n)? y



-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y

Serial console is in use.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - 7000 Series ZFS Appliances

Changes

A Console Management program has been installed on one of the systems used to access the Service Processor.

Cause

A Console Management program, such as ConMan, has been installed on one of the systems used to access the Service Processor.

The Console Management program is re-establishing the console connection after each time that the console session is stopped.

This happens so quickly that any other sessions cannot take hold of the seriallink process to take ownership of the console session.

Solution

Either only use the Console Management program to access a console session from the host that it is installed on, or uninstall the Console Management software entirely.

If you are not sure whether you have a host with a Console Management program installed but are seeing the symptoms described then please open a call with Oracle Support

who will be able to use a procedure to identify the IP address of any system that is continuously grabbing the lock on the serial link process, provided a remote support session can be provided via Shared Shell or Webex.

Use the following procedure to identify which host is reestablishing the console:
1) Log in as sunservice (passwd same as root) and check for the serial link lock file /var/lock/LCK..ttyS0, note the timestamp on this file if it exists.
2) Check "ps -ef" output for sshd processes to check if other sessions are logged in at the same time.
3) Check "netstat -an | grep :22" to find out where other sshd sessions have come from.
4) kill any unknown sshd sessions (not the one you logged in on), and repeat steps 1-3 again.
5) Identify IP address that is reestablishing the connection from the "netstat -an | grep 22" command.

Whilst logged in a sunservice open another session, or have the customer do so, log in as root and attempt to start the console to check problem is still happening:

-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y

Serial console is in use.



-> stop /SP/console

Are you sure you want to stop /SP/console (y/n)? y



-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y

Serial console is in use.

Back in the sunservice session check for existence and timestamp on lock file, along with current date/time:

[(flash)root@s7000-sp:/]# ls -l /var/lock/LCK..ttyS0

-rwSr-x--t    1 root     root            0 Sep 26 16:19 /var/lock/LCK..ttyS0

[(flash)root@s7000-sp:/]# date

Wed Sep 26 16:21:02 GMT 2012

Check for sshd sessions:

[(flash)root@s7000-sp:/]# ps -ef | grep sshd

 373 root        752 S   /usr/sbin/sshd -q

31038 root       1820 S   sshd: root@pts/0

21550 root       1832 S   sshd: root@pts/1

21843 root       1832 S   sshd: sunservice@pts/2

24062 root        492 S   grep sshd

Check where sessions are coming from:

[(flash)root@s7000-sp:/]# netstat -an | grep :22

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN

tcp        0      0 192.168.0.2:22        192.168.0.150:17007      ESTABLISHED

tcp        0      0 192.168.0.2:22        192.168.0.100:38923      ESTABLISHED

tcp        0      0 192.168.0.2:22        192.168.0.100:39265      ESTABLISHED

Kill unwanted sessions, ensure you know the IP address you are connected in from - so that you do not kill that session:

[(flash)root@s7000-sp:/]# kill 31038



[(flash)root@s7000-sp:/]# ps -ef | grep sshd

 373 root        752 S   /usr/sbin/sshd -q

21843 root       1832 S   sshd: sunservice@pts/2

28229 root       1820 S   sshd: root@pts/0

28556 root        492 S   grep sshd



[(flash)root@s7000-sp:/]# kill 28229



[(flash)root@s7000-sp:/]# ps -ef | grep sshd

 373 root        752 S   /usr/sbin/sshd -q

21843 root       1832 S   sshd: sunservice@pts/2

29186 root        492 S   grep sshd



[(flash)root@s7000-sp:/]# ps -ef | grep sshd

 373 root        752 S   /usr/sbin/sshd -q

21843 root       1832 S   sshd: sunservice@pts/2

29867 root       1820 S   sshd: root@pts/0                                                   <----- "this one came back"

30862 root        492 S   grep sshd

Identify where the session is coming from that is coming back:

[(flash)root@s7000-sp:/]# netstat -an | grep :22

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN

tcp        0      0 192.168.0.2:22        192.168.0.150:25953      ESTABLISHED <----"This is the client that is continususly re-establishing the console connection"

tcp        0      0 192.168.0.2:22        192.168.0.100:39265      ESTABLISHED

tcp        0      0 192.168.0.2:22        192.168.0.100:44409      ESTABLISHED

Attachments

This solution has no attachment