CRS-4662 CRS-4000 Errors While Trying To Delete Crashed Node

Asset ID:	1-72-1381059.1
Update Date:	2012-02-09
Keywords:

Solution Type Problem Resolution Sure

Solution 1381059.1 : CRS-4662 CRS-4000 Errors While Trying To Delete Crashed Node

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.2 - Release: 11.2 to 11.2
Exadata Database Machine V2 - Version: Not Applicable and later [Release: N/A and later]
Information in this document applies to any platform.

Symptoms

A node in the cluster has crashed due to hardware or OS problems and is not recoverable. You are attempting to remove the node from the cluster, but you receive the following errors:

[root@dbm1db01 ~]# /u01/app/11.2.0.2/grid/bin/crsctl delete node -n dbm1db04
CRS-4662: Error while trying to delete node dbm1db04.
CRS-4000: Command Delete failed, or completed with errors.

Changes

One of the nodes has crashed irreversibly.

Cause

Bug 11705438 - CRSCTL DELETE NODE RECEIVES CRS-4662 AND CRS-4000

The voting file has become inconsistent and cannot be changed without the crashed node.

Solution

The bug is fixed in 11.2.0.2 GI PSU5, 11.2.0.3 and 12.1.

As a workaround, you can move the voting file to other location (another ASM diskgroup, supported cluster filesystem or supported NFS):

crsctl replace votedisk <new-location>

This can be done on a running cluster. You can check the removal by running olsnodes to see if the node is gone. You can also run the following cluster verify utility command:

[oraclenodedb01 bin]$ ./cluvfy stage -post nodedel -n nodedb03 -verbose

Performing post-checks for node removal

Checking CRS integrity...
The Oracle Clusterware is healthy on node "nodedb02"
The Oracle Clusterware is healthy on node "nodedb01"
The Oracle Clusterware is healthy on node "nodedb08"
The Oracle Clusterware is healthy on node "nodedb07"
The Oracle Clusterware is healthy on node "nodedb06"
The Oracle Clusterware is healthy on node "nodedb05"
The Oracle Clusterware is healthy on node "nodedb04"

CRS integrity check passed
Result:
Node removal check passed

Post-check for node removal was successful.

Please see OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) <Document 428681.1> for more details.

References

<BUG:11705438> - CRSCTL DELETE NODE RECEIVES CRS-4662 AND CRS-4000
<BUG:13330999> - CRS-4662: ERROR WHILE TRYING TO DELETE NODE DM01DB04
<NOTE:428681.1> - OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)

Attachments

This solution has no attachment