When using a ZFS appliance with Exadata- Infiniband traffic is severely affected

Asset ID:	1-72-1351805.1
Update Date:	2012-07-24
Keywords:

Solution Type Problem Resolution Sure

Solution 1351805.1 : When using a ZFS appliance with Exadata- Infiniband traffic is severely affected

Related Items


Exadata Database Machine X2-2 Qtr Rack
 Sun ZFS Storage 7320
 Oracle Exadata Storage Server Software

Related Categories


PLA-Support>Database Technology>Database>Ad Hoc KM>Ad Hoc
 .Old GCS Categories>Infrastructure>LinuxOS>Storage>NFS
 .Old GCS Categories>ST>Server>Storage>ASM>Troubleshooting
 .Old GCS Categories>ST>Server>Engineered Systems>Exadata>Performance, Tuning and Hanging issues

After attaching a SUN 7XXX ZFS Appliance to an Exadata via Infiniband, all nodes on the Exadata system experience issues related to IB communications.

In this Document

Symptoms

Changes

Cause

Solution

Created from <SR 3-4248778631>

Applies to:

Sun ZFS Storage 7320 - Version Not Applicable and later
Exadata Database Machine X2-2 Qtr Rack - Version Not Applicable and later
Oracle Exadata Storage Server Software - Version 11.2.2.3.0 and later
Information in this document applies to any platform.
ASM has issues in communicating with the cells and celldisks:
ORA-15080: synchronous I/O operation to a disk failed

At the server level- the NFS mounts on the ZFS Storage server also suffer.
nfs:nfs_wait_on_write_congestion

The entire Exadata cluster will either crash or be severely impaired

Symptoms

ASM has issues in communicating with the cells and celldisks:

ORA-15080: synchronous I/O operation to a disk failed

At the storage server level- the NFS mounts on the ZFS Storage server also suffer.

In /var/log/messages:

nfs:nfs_wait_on_write_congestion

The entire Exadata cluster will either crash or be severely impaired

IB related messages will be found in many locations.

Changes

If the ZFS Storage Appliance is running on software release <Q3.3 , then this issue can occur.

Cause

The Firmware/Storage software on the ZFS Storage Appliance can cause IB network collisions.

Solution

Upgrade the ZFS Storage Appliance to software version Q3.3 or later.

Attachments

This solution has no attachment