KMS - KMS 2.2 Software Has an Issue When Key Export is Used

Asset ID:	1-72-1338473.1
Update Date:	2011-07-18
Keywords:

Solution Type Problem Resolution Sure

Solution 1338473.1 : KMS - KMS 2.2 Software Has an Issue When Key Export is Used

Applies to:

Oracle Key Manager - Version: 2.2 and later [Release: 2.0 and later ]
Sun StorageTek Crypto Key Management System - Version: Not Applicable and later [Release: N/A and later]
Information in this document applies to any platform.

Symptoms

KMS 2.2 software has an issue when key export is used. The risk is slight for new KMS 2.2 installations but higher for customers upgrading to KMS 2.2.
For customers upgrading to KMS 2.2 once the upgrade completes on a particular KMA any drive access which appends data to one of these exported tapes with a new key from an upgraded KMA will trigger a replication update to the other (replication 8,9 or 10) KMAs causing them to crash.
Note:
Any tape exported over the lifetime of the KMS that is appended to may trigger this.

Once all KMAs in the KMS cluster have completed their upgrade to 2.2 the risk of this issue significantly drops.

Customers that have completed their 2.2 upgrade to all KMAs have a very slight possibility of encountering a problem when all of the following occur:
– A replication lag exists that is large enough to open up a timing window for the conflict described in the following bullet.
– A tape is exported from a KMA while a drive appends to that tape with a new key. These independent events would have to happen concurrently. The duration of a key export operation depends on the number of data units being exported so smaller exports (i.e. specifying fewer data units) shortens the duration and therefore minimizes the possibility for this conflict.

Due to the lag situation a merge conflict can then result when the conflicting updates are handled and cause a server crash. Merge conflicts are indicated in the KMS audit event history with an Operation of “Merge Updates” and Condition of “Update Overwritten due to Conflict” or “Update Suppressed due to Conflict”.

Changes

KMS firmware upgrade.

Cause

Risk Determination:
Use the KMS Manager GUI to determine if any Data Units have been exported . Use the Data Unit list
panel and filter on “Exported” with operator “equals” and value “True”. Go through this list of Data Units using the Details button and Key List tab to find situations where there are multiple keys. This will provide some evidence that your site is producing multi-key exported tapes, i.e. exported tapes are being appended to with new keys.
Key policy and tape access patterns affect the probability of having multi-key tapes. The shorter the encryption period the higher the likelihood of having a scenario where appending data will require a new key. Consider tape access patterns in light of the key policy encryption period. Multiple appends to a tape that span the policy encryption period will result in a multiple keys being associated with a tape.
Note:
Initialized tapes (e.g. scratch tapes that have just an encrypted volume label) residing in a scratch pool longer than the encryption period are prime candidates for being appended to with a new key when next allocated to an encrypting drive.

Solution

Engineering does not want any further accounts to upgrade to KMS 2.2. If you are already at KMS 2.2.
avoid performing exports during periods of heavy tape activity . New KMS 2.2 installations have a slight possibility of encountering this issue but the primary risk is for customers upgrading to 2.2.
If you must perform key exports then halt tape activity and wait for the replication lag to drop to 0 in order to avoid the merge conflict issue. Perform exports using a KMS Manager GUI connection to one of the down-level KMAs. Once exports have completed and lag is close to 0 then it is safe to resume tape activity. Replication is always fastest between KMAs on the same LAN, i.e. within the same site.
Therefore, the merge conflict window is smallest when key export is performed from the same site where the tapes are being used by drives. So, the recommendation is to perform exports from the same site as the drives that may append data to a tape.
If you encounter this problem you will likely see cluster performance degrade as KMAs go into a crash
loop. Temporarily remove the upgraded KMA(s) from the cluster by shutting them down and see if
performance improves. You can also examine the Audit Event List using the KMS Manager connected to a down-level KMA (i.e. one of the KMAs that are crashing) and filter the event history on “Operation” equals “Start Server”. Check the dates on these events to verify that they indicate numerous server restarts following the upgrade to KMS 2.2.
If symptoms match the above description then log the upgraded KMAs out of the KMS cluster by
resetting their passphrases. You can now power up those KMAs, reset them to factory default and rejoin the KMS cluster. A detailed recovery procedure is available from Oracle KMS engineering upon request and also attached to the CR.

KMS 2.2.2
KMS 2.2.2 is now available on MOS, customers should upgrade to this version.

NOTE:
See the 2.2.2 release notes for specific upgrade instructions as some KMAs have been, and still
are being, manufactured with KMS 2.2 code. Manufacturing will be switching to produce KMS 2.2.1
as soon as possible.

Provided by Stephen Patching.

Attachments

This solution has no attachment