VTL - Reclamation failed with error "Index Reclamation did not complete. 1 non-existent folders in garbage collect list"

Asset ID:	1-72-1353624.1
Update Date:	2011-09-22
Keywords:

Solution Type Problem Resolution Sure

Solution 1353624.1 : VTL - Reclamation failed with error "Index Reclamation did not complete. 1 non-existent folders in garbage collect list"

Applies to:

Sun StorageTek VTL Prime System - Version: 1.0 - Build 1813 to 1.1 - Build 2076 - Release: 1.0 to 1.0
Information in this document applies to any platform.

Symptoms

Backups failing due to limited VTL cache space
Reclamation failing for SIR database
Garbage collection folder missing or non-existent

Example messages leading up to reclamation error:

Aug 29 11:33:36 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635616||I||0x00004689||Index Reclamation in progress. Approximate percent completed: %1||95
Aug 29 11:35:55 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635755||I||0x0000c6d5||SIR Client initialization started.
Aug 29 11:35:55 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635755||I||0x0000c6d7||SIR Client initialized successfully.
Aug 29 11:36:09 vtl01 ipstorcomm: [ID 702911 daemon.notice] IPSTOR||1314635769||I||0x00002bd7||logged in successfully with read/write privileges||root - 172.16.66.156
Aug 29 11:36:18 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635778||I||0x0000c6d5||SIR Client initialization started.
Aug 29 11:36:19 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635779||I||0x0000c6d7||SIR Client initialized successfully.
Aug 29 11:36:19 vtl01 ipstorcomm [sir_util.c:SIRUTIL_getStatistic:1498][5753]: [ID 766872 daemon.notice] cmd:"/usr/local/vtl/bin/sirconfig stat > /usr/local/vtl/log/sir/sirstat ", ret=0
Aug 29 11:36:21 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635781||I||0x0000c6d5||SIR Client initialization started.
Aug 29 11:36:21 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635781||I||0x0000c6d7||SIR Client initialized successfully.
Aug 29 11:36:21 vtl01 ipstorcomm [sir_util.c:SIRUTIL_getStatistic:1498][5753]: [ID 766872 daemon.notice] cmd:"/usr/local/vtl/bin/sirconfig stat > /usr/local/vtl/log/sir/sirstat ", ret=0
Aug 29 11:37:20 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635840||I||0x0000c6d5||SIR Client initialization started.
Aug 29 11:37:20 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314635840||I||0x0000c6d7||SIR Client initialized successfully.
Aug 29 11:45:16 vtl01 rde: [ID 702911 daemon.notice] IPSTOR||1314636316||E||0x00004687||Index Reclamation did not complete. Reason %1||1 non-existent folders in garbage collect list
Aug 29 11:45:16 vtl01 rdereposit[6111]: [ID 259570 local3.notice] RDE check_garbage error: 1 non-existent folders in garbage collect list

Cause

1. The initial reason that reclamation failed was due to the garbage collection folder (under usr/local/etc/rde) for virtual tape "PROD038P" was missing.
The logs show this folder was present on Aug 28th at 07:16am, so RCA underway to try to find out why this folder went missing.

|D:2011-08-28 07:16:41|P: 2712|T: 1|L:I|O:Solaris| rde_exec_start: Created SIR folder 53:a2:cf:9f:1a:66:24:29:93:0d:21:1f:a3:60:f4:8c; Tape PROD038P; Drive 962FS00L02

This also corresponds to when customer backups stopped working due to space issues as a result of reclamation not working.

2. Also to the question of why the pre-scan and sir crawler processes didn't flag this missing folder and allowed reclamation to run until 95% done before failing (2-1/2 hrs later)?

Once the reclamation process is at 95%, it begins to actually delete which is why the last 5% takes the longest to complete in most cases. The problem in this case is that there was a folder found by crawler allowing it to complete successfully, which later was not found when it was time to delete the folder and at which time the GUID for that folder is placed into gc_missing_folders file in $ISHOME/etc/

Solution

1. First step is to identify the virtual tape associated with the missing folder.

    a. Check Xray for file called "gc_missing_folders"
        Alternatively, can check on customer's VTL server, look in /usr/local/vtl/etc

    b. Inside gc_missing_folders file will be GUID(s) for missing folder(s)

    c. Grep for this GUID (formatted with colon after each 2 bytes, see example below) in log files located in /usr/local/vtl/rde.
        Example:
        # grep -r "53:a2:cf:9f:1a:66:24:29:93:0d:21:1f:a3:60:f4:8c"
        Sample output returned:
        |D:2011-08-28 07:16:41|P: 2712|T: 1|L:I|O:Solaris| rde_exec_start: Created SIR folder 53:a2:cf:9f:1a:66:24:29:93:0d:21:1f:a3:60:f4:8c; Tape PROD038P; Drive 962FS00L02

    d. Record virtual tape barcode (in above example it is "PROD038P")

2. Next, get missing folder recreated:

a. VTL does have a fail safe for these folders, so try to manually submit a dedup jobs against virtual tape in question. When it reruns, and if VTL is able to reconcile the missing folder, the folder will be re-created and the dedup job will successfully complete.

    b. If step a fails, try to restore the data from the "suspect" tape found above and rewrite data to another virtual tape (via backup application); then delete the "suspect" tape.

c. If step b is not possible and data is not needed or can easily be recreated, just delete and reallocate the suspect tape via the VTL Console.

3. Rerun reclamation on SIR database (via VTL Console)

Attachments

This solution has no attachment