Roll forward fails on hotspare db with (831), (832), (833), and (11014) after offline PROBKUP performed crash recovery

Roll forward fails on hotspare db with (831), (832), (833), and (11014) after offline PROBKUP performed crash recovery

Previous roll forward had an in-flight transaction which carried over to the next AI file.

Database log file:

RFUTIL   : (451)   Roll forward session begin for root on batch. 
RFUTIL   : (7129)  Usr 0 set name to root. 
RFUTIL   : (15321) Before Image Log Initialization at block 995  offset 1791. 
RFUTIL   : (660)   Beginning roll forward of after-image file <AI file>. 
RFUTIL   : (1633)  After-image dates for this after-image file: 
RFUTIL   : (1640)      Last AIMAGE BEGIN Sun Jun 22 09:38:39 2014 
RFUTIL   : (1641)      Last AIMAGE NEW Fri Nov 21 12:35:56 2014 
RFUTIL   : (1642)      This is aimage file number 12903 since the last AIMAGE BEGIN. 
RFUTIL   : (1643)      This file was last opened for output on Fri Nov 21 12:35:56 2014. 
RFUTIL   : (15109) At Database close the number of live transactions is 1. 
RFUTIL   : (1634)  72 notes were processed. 
RFUTIL   : (3785)  0 in-flight transactions. 
RFUTIL   : (1635)  6 transactions were started. 
RFUTIL   : (11138) 5 transactions were completed. 
RFUTIL   : (1636)  At the end of the .ai file, 1 transactions were still active. 
RFUTIL   : (662)   Roll forward completed. 
RFUTIL   : (334)   rfutil -C roll forward session end. 

BACKUP   : (451)   Probkup session begin for root on batch. 
BACKUP   : (5326)  Begin Physical Redo Phase at 896 . 
BACKUP   : (7161)  Physical Redo Phase Completed at blk 995 off 7981 upd 29. 
BACKUP   : (13547) At end of Physical redo, transaction table size is 5445. 
BACKUP   : (7163)  Begin Physical Undo 1 transactions at block 995 offset 8017 
BACKUP   : (5331)  Physical Undo Phase Completed at 995 . 
BACKUP   : (7162)  Begin Logical Undo Phase, 1 incomplete transactions are being backed out. 
BACKUP   : (11231) Logical Undo Phase begin at Block 995 Offset 7515. 
BACKUP   : (12095) Logical Undo Phase completed at block 995, offset 7515. 
BACKUP   : (7129)  Usr 0 set name to root. 
BACKUP   : (6686)  156532 active blocks out of 479584 blocks in <db directory> will be dumped. 
BACKUP   : (6688)  0 BI blocks will be dumped. 
BACKUP   : (9285)  Backup requires an estimated 612.7 MBytes of media. 
BACKUP   : (9286)  Restore would require an estimated 156532 db blocks using 611.5 MBytes of media. 
BACKUP   : (12850) Backup blocks will be written to <backup file>. 
BACKUP   : (1362)  Full backup started. 
BACKUP   : (5461)  Begin backup of Data file(s). 
BACKUP   : (5462)  End backup of Data file(s). 
BACKUP   : (13625) Wrote a total of 4222 backup blocks using 560.7 MBytes of media. 
BACKUP   : (1364)  Full backup successfully completed. 
BACKUP   : (15109) At Database close the number of live transactions is 0. 
BACKUP   : (15743) Before Image Log Completion at Block 996 Offset 563. 
BACKUP   : (334)   Probkup session end. 

RFUTIL   : (451)   Roll forward session begin for root on batch. 
RFUTIL   : (7129)  Usr 0 set name to root. 
RFUTIL   : (15321) Before Image Log Initialization at block 996  offset 563. 
RFUTIL   : (660)   Beginning roll forward of after-image file <AI file>. 
RFUTIL   : (831)   ** The database was last changed Fri Nov 21 13:15:18 2014. 
RFUTIL   : (832)   ** The after-image file expected Fri Nov 21 12:36:03 2014. 
RFUTIL   : (833)   ** Those dates don't match, so you have the wrong copy of one of them. 
RFUTIL   : (11014) roll forward open <AI file> error: -1. 
RFUTIL   : (15109) At Database close the number of live transactions is 0. 
RFUTIL   : (15743) Before Image Log Completion at Block 996 Offset 563. 
RFUTIL   : (334)   rfutil -C roll forward session end. 

This is expected behavior: crash recovery was carried out as part of the PROBKUP process

• The previous roll forward contained an in-flight transaction. 
• PROBKUP then performed crash recovery and backed out the in-flight/incomplete transaction
• The next roll forward fails because the AI dates no longer match.

Similarly, truncation of a bi file on a warm standby database can and often will result in ai files not being able to be applied to a database due to transactions that may have been processed when going through crash recovery.  Refer to Article:  

•  Why you should NOT truncate the bi of the target database when using an ai disaster recovery plan ?  

Recover the Hotspare Baseline:

1. Restore the last backup of the production database.
2. Roll forward the relevant ai extents.

In the future if the hotspare / standby database needs to be backed up and still available for future roll forward:

1. Use probkup -norecover. This will skip crash recovery and avoid this issue when in-flight transactions are at play. 

•  Should probkup -norecover be used for an AI-enabled database?  

2. Use OS utilities to copy all the files belonging to the database structure
 

Progress Articles:

 How to use After-imaging (AI) to maintain a hotspare database
 Should probkup -norecover be used for an AI-enabled database?  
 Why you should NOT truncate the bi of the target database when using an ai disaster recovery plan ?