DBDOWN with timestamp errors on AI files 9215 9213

Database inacessible with 'Last open date mismatch. (9215)' errors for AI-files
Database is down because of error 9215 on an AI file
Database will not start with errors 9215,9213,1486
Last open date mismatch error on ai file  

Examples:
Last open date mismatch. (9215)
Extent /database/folder/dbname.a1 has a different last opened date (9213)
Control Area has a last open date (9217)
Probable backup/restore error. (605)
Database is damaged, see documentation. (1486)

Last open date mismatch. (9215) 
Extent /database/folder/dbname.a1 has a different last opened date <timestamp>, (9213) 
Database is damaged, see documentation. (1486)
 

Error 9215 is produced either when connecting to a database in a single-user mode, or when trying to start the database multi-user

The database is inacessible because the .a<n> file referenced in the 9213 message is out of sync with information stored in the database Control Area.

There are various reasons why this may occur, some of these are:

1. Database files are OS copied across the network without date-preservation flags, the timestamp difference is typically caused by large variable ai file sizes
2. Database files are OS copied to the wrong directory and overwrite the ai file of an_other database with the same name
3. File system that AI is mounted on, did not mount correctly when the .DB file (Control Area) was accessed.
4. Database files copied to a new disk location went 'out-of-sync' with its AI-files because they were accessed while the copy was taking place. Apart from non-Progress utilities,  files on the old disk were possibly accessed by a background RFUTIL process for handling AI-files after the database was shutdown which updated the remaining database file timestamps as they were being copied
5. After image extents were moved instead of copied after filling up, then moved back again thereby changing the timestamps
6. AI extents with older timestamps are found to be in the same directory as the current ai extents should be, possibly after splitting a mirror. 
7. Not having the new location of the OS copied AI extents in the structure file (.st) when subsequently running the PROSTRCT utilities to update the Control Area
8. A Hotstandby database, previously used as the source in the past then reverted to hotstandby status detailed in Article:

•  PROBKUP - Creation date mismatch on AI files  

The term "OS Copy" in this Article refers to OS disk mirroring, VSS, VM snapshot Volume snapshot or any other Third Party Tool utilities are used to provide data redundancy as part of a backup and recovery strategy.

If the original database is still available at the previous location:

1. Ensure the database is still accessible, without ai file corruption.
2. Stop all running processes against this database including cron jobs.
3. Truncate the bi file and take the OS copy again.
4. To use the OS copy database files in a new location, update the Control Area by running PROSTRCT REPAIR. For detailed instruction refer to Article:

•  How to use PROSTRCT repair?  
•  How to create an offline OS database copy without AI enabled?  

To use this database copy to apply transaction notes recorded after the OS copy was taken, refer to Article:

•  How to use an OS backup with after imaging enabled - post Progress 9.1E?  

If the original database is still running and an Enterprise Database License is installed:

1. Quiesce the database with : proquiet dbname enable 
2. Take the OS copy and assure all files are copied including the ai files. When PROSTRCT REPAIR is run against this OS copy, assure a complete structure file, including ai extents with full paths is available (<dbname>.st)
3. Disable database quiescense: proquiet dbname disable
For further information refer to Article  What are Database Quiet Points?  

If the original database is no longer available or the original database has ai file corruption:

The key factor on this repair is that only the AI Extents have timestamp mismatch errors.

1. Stop after-imaging:
2. Remove current AI files. 

It is imperative that corrupt AI files are removed. When enabling ai later, (Step 4) it is only the header which is reformatted not the content on existing ai files which may work but will more likely than not result in the same problem when the database is next accessed.

The following command needs to be run once for each AI file to remove all AI extents
3. Add new AI files

Create a structure file 'add.st' file (for example) and then edit this text file, so that it only contains AI extents' definitions needed.

$   prostrct add dbname add.st

4. Enable After-Imaging and the AIMGT deamon if required

$   rfutil dbname -C mark backedup
$   rfutil dbname -C aimage begin -G 0
$   rfutil dbname -C aiarchiver enable

An alternate and recommended method to OS copies:

When this type of situation is caused by someone having run an OS copy then not having the new location of the copied extents in place when subsequently running the PROSTRCT utilities. You end up with two control areas accessing different database extent locations.

Restore the database with 'prorest' command (from Progress backup taken with 'probkup') to create the database on the new server / disk location.
Once restored, AI files can be added if not included in the restore structure and AI re-enabled as required on the restored database.


 

Progress Article:

 What to do when an After Image file is corrupted.  
 What is the difference between PROSTRCT REPAIR and BUILDDB?  
 What do I do if I deleted some database extent files from OS that have data in them?