Before Image (.bi) file is missing, corrupt, or lost and the database fails to start

Before Image (.bi) file is missing, corrupt, or lost and the database fails to start
Database fails to start with error messages on the BI file
Database BI file is lost, missing or damaged
Cannot start database due to BI file errors

The result of hardware problems and/or operating system bugs that impact database integrity without forewarning. A combination of these error messages indicate that the .bi file is lost, corrupted or is pointing to the wrong bi file at startup.

This scenario can also be a consequence of user error during OS backups, when failing to list all related extents correctly for the database, or when using automated scripts and commands are executing in the background - there is not enough time between commands for the previous command to finish. For example: truncating the bi while the database is still shutting down, resulting in the incorrect OS copy of the bi file when it is restored from a backup. Whatever the initial cause, a combination of these error messages will result and indicate that the cause of failure is that the .bi file is lost, corrupted or is pointing to an inappropriate bi file at startup.

This scenario can also be caused by killing a process with a signal 9 when the process is in single user mode or a the primary broker process which controls shared memory connection.  The hard kill may prevent the process from properly updating the timestamps of the database files or updating the last open / close time of the database.

Progress recommends restoring a database backup and rolling forward AI files if enabled, or failing over to the Replication target database, in this scenario.

Before considering either of the workarounds below, please review the following Article:

• "Consequences of using the -F startup option"

The workarounds outlined below are provided for situations where going to back up is not possible:

• These are highly reliant on an accurate database structure file. If the database structure has been altered or extents moved since last known dbname.st, then going to last fully verified backup should be the only course of action. 
• While skipping crash recovery will allow the database to be accessed again, uncommitted transactions will result in logical application integrity which needs to be addressed backend by application development teams.

WORKAROUND #1:

1.) Ensure that a valid structure file exists where:

• The dbname.db file resides in the same directory as the dbname.st structure file
• The structure file lists fully qualified paths, describing every extent that should be in place on disk including all the bi and ai extents.

Run: prostrct list dbname
View the output of dbname.st in a text editor

2.)  If the bi extents are still present, remove them then replace the bi extent(s) with placebo bi extents.  The prostrct unlock utility uses the information in the dbname.st file, which is a reflection of the information in the database control area (dbname.db) to re-create them.

Run: prostrct unlock dbname -extents
Answer "y" (without quotes) to skip crash recovery

For each bi extent messages similar to the following will result
3.)  Force crash recovery again:
Run: proutil dbname -C truncate bi -F


WORKAROUND #2:

The workaround involves creating a new database from an empty database and then creating the missing bi files.  Once created, these files will be used as a replacement for the missing bi files.

1.)  Take an OS backup of the database before proceeding.  This backup can be used to restore from if problems arise with this workaround.

2.)  Create a new database to mirror the existing database:
     a.  Copy the existing dbname.st file to newdbname.st file.
     b.  Replace the "dbname" in the newdbname.st file with the newdbname.
     c.  Create a void database: prostrct create newdbname newdbname.st -blocksize (same blocksize as original dbname). 
     d.  procopy <DLC>/emptyN newdbname
         N = block size used above. 
         Example: if -blocksize 8192 was used to create the void database:
          $   procopy <DLC>/empty8 newdbname

3.)  Once the new database has been created, copy the "newdbname.b1" file to replace the missing/existing dbname.bi file.  During the copy, rename the file to the name of the original database.  Example:  copy newdbname.b1 dbname.b1
 
If there was more than 1 bi file, copy all of the bi files from newdbname to dbname in order to fully recover the extents.

4.)  Once the new bi extent is in place, run "prostrct unlock" to fix the timestamp differences:

     Command:  prostrct unlock dbname

For each bi extent, a messages similar to the following will result confirming the timestamp inconsistency corrections:

Progress Articles:

• DBDOWN with timestamp errors on AI files 9215 9213  
• What do I do if I deleted some database extent files from OS that have data in them?