How to use After-imaging (AI) to maintain a hotspare database

How to use After-imaging (AI) to maintain a hotspare database?
How to create a hot backup with AI files
How to use After-imaging to create and maintain a hotspare database.
How to use after-imaging for standby database.
After-image disaster recovery plan.
How to maintain a hotspare with roll forward?

Overview

The following is applicable where two machines are available, one for live production, and one for backup in case the live machine goes down due to hardware failure.
The idea is that a copy of the live database is initially restored on the backup machine (hotspare) and, through time, the copy is kept in sync with the live machine via after-imaging.


Setting up the live machine

Since OpenEdge 10.1B, both AI and the AI Archiver can be enabled online at the same time or singularly as needed with the PROBKUP utility, in which case the database backup pre-requisite to enabling after-imaging is carried out in parallel. Please refer to Article How to enable or disable after imaging (AI) online  

Prior OpenEdge 10.1B, enabling after-imaging (AI) on a database can be done only while the database is offline. This Article describes enabling offline applicable to any version

1) Database server shut down.

2) Disable after-imaging if previously enabled.
4) Assuming prior to this point the database had no after-image extents, at this stage they are added (but left inactive). Alternatively existing ai extents can be restructured by removing (prostrct remove sports a) and re-adding them. Please see Progress Article How To Add AI Extents in an Existing Progress Database for further details.   5) Verify the sports.st file with the new AI extents included.
6) Backup the database

7) Fully verify backup before creating the hotspare database.
$   prorest sports sports.bk -vf

8) Transfer the backup (sports.bk) onto the backup machine and create the hotspare database. This is detailed in section "Setting up the hotspare machine" below.

9) After-imaging enabled on live system.
$   rfutil sports -C aimage begin

At this point there will be a backed up database on the live machine and a new reference point for rolling forward the after-image extents, on the hotspare machine.

10) Start the Database

11) Via operating system utilities (for example: cron on UNIX, or Scheduled Tasks on Windows), a script is fired every N minutes in order to forward any full after-image extents to the hotspare machine.
To determine which after-image extent is full. To mark the full after-image extent as empty and ready for re-use, after it has been backed up with os utilities How often this script should fire depends on the size of the after-image extents and the load of your system: basically you must avoid that all after-image extents become full at any time. Please refer to Article Basic guide to After-Imaging for further details.

Since OpenEdge 10.1A a new feature on Enterprise Databases called the "After-image Archive Manager" can be enabled on the database and manage the archiving and emptying of Full AI extents automatically. The Database Administration Guide and Reference has further detail on this feature and the following article describes the method: Article How to enable automated AI File Management? 

12) [Optional] Online backup of "live" database: This will generate an after-image extent switch and the full AI extent can be rolled forward on the current hotspare database, or the next-full ai extent can be rolled forward against the prorest of this backup. This is also the method that can be employed to take a new hotspare baseline should the need arise in future. The ai script management set up in the previous step should take care of this case.

Setting up the hotspare machine:

1) Restore the database backup create in step 6 above. Ensure that a dbname.st file exists in the directory the dbname.db file will be restored to, in order that the correct pathnames are modified (if needed). Consider adding 'extra' extents per Storage Area for the hotspare database, so that when more space needs to be added to the production database, these are already in place for the hotspare thereby negating the need for a new hotspare baseline. In particular, the hotspare should have more bi file space than the production database as by nature, the bi file on the hotspare will be larger than it's equivalent on the production database.  For history purposes you may consider including the log file for the production database. 2) Via operating system utilities, a script is fired every N minutes in order to check whether any after-image extent has been forwarded from the live machine. If so, roll forward to the hotspare database.
3) No process except for the roll forward utility can access this database.

4) [Optional] Off-line backup of the "hotspare" database without truncating the bi.
The -norecover parameter is an absolute requirement, without it, probkup would perform bi recovery before backing up the database, thus invalidating the roll forward sequence.

Other considerations:

a) Whatever disaster recovery plan that you decide to implement needs to be fully verified. Experience has proven that not testing or not documenting the after-image strategy put in place has detrimental (and costly) consequences.

b) In a production environment, an important consideration is when the structure of the live database will eventually need altering, that is, adding/removing after-image extents to the live database without breaking the after-imaging process. While adding a new after-image extent poses no problems, removing after-image extents from the live database is unfortunately not possible: in order to remove an after-image extent, after-imaging must first be disabled, therefore interrupting the chain of after-image files to be fed to the hotspare database.

Similarly, data extents can only be added to but not removed from the two databases and before-image extents cannot be added to the hotspare database without first truncating the bi file (which is not supported as it can and does cause roll-forward to fail afterwards, in which case a new baseline of the hotspare will need to be established).

c) Progress does not support cross platform compatibility, so the same operating system should be used on both the live and hotspare machines. Please ensure that both machines are running exactly the same level of Progress (that includes Service Pack levels and bit-version of the OpenEdge product).

d) The after image extents for the live database should not be placed on a remote drive. This is not supported for safety reasons, since writing across a network is likely to create unexpected behavior and can impact performance severely. The after image extents should be placed on the same machine as the live database, but on a different disk.  The Full AI files that are archived should however be placed on seperate media to the database. Typically AI files are housed with the periodic backup volumes. How many AI files to keep all depends on the DR solution designed. It is typically deemed prudent to keep at least the past two weeks of backup media, having the most recent media local and dated media remote.

e) It is not advised to run after-imaging with only one AI extent. Furthermore, extents should be defined as FIXED length and in this way not only prevent hitting the 2GB limit (in versions prior to Progress 9.1C), but also control when an after-image extent switch occurs, how much disk space each extent uses and ensuring this disk space is reserved.

Whether fixed or variable AI extents are employed, if a "disk space shortage" or a "no empty AI extent available" scenario arises, emergency maintenance will need to be performed, as Progress will force a shutdown. To prevent this from happening, the -aistall startup parameter should be considered if the database is in multi-user mode. Once -aistall is used, the database will then instead, be suspended, a message will be sent to the log file, the extent can be archived and marked as empty and the system will automatically switch to this extent as database activity resumes.

To estimate the amount of after image space needed, view the "BI Log Activity" using the PROMON utility. Measure the BI Bytes statistic toward the end of the period that you are considering using after-imaging. Typically 0.05% fewer bytes are written to the AI file when Replication is not enabled. Always allow extra disk space for unanticipated growth.

f) In case the live machine goes down and you need to switch to the hotspare you have to consider that the hotspare will not have after-imaging enabled. In the event the hotspare machine goes down as well, you risk losing all the work since the last backup. Therefore a disaster recovery plan must be set up for the now-live database. Once the machine that went down is available again, you must set up both machines again, as described at the beginning of this Article.

g) There are licensing issues to be considered: one license is needed for each machine where Progress is running. These are best discussed with your Account Manager.

h) If incremental backup was taken on live it needs to be applied on the hotspare database.
 

References to Other Documentation:

Progress Article(s):

Other methods of Hotspare database creation are outlined in: