Any AI file that has not been completely applied to the replication target database is marked as a LOCKED AI file. These
Locked AI files can be manually applied to the target database should the need arise, with the DSRUTIL utility.
Use-Case Reasons:Probable production situations that should consider manually apply AI extents to the target database are:
- To first apply remaining LOCKED source ai extents that have not yet been applied to the replication enabled target database, provided these are still available, before transitioning the target database to a source production or normal database.
- Intensive batch processing that introduces high network latency between the RPLS and RPLA, eventually causing the replication service queue (RPLS-Q) to fill resulting in source database performance problems.
- The RPLS and RPLA loose connection which goes unnoticed causing the target database to be significantly behind the source database and many LOCKED AI extents accumulate on the source database as a result, eventually causing the source database to stall when -aistall is used or to shutdown otherwise.
Conditions for DSRUTIL applyextent:Conditions the target database must meet in order to manually apply AI extents to the target database:
- The Agent must not be configured for auto transition.
- The Agent must be in Pre-Transition mode before AI extents can be manually applied.
METHOD 1:
- Where the starting point is that the Source and Target databases were previously synchronised, and
- The Replication Server (RPLS) and Replication Agent (RPLA) are still connected processing AI transaction notes
- Until such time as either:
- It is noticed that the target has fallen too far behind the source (replication latency)
- Replication broke, leaving the Replication Agent running in PRE-TRANSITION mode waiting for re-connection from the Replication Server
- Scheduled batch jobs need to run on the source database, known to cause latency problems due to the high transaction activity.
In the instructions that follow, the following short-hand notation depicts:
[S]: source machine
[T]: target machine
[S]: 1. Stop the Replication Server (if still running)
$ dsrutil source -C terminate server
Once the RPLS is confirmed as stopped, release waits
$ dsrutil source -C RELWAITS
[S]: 2a. If After-Image Management is enabled, re-set the AIMGT archive interval to '
on demand', to minimise ai extent switching to controlled events (when fixed AI extent fills, or a manual AI switch with
RFUTIL -C aimage new) until replication resumes.
$ rfutil source -C aiarchiver setinterval 0
[S]: 2b. Manually force a new AI file switch and note which AI File is the current "BUSY" extent. This provides a baseline for exactly how many AI files need to be manually applied to target up to this point in time, provided there is an available EMPTY ai extent to switch to.
$ rfutil source -C aimage new
$ rfutil source -C aimage list
If necessary, more ai extents can be added to the database structure:
$ prostrct addonline source addai.st
[S]: 3a. If (further) batch processing needs to run, do so.
Decide how to control the AI switch so that each LOCKED AI extent can be transferred to the target machine to be manually applied against the target database, while this process is running in order to remove the network transfer overhead and save synchronisation time later.
For example,
manually instruct an AI switch every 30 minutes:
$ rfutil source -C aimage new
Otherwise re-set the timed AIMGT interval:
$ rfutil source -C aiarchiver setinterval 1800
Consider that FIXED ai extents may fill and switch beforehand, in which case there will be more than one ai file to manually apply to the target database within this timeframe.
[S]: 4. Copy LOCKED AI files to the TARGET machine:
AI files should be compressed (zip/tar) before transferring to the target machine, to minimise network bandwidth and ensure the file(s) arrive without transfer issues.
[S]: 4a. If
After-Image Management is enabled, each LOCKED AI extent will be ARCHIVED to the
-aiarcdir. These ai files will already be extracted and only need transferring to the target machine.
[S]: 4b. If After-Image Management is not enabled, or specifically new LOCKED extents are being monitored during this activity, first extract then COMPRESS/TAR all LOCKED AI files.
$ rfutil source -C aimage extract -a source.a9 -o source.09
It is important to select either the Archived AI extents or the current Locked extents, not a mixture of both to apply to the target database. If these methods are properly managed, it should not be necessary to manually apply the last BUSY extent. Prior to OpenEdge 11.7, there is a product limitation in this regard to be aware of. Before proceeding refer to Article
Replication Agent crashes applying the last ai extent
[T]: 1. Confirm that target is in PRE-TRANSITION mode:
$ dsrutil target -C monitor -> A
Since the RPLS stopped or was stopped previously, the RPLA should already be in this state. Otherwise, force PRE-TRANSITION mode:
$ dsrutil target -C triggerTransition
[T]: 2. Confirm the expected AI file the target is expecting. Refer to Article
How to determine the correct ai extent number to apply to target first?
$ dsrutil target -C recovery agent
[T]: 3. UNCOMPRESS/UNTAR the copied AI files and apply these manually to the target database in sequential order.
$ dsrutil target -C applyextent <fullpath>\source.a<n>
[S]: 5. Restart the RPLS once the batch processing has ended AND/OR last LOCKED AI file has been successfully applied to target
$ dsrutil source -C restart server
Once the RPLS and RPLA connect and synchronize, all LOCKED AI FILES that were manually applied to the target database are rapidly UNLOCKED and re-set to EMPTY available for use as a result.
METHOD 2:If for some reason the
"starting point" in
Method 1 is not true, then extents can still be applied to the target database by paying attention to the configured
connect-timeout value in the repl.properties file and ensuring that
manual transition is set.
Example: The RPLS terminated or there was some RPLS/RPLA communication failure that did not recover. The RPLA will either no longer be running or without having synchronised, will be waiting in PRE-TRANSITION mode depending on the
[Server] agent-shutdown-action configured in the
source.repl.properties file
1. Ensure RPLS is stopped on source (as above)
2. Edit the
target.repl.properties file.
Change the
connect-timeout in the
target.repl.properties to (say) 3 hours or as long as the APPLYEXTENT is going to take.
3. Restart the TARGET database then immediately force to Pre-Transition, in other words
no synchronisation will have completed.
Do not use the force option when stopping the target.
$ proshut target -by
$ proserve target [+ startup parameters ] -DBService replagent
$ dsrutil target -C triggerTransition
4. Apply LOCKED extents as outlined in Method 1 above, however:
- The "APPLYEXTENT" against target must complete within the connect-timeout in the target.repl.properties.
- When connect-timeout expires before applyextent, then the RPLA will shutdown and prevent further AI files to be applied:
RPLA 5: (11700) This Replication Agent has never been contacted by a Replication Server. The Agent is ending so the Target database can be shutdown normally.
RPLA 5: (10506) The Fathom Replication Agent agent1 is ending.
Once the RPLA has stopped, the next applyextent will fail:
RPLU (10829) Database connections are not allowed at this time.
This is not necessarily a problem per se as the target database can be restarted, then force pre-transition and applyextent again within the connect-timeout in the target.repl.properties (as above).
However, we know that in this situation the target could consequently take a long time to complete the REDO phase. Essentially this situation can and should be prevented by temporarily increasing the target agent
connect-timeout value longer than the anticipated "APPLYEXTENT" time.
5. Once all the manual
applyextents have completed:
- Shutdown target,
- Reset the connect-timeout and restart target database
- Restart the Replication Server
- Once the RPLS and RPLA they connect, they synchronize and all the LOCKED ai files are released.
The
connect-timeout described in Method 2 does not apply in the first scenario described in Method 1.
- This timeout occurs only when an Agent connection never connects with a Server.
- Once the Agent has connected, the connect-timeout is cancelled. It is important to be aware of the difference. Enhancement OE00231338 / PSC00253380 has been raised to cancel the connect-timeout when the RPLA is manually set to PRE-TRANSITION, prior to synchronisation completing. It has not been implemented to date (10.2B08, 11.7)
