PROSTRCT addonline hangs if all AI extents are LOCKED and BUSY extent is full

PROSTRCT addonline hangs if all AI extents are LOCKED and the BUSY extent is full error 12867.
Database is stalled due to using -aistall database startup parameter.
_dbutil running the addonline needs to be killed from the process list.

prostrct addonline option was added in OpenEdge 10.1A
Database uses the -aistall startup parameter.
 

This problem is not caused by Replication, but Replication makes it more difficult to empty AI extents. When transaction notes in the extent have not been replicated to the target, the extents are in LOCKED status as opposed to FULL and therefore cannot be EMPTY'd.

The addonline process ends up blocking, waiting for the resources and latches that are in use currently held by the process blocked by the AI stall.

• When the database is in an AI stall (-aistall in use), the addonline process is attempting to write a note to the current AI extent.  This process remains blocked until the AI stall has been resolved by emptying an extent. 
• If the AI stall is never resolved, the addonline process will block waiting for the stall to be resolved, effectively hanging the addonline process. 
• The addonline command depends on the same resources as the process that is blocked behind the AI stall and 2 processes cannot use the same resources at the same time.

Upgrade to OpenEdge 10.1B03, 10.1C and above, where PROSTRCT addonline is prevented from waiting by first checking for the aistalled state and exiting gracefully if it is not able to complete the requested utility.  
 

Workaround 1:

If any AI areas are FULL, as opposed to LOCKED:

• Archive and empty the AI extents;then the PROSTRCT addonline command should no longer be blocked if still running.
• Otherwise rerun the PROSTRCT addonline command if it is no longer running after existing AI extent space has been made available.

Workaround 2:

For this Workaround, please note the Article references below for important related issues in this area: 000011649, 000011626 

Of particular importance prior to OpenEdge 10.1B03:

• Do not shut the source database down before disabling site replication online using DSRUTIL under these conditions.
• AI extents cannot be reordered. Only if the last ai extent is the BUSY extent can AI extents be added pre 10.1B03.

Workaround 2a:

Prior to OpenEdge 10.1B03, when OpenEdge Replication is enabled and all AI extents are LOCKED, additional AI areas cannot be added unless the last AI extent is the BUSY extent and the database is not in an -aistall condition.  In which case OpenEdge Replication will need to be disabled online, at which time the LOCKED extents will become FULL extents that can then be archived and emptied:
 
After disabling replication, re-create the replication target database and re-enable replication when next convenient.
Do not shut the source database down before disabling site replication online using DSRUTIL.

Workaround 2b:

1. Reset the ai archiver deamon time switch to on demand if timing is in place:
or stop the cronjob that manages ai switching if AIMGT is not in use.
Once the ai file issues have been resolved these utilities can be restarted.

2a. Get Replication to unlock the AI Extents.

The ai extents on the source are LOCKED in the replication environment most likely because replication has stopped. 

To verify this, run the DSRUTIL source -C monitor utility, when not running if it returns:
Restart replication:

a. Restart the target database: b. Restart the replication server on the source database:
Monitor the status of these with the "DSRUTIL source -C monitor" utility.

When the source and target databases are synchronised the LOCKED extents will become FULL. 
These FULL extents will become available for ai notes once they are backed up (for roll forward plans) then emptied with the 'RFUTIL source -C aimage empty' utility or by the AI Management Daemon.

2b. If the aistall cannot be lifted, add new ai extents offline. OpenEdge 10.1B03 or later will have to be installed to re-order the ai files (PSC00185435)
Where newaifiles.st contains the structure definitions of the new ai files, eg:

a . v 200000
a . v 200000
a . v 200000

$ prostrct reorder ai source (fixed in 10.1B03, PSC00185435)
$ prostrct list source # to update the .st file
$ rfutil source -C aimage list # to check that there are now EMPTY extents after the current BUSY extent.

Finally re-start the source database.

NOTE: Sufficient ai file space needs to be added to accommodate the time to restore the replication baseline or to resolve the source <> target replication processing. Otherwise, consider disabling replication until such time as the replication environment can be restored.

Progress Articles:

 Cannot disable replication if database is shutdown, AI extents are all LOCKED, and busy extent is full   
 Prostrct reorder ai fails if AI extents are all FULL/LOCKED and busy extent is full