Database crash on IDXCOMPACT 1124 crash recovery fails

Database crashes during IDXCOMPACT 4232 10560 10561 1124
When IDXCOMPACT is run with -MemCheck additional messages 14044 14039
Running with -MemCheck does not catch it earlier to prevent crash recovery from failing
IDXCOMPACT crashes with Block Corruption 1124,

Database must be restored as BI recovery fails during Physical Undo 645

Re-running IDXCOMPACT after restoring the database completes without error
An index by compact @ x% then crashes with compact @ x+1`%

stack trace from _dbutil idxcompact:

IDXCHECK / IDXFIX are clean prior to idxcompact crashing out
The table.index that was being compacted has no record or field corruption
There is no Block Corruption reported prior to running idxcompact

There were no schema changes associated with the table idxcompact fails on

IDXCOMPACT is granted with the ulimit of owner of the process where fsize is unlimited (ulimit -f)
There are SQL reporting and ABL compile running at the time
IDXCOMPACT does not interfere with other connect processes as it locks one to three index blocks at a time and there are no record or table locking during the execution
 

This is a corner case which had the potential to fail in earlier versions since Large Keys (LKE) were introduced in 10.1B when the index has more than 5 level index trees. The failed crash recovery is a consequence of not being able to undo these changes.

• Databases that use 4KB blocksizes are more susceptible, as the index block (4004 bytes) must always have enough space for any two entries therefore the maximum number of index entries limits on block size.
• An 8K block will always fit 2 LKE entries of max size 2 x (1970 + header = 1983 bytes).

1. When compression replaces a non-LKE entry with a LKE entry, the remaining index space is occasionally off 3 bytes
2. Eventually this results in an overflow in the compression calculation which should not happen and is therefore not caught
3. When index compact tries to release the buffer and buffer manager does not have the buffer dbkey it expects
4. BI recovery therefore fails in the Physical Redo on a different block (not the block reported in idxcompact failing and is not a block found for said index in an idxblockreport )
5. Additionally, Memcheck doesn't catch some compression errors early enough allowing compression to continue altering the tree structure

Fixed version(s): OpenEdge 12.4, OpenEdge 12.2.4, OpenEdge 11.7.9

Option 1: Restore Backup

Option 2: Rebuild the index that crashed IDXCOMPACT

Force in (-F) to open the database, which can be used if only idxcompact was running and there were no other open transactions, otherwise further physical and logical repair is required.
Use IDXBUILD to completely rebuild or IDXMOVE the index to a new area.  This reduce the number of level index trees.

Additionally: Pre-determine specific indexes that are vulnerable
Run PROUTIL idxanalys or an idxblockreport on specific indexes for finer detail to find any index with 5 levels.  Schedule an IDXBUILD / IDXMOVE instead of IDXCOMPACT.

Table                      Index  Fields Levels         Blocks    Size  % Util  Factor
PUB.tablename
  indexname                   13       4      5        1135070    4.3G    52.8     1.9

Progress Article:

 What is the -MemCheck Parameter?