Index corruption causes IDXCHECK to loop on the 8781 error

When the IDXCHECK reports a 8781 out of sequence error, the same error is repeated until the error limit is reached and IDXCHECK exits.
Index corruption causes IDXCHECK to loop on reporting an 8781 error.
Index block corruption can cause IDXCHECK to stop reporting on all indexes after the 8781 error
Error 8781 is only reported when "Validate key order" is enabled (by default) in the IDXCHECK Validation Options.

Index Block point sequence corruption breaks the idxcheck utility by causing it to either loop indefinitely if the recid sequence corruption is pointing backwards to a previously scanned recid, or will cause it to skip past recids completely if pointed forward towards the end of the index. This error prevents the utility from ever being able to complete the scan all the way through the index.

When performing key order validation, IDXCHECK traverses the b-tree, comparing the key value of each key with previous one.

• Raw values of index keys are expected to be in ascending order on the index B-Tree.
• If the current key is not larger the 8781 error is reported.

Since the corrupt key that has an "out of sequence" key order instead of moving on to the next key,
it causes IDXCHECK to move back a position in the btree that hits the corrupt key again and again. This kind of corruption currently causes endless duplicate results eventually exhausting the IDXCHECK error limit (default 500)

The root cause is index corruption which must be addressed.

This enhancement has not been implemented in the product, however has been confirmed to no-longer cause IDXCHECK to loop when the sequence of index keys was out of order on the 8781 error in recent OpenEdge 11.7 incidents, where 32/64 bit boundary corruption has previously been addressed with IDXBUILD.  

There may still be cases where this situation is evident, where an ehnancement has been raised to guage future occurrences: Enhancement Request: IDXCHECK should continue scanning index after recid out of sequence error.

The presumption was, while there are an infinite number of ways that indexes can be corrupted which will cause the utility to loop, in certain cases where it is known that errors are occurring and will need to be fixed anyway, it is worthwhile to skip over them and start processing the next index so that one can find out which indexes need to be rebuilt without exhausting the IDXCHECK error limit on the same index. As a consequence the 15913 message will report more index entries than records found. Part of these will be due to the key order validation traversing the b-tree:

• comparing the key value of each key with previous one
• finding the current raw value of the index key is not larger
• and skipping the index after reporting the "out of sequence" key order 8783 error

 

$ proutil -C IDXCHECK -lkwtmo 30 -Bp 32

• Select: O (Validation), then L (Lock tables during the check) & "C"
• Select: T (By Table), then toggle the number next to the required table(s) & "G"
• Toggle the number next to the required index & "G" & "Y"

Run IDXFIX Option 2  (scan indexes) offline, assuring "Validate recids for index entries" & "Fix indexes on Scan" are confirmed (Y).

• When IDXFIX is run online, it cannot and does not detect the errors found by Option 4 of IDXCHECK. The key order can only be validated when IDXFIX is run offline.  Once the errant index has been rebuilt, the sequence order will be restored.  
• While IDXFIX offline will fix the out-of-sequence, IDXBUILD is faster and better in that it rebuilds from scratch, except after-imaging needs to be disabled. Strictly, you would only need to rebuild this index. 

When IDXCHECK repeats on the 8781 out of sequence error,
re-run the report by not selecting "Validate key order" in the IDXCHECK Validation Options.

An index that has it's key order out of sequence, is in need of attention anyway and is almost certainly going to raise an error in the other IDXCHECK Validation Options:
Then address the index corruption as outlined in the Resolution above.

Progress Articlea:

 What are the implications of the online IDXCHECK locking options?   
 Client processes (and idxfix) hang, loop on (8828)  
 IDXCHECK: Key is out of sequence error 8781 after online IDXFIX