Application runtime errors 16488 5071 995 after adding new active indexes online to an existing table

After adding new indexes online using the Add Objects Online  Option queries that use the new index fail with error 16488
Application code re-compiled against the new schema that references the active index fails with error 16488

The new index is ACTIVE when verified through the _index._active field or Data Dictionary

IDXBUILD Menu Options by inactive activation or by Table or Area with inactive indexes does not find the newly added indexes as inactive indexes to be built: error 2783
IDXBUILD command line inactiveindexes does not find the newly added indexes as inactive indexes to be built: 16274

ABL query (_index._active) reports the new index as Active
Data Dictionary shows the new index as Active.
Online IDXFIX Option 3 shows the new index as Inactive, error 5071

IDXBUILD by inactive activation or by Table or by inactive indexes does not find the newly added indexes as inactive indexes to build:

• Indexes are created as inactive in the _StorageObject._storageObject.objectstate when an active index is added to an existing table online. 
• With multi-tenancy and table partitioning the _Index._Active field does not represent the state of the index partitions for all tenants, this is no longer managed at the _index level. Therefore, an architectural change to stop using the _index._Active flag and use the attributes in the _StorageObject record instead. This is what allows indexes to be loaded online when not marked as INACTIVE in the .df. Prior to MT/TP, an index could not be added online as an "ACTIVE" index to an existing table.
• The _index._active flag on the _Index record is set to "yes" is by design, in order to to deal with schema cache on the client side and the online index activate utility (IDXACTIVATE, MTIDXBUILD). The Data Dictionary tool gets information about active indexes from the _index._active state, this does not mean the index is built.
• The only guarantee with _Index's _Active field, is that if the field is 'no', all tenants (default tenant when not enabled) or partitions (default partition when not enabled) of the index are inactive. 
• When the _Index's _Active field is 'yes", to confirm if the index is built: the _StorageObject._Object-State record needs to be verified ( _Index . _Idx-num = _Object-Number = _Idx-num  )
• _StorageObject._Object-State is a bitmask where the bit with value '1' when the (index's) object state is marked inactive. When MT or TP are enabled the _StorageObject._Partition-Id must be used to find the correct index.
• While the _index._active state can be set to false with ABL to deactivate indexes backend, when true it cannot be used to reflect the current state of the index keys. In otherwords it is not a safe mechanism to detect whether an index is activated or not.

Irrespective, any index ( active or marked INACTIVE) added online to an existing table with a .df, or after a binary load, must then be built afterwards with the index utility of choice (idxbuild, idxactivate). 

Upgrade to OpenEdge 11.7.6, 12.3 where the DataDictionary and Index utilities have been improved to correctly report and handle the object's state:

1. The Data Dictionary checks if the index is inactive in the _StorageObject._Object-State to display the value in the UI. 
2. The index rebuild utility will detect inactive indexes by checking both _index._active and _storageObject._Object-state in order to determine inactive indexes to build.

To determine if an index is active with ABL Query on the meta-schema:

1. The only guarantee with _Index's _Active field, is that if the field is 'no', all tenants or partitions of the index are inactive. 
2. When the _Index's _Active field is 'yes,':

• The _StorageObject record for the object needs to be checked. Use the _Index . _Idx-num to find the matching _StorageObject where _Object-Number = _Idx-num and display the _Object-State. 

The  _Object-State  field is a bitmask where the bit with value '1' is set if the object state is marked inactive. 
When using MT or TP, find the _StorageObject for the right _Partition-Id 

Example: An index is inactive:

• If  _index._active == no  
• If  _index._active == yes AND _StorageObject._object-state == 1

DEFINE VARIABLE object-num AS INTEGER.

FIND _file WHERE _file-name = "Customer" AND _Tbl-Type = 'T' NO-LOCK.
FOR EACH _Index NO-LOCK WHERE _Index._File-Recid = RECID(_File):

object-num = _Index._idx-num.

FIND _StorageObject NO-LOCK WHERE _StorageObject._Db-recid = _File._Db-recid 
    AND _StorageObject._Object-type = 2 
    AND _StorageObject._Object-number = object-num NO-ERROR.

DISPLAY
object-num _index-name SKIP
_index._active  _StorageObject._object-state.

 

Methods to reliably build an inactive index prior to this fix as the Data Dictionary and index tools handle the state incorrectly prior to OpenEdge 11.7.6, 12.3 are:

[offline]: IDXBUILD
Use the SOME Option to select the index explicitly to build. (as opposed to the "By Table" and "By Activation" Options which will work with the fixed versions)
Refer to Article: How to run idxbuild some from script   

[online]: Use IDXACTIVATE: