Openedge Management console hangs with orient db exceptions

OEM console hangs with error messages in admserv.log and ads0.exp referring to cachedb.

Management console hangs with orient db exceptions

Every POLL (5 minutes) with errors saving records due to OSBTreeException: where SBTree is in corrupted state

Example:
SEVERE: $ANSI{green {db=cachedb}} Error on using index connectionactivityidx in query 'select from ConnectionActivity WHERE resourceKey = "localhost:resource.database.name" AND time < 1513537458017 LIMIT 500'. Probably you need to rebuild indexes. Now executing query using cluster scan
com.orientechnologies.orient.core.index.sbtree.local.OSBTreeException: We reached max level of depth of SBTree but still found nothing, seems like tree is in corrupted state. You should rebuild index related to given query.
DB name="cachedb"
Component Name="connectionactivityidx"
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.findBucket(OSBTree.java:1759)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.iterateEntriesBetweenAscOrder(OSBTree.java:1480)

Unable to delete expired graph data for localhost:resource.database.remote.name entity ConnectionActivity: We reached max level of depth of SBTree but still found nothing, seems like tree is in corrupted state. You should rebuild index related to given query.
DB name="cachedb"
Component Name="connectionactivityidx"

The indexes listed are specific to an earlier failure when the orient databases were not properly closed when the AdminServer was shutdown.

Earlier problems with the configdb required it to be rebuilt from the crash log:
WARNING: $ANSI{green {db=configdb}} Storage 'configdb' was not closed properly. Will try to recover from write ahead log

SEVERE: $ANSI{green {db=configdb}} Error on using index datasourceidx in query 'select from StorageDataSource where key = ?'. Probably you need to rebuild indexes. Now executing query using cluster scan
java.lang.NullPointerException
at com.orientechnologies.common.directmemory.OByteBufferPool.acquireDirect(OByteBufferPool.java:303)

Caused by: java.lang.NullPointerException
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation$FileChanges.access$300(OAtomicOperation.java:533)

Thereafter the problem is graph cached database corruption which re-surfaces after deleting the cachedb

This orient db corruption did not occur in OpenEdge 11.5

After stopping the adminserver and removing the cachedb directory files, the problem was solved.
The problem re-surfaces after some weeks where downtime is scheduled to re-create the cachedb.

 

orientdb signal handling intercepts SIGTERM and can corrupt the database due to out of order shutdown
 

Upgrade to OpenEdge 11.7.2 Service Pack or later.  The orient databases are upgraded to version 2.2.27 which has these issues which trigger subsequent failures.

1. Stop the AdminServer

2. Dump the configuration database which holds your customised jobs/reports and resource configuration
$   fathom -dump fathom.xml -user admin

3. Apply 11.7.2 Service Pack, which updates the orientdb to 2.2.27.

4. The orient databases need to be deleted so they are created with a fresh 2.2.27 version when the 11.7.2 AdminServer is next started.

Delete the config and cache orient databases, by deleting the *content* of the directories

## orient configuration database: 
echo y | del <oee_install>/config/configdb/*

## orient cache database: 
echo y | del <oee_wrk>/cachedata/cachedb/*


5. place the .xml file that was dumped previously in the configdb directory, rename it if necessary to 'fathom.xml':
<oee_install>/config/fathom.xml

6. Start the 11.7.2 AdminServer
$ proadsv -start

Wait for the fathom Web Server to initialise

$ fathom -query
>> Running

The orient databases will get re-created on restart.

 

Progress Article

 How to reclaim space used by the graph/activity database