Understanding and resolving 'Detecting' and 'Detected' notifications in ErrorLogs.


Article Type: Tips and Tricks 

This message will appear immediately after forests are unmounted and subsequently remounted by MarkLogic Server. Detecting indexes is a relatively lightweight operation and usually has minimal impact on performance.
 

What would cause the forests to be unmounted and remounted

 

• Forest failovers
• Heavy network activity leading to a cluster (XDQP) "Heartbeat" timeout
• Changes made to forest configuration or indexes
• Any incident that may cause a "Hung" message

Apart from the forest state changes (unmount/mount), this message can also appear due to other events requiring index detection.

What are "Hung" messages?
 

Whenever you see a "Hung" message it's very often indicative of a loss of connection to the IO subsystem (especially the case when forests are mounted on network attached storage rather than local disk). Hung messages are explained in a little more detail in this Knowledgebase article:

https://help.marklogic.com/Knowledgebase/Article/View/35/0/hung-messages-in-the-errorlog

What do the "Detected" messages mean and what can I do about them?

Whenever you see a group of "Detecting" messages:
 

2015-01-14 13:06:26.016 Debug: Detecting indexes for database XYZ
 

There was an event where MarkLogic chose to (or was required to) attempt to unmount and remount forests (and the event may also be evident in your ErrorLogs).
 

The detecting index message will occur soon after a remount, indicating that MarkLogic Server is examining forest data to check whether any reindexing work is required for all databases available to the node which have Forests attached:
 

2015-01-14 13:06:26.687 Debug: Detected indexes for database XYZ: ss, wp, fp, fcs, fds, ewp, evp, few, fep
 

The line immediately below indicates that the scan has been completed and the database has been identified as having been configured with a number of indexes. For the line above, these are:
 

ss

stemmed searches

wp

word positions

fp

fast phrase searches

fcs

fast case sensitive searches

fds

fast diacritic sensitive searches

ewp

element word positions

evp

element value positions

few

fast element word searches

fep

fast element phrase searches

From this list, we are able to determine which indexes were detected.  These messages will occur after every remount if you have index detection set to automatic in the database configuration.
 

Every time the forest is remounted, in addition to a recovery process (where the Journals are scanned to ensure that all transactions logged were safely committed to on-disk stands), there are a number of other tests the server will do. These are configured with three options at database level:
 

• format compatibility
• index detection
• expunge locks

By default, these three settings are configured with the "automatic" setting (in MarkLogic 7), so if you have logging set to "Debug" level, you'll know that these options are being worked through on remount:
 

2015-01-14 13:06:26.016 Debug: Detecting indexes for database XYZ (represents the task for "automatic" index detection where the reindexer checks for configuration changes)
2015-01-14 13:06:26.687 Debug: Detecting compatibility for database XYZ (represents the task for "automatic" format compatibility where the on-disk stand format is detected)
 

These default values may change in accross releases of MarkLogic Server. In MarkLogic 8, expunge locks is set to none but the other two are still set to automatic.

Can these values be changed safely and what happens if I change these?

 

Unmounting / remounting times can be made much shorter by configuring these settings away from automatic but there are some caveats involved; if you need to upgrade to a future release of the product, it's likely that the on-disk stand format may change (it's still 5.0 even when MarkLogic 8 is released) and so setting format compatibility to 5.0 should cause the "Detecting compatibility" messages to disappear and speed up remount times.
 

The same is true for disabling index detection but it's important to note that changing index settings on the database will no longer cause the reindexer to perform any checks on remount; in this case you would need to enable this for changes to database index settings to be reindexed.

 

 

Related Reading

How to handle XDQP-TIMEOUT on a busy cluster