Best Practice for Adding an Index in Production

It is sometimes necessary to remove or add an index to your production cluster. For a large database with more than a few GB of content, the resulting workload from reindexing your database can be a time and resource intensive process, that can affect query performance while the server is reindexing. This article points out some strategies for avoiding some of the pain-points associated with changing your database configuration on a production cluster.

Oct 15, 2024
Knowledge

Title

URL Name

best-practice-for-adding-an-index-in-production

Article Number

000267186

Environment

Product: MarkLogic Server Version: All Supported Versions OS: All Supported Platforms Component: Configuration, Data Management, Memory Management, Performance, Tuning

Question/Problem Description

Article Type: Best Practice

Steps to Reproduce

Clarifying Information

Error Message

Defect Number

Enhancement Number

Cause

Resolution

Preparing your Server for Production

In general, high performance production search implementations run with tight controls on the automatic features of MarkLogic Server.

Re-indexer disabled by default
Format-compatibility set to the latest format
Index-detection set to none.
On a very large cluster (several dozen or more hosts), consider running with expunge-locks set to none
On large clusters with insufficient resources, consider bumping up the default group settings
- xdqp-timeout: from 10 to 30
- host-timeout: from 30 to 90

The xdqp and host timeouts will prevent the server from disconnecting prematurely when a data-node is busy, possibly triggering a false failover event. However, these changes will affect the legitimate time to failover in an HA configuration.

Preparing to Re-index

When an index configuration must be changed in production, you should:

First, index-detection should be set back to automatic

Then, the index configuration change should be made

When you have Database Replication Configured:

If you have to add or modify indexes on a database which has database replication configured, make sure the same changes are made on the Replica cluster as well. Starting with ML server version 9.0-7, index data is also replicated from the Master to the Replica, but it does not automatically check if both sides have the same index settings. Reindexing is disabled by default on a replica cluster. However, when database replication configuration is removed (such as after a disaster), the replica database will reindex as necessary. So it is important that the Replica database index configuration matches the Master’s to avoid unnecessary reindexing.

Note: If you are on a version prior to 9.0-7 - When adding/updating index settings, it is recommended that you update the settings on the Replica database before updating those on the Master database; this is because changes to the index settings on the Replica database only affect newly replicated documents and will not trigger reindexing on existing documents.

After the Re-index

After the re-index has completed, it is important to return to the old settings by disabling the reindexer and setting index-detection back to none.

If you're reindexing over several nights or weekends, be sure to allow some time for the merging to complete. So for example, if your regular busy time starts at 5AM, you may want to disable the reindexer at around midnight to make sure all your merging is completed before business hours.

By following the above recommendations, you should be able to complete a large re-index without any disruption to your production environment.

Workaround

Notes

Keyword Phrase

Last Modified Date

10/15/2024 4:51 PM