Scripting Failover: "flipping" replica forests back to their masters using XQuery

If forest level failover is configured on MarkLogic cluster, in the event that a single host in the cluster loses contact with the other hosts, the forests will fail over to the backup set of forests: the replica forests. This article will talk about how to give the control back to master once master and replica are in sync.

Oct 15, 2024
Knowledge

Title

URL Name

scripting-failover-flipping-replica-forests-back-to-their-masters-using-xquery

Article Number

000268309

Environment

Product: MarkLogic Version: All supported versions OS: All supported platforms Component: Data Management, Failover

Question/Problem Description

When there is forest level failover configured on MarkLogic cluster, in the event that a single host in the cluster loses contact with the other hosts, the forests will fail over to the backup set of forests: the replica forests.

Failover shifts the responsibility for a given set of forests over to other hosts in the cluster; if the failing host "loses" control of its' forests, control is not automatically given back when the master becomes available; failing forests has to happen manually.

Article Type: Solution, Tips and Tricks

Steps to Reproduce

Clarifying Information

Error Message

Defect Number

Enhancement Number

Cause

Resolution

To fail a forest back (to "flip" control back to the master), if both the replica and master forests are in sync with each other, all that's needed is to restart the replica forest. This can be done using the admin API (Configure > Forests > Forest Name > Status > Restart), or XQuery (xdmp:forest-restart):

https://docs.marklogic.com/xdmp:forest-restart

flip-forests.xqy

xquery version "1.0-ml";
 
(: This script looks for forests configured as master on this host that
   are currently acting as a replica.  This implies that a failover has
   occurred and a replica for the forest has taken over as master, and
   also that the master is now back online but is in the role of replica.
  
   Restarting the acting master will cause the configured master to
   resume its role as master.  That will allow the server node that
   owns the master forest to resume handling query load for it.
  
   This module is intended to be installed in the modules database and
   run by a periodic scheduled task about once a day.
:)
 
declare namespace fs = "http://marklogic.com/xdmp/status/forest";
import module namespace admin = "http://marklogic.com/xdmp/admin" at "/MarkLogic/admin.xqy";
 
let $replicas :=
  let $me := xdmp:host()
  for $db in xdmp:databases()
  for $f in xdmp:database-forests($db)
  let $status := xdmp:forest-status($f)
  where $status/fs:state = "sync replicating"
  return
    for $replica in $status/fs:replica-forests/fs:replica-forest
    let $replica-status := xdmp:forest-status($replica)
    where ($replica-status/fs:state = ("open","open replica")) and ($replica-status/fs:host-id = $me)
    return $replica
let $replica-count := fn:count ($replicas) let $msg := fn:concat (
  "Restarting replica forest",
   if ($replica-count gt 1) then "s" else "",
   " to re-target master: xdmp:forest-restart((",
   fn:string-join($replicas/xdmp:forest-name(.), ", "),
   "))")
 
return
if ($replica-count gt 0)
then ( xdmp:log ($msg), xdmp:forest-restart ($replicas) ) else xdmp:log ("No acting-master replica forests found that need restarting", "info")

The above code is intended as a sample for something that could be used as a scheduled task that will automatically check for failed over forests and to flip them back where possible.

It's also worth noting that this may not be something you'd want to do; in many cases, a failover event might be a warning of a problem that occurred that needs to be investigated (for example: a disk error), so if you are planning on managing failing back forests automatically, you may want to ensure that you are monitoring the ErrorLogs for evidence of failover events so you know that they're happening.

Workaround

Notes