Marc Messéant (Customer) asked a question.

Intermittent ADMIN-DUPLICATESCHEDULEDTASKIDS on POST /manage/v2/tasks, although the task is created (multi-host cluster)

MarkLogic v11 ml-gradle 6.2.0

Symptom

mlDeployApp fails randomly while creating a scheduled task. The failing task changes from one run to the next:

Deleting task at path /manage/v2/tasks/17163246838362614268?group-id=Default

Deleted task at path /manage/v2/tasks/17163246838362614268?group-id=Default

Creating task: /ext/schedules/xxx-task.xqy

POST /manage/v2/tasks?group-id=Default

{"errorResponse":{"statusCode":"500", "status":"Internal Server Error",

"messageCode":"ADMIN-DUPLICATESCHEDULEDTASKIDS",

"message":"ADMIN-DUPLICATESCHEDULEDTASKIDS: (err:FOER0000) Duplicate scheduled task ids.: %s"}}

 

After the failure:

  • the group configuration contains no duplicate task id (admin:group-get-scheduled-tasks());
  • the task was created, exactly once, with a new id (17079391720829777448, not the deleted one).

The POST therefore succeeded, yet the client got a 500.

.

Only one deploy was running, and no other client was using the Manage API.

 

What we found in the Manage API code (11.3.4)

  • ml-app-deployer never updates a task: an existing task is deleted, then created again. Every deploy POSTs all the tasks.
  • tpmod:create-task() (/MarkLogic/manage/models/task-properties-model.xqy) builds the gr:scheduled-task element, with its task-id from xdmp:random(), before calling config-lock:lock-config().
  • Inside the lock, tpmod:do-create-task-locked() calls admin:group-add-scheduled-task() and admin:save-configuration(). This runs in the update transaction opened by hof:apply-update-in() on App-Services (/MarkLogic/Admin/config-lock.xqy).

Our hypothesis: admin:save-configuration() writes groups.xml immediately. If this update transaction is then restarted (lock conflict or deadlock between hosts, possibly with the config-lock taken by the preceding DELETE), the function runs again with the same task element and the same id. The configuration already contains that task, so the duplicate check in admin:group-add-scheduled-task() fails. That would explain why the task exists once with the new id while the client receives the error.

 

Workaround we use

We subclassed TaskManager in our build.gradle. On this error code only, it checks that exactly one task with that task-path now exists, and if so treats the creation as successful. Otherwise it rethrows the error.

 

This works, but we would prefer a server-side fix.

 


Mads Hansen likes this.