Geert Josten (Datavid) asked a question.
This is probably old stuff, and might be speaking of undocumented features, not 100% sure. We (SensingClues) have been using an import using MLCP, which has been using /data-hub/5/transforms/mlcp-flow-transform.sjs for as long as I remember to trigger the running of a datahub flow as part of the ingest.
Recently we discovered that something wasn't working anymore. After investigating, I discovered that that module has logic to run a flow once per batch, but that didn't seem to work anymore as transforms are now somehow executed in a different way so that execution context was no longer shared. Maybe because of a change in MLCP, or ML?
The workaround is to use batch_size 1 (and compensate with a higher thread_count if needed/possible).
I mostly wanted to share this for posterity, and it might be something worth checking in latest datahub framework. We are still running DHF 5.8.1, and ML 11 currently, and working on replacing this old approach entirely.
When reporting that something isn't working (especially when it used to), then it's helpful to provide error messages and specific version numbers of the components, especially given the (sometimes breaking) changes introduced recently to remediate Security issues for Server and MLCP.
https://developer.marklogic.com/products/support-matrix/#mlcp
Important backward incompatibility exception:
Previous versions of MarkLogic Content Pump (mlcp) are incompatible with MarkLogic Server 12.1.0 and later and MarkLogic Server 11.3.7 and later.
A change to how isolated code executes in MarkLogic Server 12.1.0 and MarkLogic Server 11.3.7 causes this incompatibility. Attempts to run previous versions of mlcp with either server release result in the following error:
java.io.IOException: com.marklogic.xcc.exceptions.XQueryException:
XDMP-ISOLATED: xdmp:get-request-header("x-forwarded-for")
-- This function can't be used because code runs isolated.
When upgrading to MarkLogic Server 12.1.0 or MarkLogic Server 11.3.7, we recommend that you upgrade mlcp to 12.1.0 or 11.3.7, respectively.
If you must continue to use an older version of mlcp, set
-restrict_hosts to true. However, using this setting can degrade performance. For more information, see the following sections of MarkLogic Content Pump (mlcp)
Does that also apply for ML 11.3.6 and MLCP 11.0.2? I upgraded to MLCP 11.3.7 2 days ago, but that didn't make a difference. We have been using DHF 5.8.1 for 3 years now, so ruling that out.
Also, confusingly, there was no error of any kind to be found. I even enabled hub-flow trace event, and it didn't log anything. Despite MLCP logging it processed many records, zero were written to the database. I had to dig in and debug datahub code to find the issue.
What is the purpose of restricting hosts in this scenario?
Note though, we are not looking for a solution, we are moving away from this import anyhow. I just wanted to give a heads up. Also, batch_size 1 gave us a workable solution until we have fully moved over.
Cheers,
Geert