ASHISH PATIL (Customer) asked a question.

July 28, 2026 at 8:56 AM
Is there a supported, index-driven way to count matching sub-elements (not documents) in place — without fn:count and without splitting into per-entry documents or fragment roots?

Data model: A <catalog-record> contains many repeating <measurement> children (one to several thousand per document). We need to count matching <measurement> entries across the corpus, not the parent documents.

 

The problem: Every counting mechanism we've tried counts documents (fragments), not sub-elements. cts:estimate, cts:values frequencies, cts:count-aggregate, and search facets all return per-fragment counts. fn:count(cts:search(...)/measurement[...]) gives the correct answer but is too expensive at scale - it materializes every matching node.

 

Reproducible example:

  1. <!-- /demo/r1.xml — 2 matching measurements -->
  2. <catalog-record id="R1">
  3. <measurement><attribute-name>alpha</attribute-name><value>75</value></measurement>
  4. <measurement><attribute-name>alpha</attribute-name><value>85</value></measurement>
  5. <measurement><attribute-name>beta</attribute-name><value>80</value></measurement>
  6. </catalog-record>
  7.  
  8. <!-- /demo/r2.xml — 1 matching measurement -->
  9. <catalog-record id="R2">
  10. <measurement><attribute-name>alpha</attribute-name><value>88</value></measurement>
  11. </catalog-record>

Query : attribute-name = alpha AND value between 70–90, scoped to one <measurement>:

  1. cts:element-query(
  2. xs:QName("measurement"),
  3. cts:and-query((
  4. cts:element-value-query(xs:QName("attribute-name"), "alpha"),
  5. cts:path-range-query("/catalog-record/measurement/value", ">=", 70.0),
  6. cts:path-range-query("/catalog-record/measurement/value", "<=", 90.0)
  7. ))
  8. )
  1. Correct answer (matching <measurement> entries) 3
  2. cts:estimate / any index API returns 2 (documents)

Why chunking is not an option: At production scale this produces ~33 billion tiny sub-KB documents - an unreasonable number of fragments to load, index, and maintain. Fragment-root fragmentation on <measurement> is ruled out for the same reasons.

  • 4 answers
  • 113 views
Mads Hansen and Chris Maden like this.

  • Bryan Shalke (Progress Software)

    For completeness' sake, I also tested the same data with TDEs. I made a very simple TDE with one table and one triple:

    1. {
    2. "template": {
    3. "context": "//measurement",
    4. "collections": ["measurements"],
    5. "rows": [{
    6. "schemaName": "test",
    7. "viewName": "measurements",
    8. "viewVirtual": true,
    9. "columns": [
    10. {
    11. "name": "name",
    12. "scalarType": "string",
    13. "val": "attribute-name"
    14. },
    15. {
    16. "name": "value",
    17. "scalarType": "integer",
    18. "val": "value"
    19. }
    20. ]
    21. }],
    22. "triples": [
    23. {
    24. "subject": { "val": "sem:iri(xdmp:node-uri(.) || xdmp:path(.))" },
    25. "predicate": { "val": "sem:iri('http://example.com/measurement/' || attribute-name)"},
    26. "object": {"val": "xs:integer(value)"}
    27. }
    28. ]
    29. }
    30. }

    I tested the table as both virtual and non-virtual with the following SQL:

    1. SELECT COUNT(*) FROM measurements WHERE name = 'alpha' AND value < 87

    The virtual version took ~12.5 seconds and the non-virtual version took ~1.5 seconds.

     

    I tested the triple version with the following SPARQL:

    1. SELECT (COUNT(*) AS ?count) {
    2. ?measurement <http://example.com/measurement/alpha> ?value .
    3. FILTER (?value < 87)
    4. }

    This took ~0.5s, making it the performance champ, and the most scalable option if you can afford the indexing time and space (unsurprising, as it is a single range lookup on the triple index)

     

    You may be wondering why I embedded the measurement name in the predicate instead of having two triples that share a single subject. This would be extremely non-performant. Unlike rows, MarkLogic assumes triples are not grouped by any particular fragment. This is because you can easily generate the same subject on multiple documents. Because MarkLogic doesn't know that other matching triples don't exist (potentially on other nodes), it will need to collect all measurement-name triples with an object of "alpha" and all measurement-value triples with an object < 87 and do a join on the E-Node. If you use the Optic API directly, you can get around this by specifying a fragment column in op.fromTriples, but this is still slower than a meaningful predicate and the join isn't completely avoided, just pushed down to the D-Node. You may as well use rows if you really want different triples for name vs value. Dynamic predicate IRIs are a very useful feature of TDEs, and I personally think they are extremely underutilized!

    Selected as Best

  • David Ennis (Clever Llamas)

    Add a TDE template on the repeating element andgenerate a one column table or a set of triples per document. The. Use grouping queries with count

  • Bryan Shalke (Progress Software)

    The reason you can't do this without materializing the node is because you have two values you're trying to filter on - measurement and attribute name - and the indexes alone can't tell whether these coexist under the same parent element.

     

    As others have said, one option is to use a TDE. A virtual TDE might be sufficient to avoid reindexing, but a materialized TDE is probably best for performance here.

     

    Another option is to change your document structure to be less rigid. If, instead of having a fixed "attribute-name" element, you named the element descriptively, you no longer need to compare two different elements to retrieve the value. To demonstrate this, I ran the following code to generate 10k documents in the database with between 1 and 100 measurements containing both ways of encoding the data:

    1. xquery version "1.0-ml";
    2.  
    3. let $measurements := ("alpha", "beta", "gamma", "delta", "epsilon")
    4. let $options := <options xmlns="xdmp:document-insert">
    5. <permissions>{xdmp:default-permissions()}</permissions>
    6. <collections>
    7. <collection>measurements</collection>
    8. </collections>
    9. </options>
    10.  
    11. for $i in 1 to 10000
    12.  
    13. let $doc := <catalog-record id="R{$i}">
    14. {
    15. for $j in 0 to xdmp:random(99) + 1
    16. let $index := xdmp:eager(xdmp:random(fn:count($measurements) - 1) + 1)
    17. let $m := $measurements[$index]
    18. let $v := xdmp:random(100)
    19. return <measurement>
    20. <attribute-name>{$m}</attribute-name>
    21. <value>{$v}</value>
    22. {element { $m } { $v }}
    23. </measurement>
    24. }
    25. </catalog-record>
    27. return xdmp:document-insert("/measurements/" || $i || ".xml",
    28. $doc,
    29. $options
    30. )

    Running the following query takes ~30 seconds on my local machine:

    1. fn:count(/catalog-record/measurement[value < 87 and attribute-name = "alpha"])

    Running this query takes <5 seconds and returns the same result:

    1. fn:count(/catalog-record/measurement[alpha < 87])

     

  • Bryan Shalke (Progress Software)

    For completeness' sake, I also tested the same data with TDEs. I made a very simple TDE with one table and one triple:

    1. {
    2. "template": {
    3. "context": "//measurement",
    4. "collections": ["measurements"],
    5. "rows": [{
    6. "schemaName": "test",
    7. "viewName": "measurements",
    8. "viewVirtual": true,
    9. "columns": [
    10. {
    11. "name": "name",
    12. "scalarType": "string",
    13. "val": "attribute-name"
    14. },
    15. {
    16. "name": "value",
    17. "scalarType": "integer",
    18. "val": "value"
    19. }
    20. ]
    21. }],
    22. "triples": [
    23. {
    24. "subject": { "val": "sem:iri(xdmp:node-uri(.) || xdmp:path(.))" },
    25. "predicate": { "val": "sem:iri('http://example.com/measurement/' || attribute-name)"},
    26. "object": {"val": "xs:integer(value)"}
    27. }
    28. ]
    29. }
    30. }

    I tested the table as both virtual and non-virtual with the following SQL:

    1. SELECT COUNT(*) FROM measurements WHERE name = 'alpha' AND value < 87

    The virtual version took ~12.5 seconds and the non-virtual version took ~1.5 seconds.

     

    I tested the triple version with the following SPARQL:

    1. SELECT (COUNT(*) AS ?count) {
    2. ?measurement <http://example.com/measurement/alpha> ?value .
    3. FILTER (?value < 87)
    4. }

    This took ~0.5s, making it the performance champ, and the most scalable option if you can afford the indexing time and space (unsurprising, as it is a single range lookup on the triple index)

     

    You may be wondering why I embedded the measurement name in the predicate instead of having two triples that share a single subject. This would be extremely non-performant. Unlike rows, MarkLogic assumes triples are not grouped by any particular fragment. This is because you can easily generate the same subject on multiple documents. Because MarkLogic doesn't know that other matching triples don't exist (potentially on other nodes), it will need to collect all measurement-name triples with an object of "alpha" and all measurement-value triples with an object < 87 and do a join on the E-Node. If you use the Optic API directly, you can get around this by specifying a fragment column in op.fromTriples, but this is still slower than a meaningful predicate and the join isn't completely avoided, just pushed down to the D-Node. You may as well use rows if you really want different triples for name vs value. Dynamic predicate IRIs are a very useful feature of TDEs, and I personally think they are extremely underutilized!

    Selected as Best