ASHISH PATIL (Customer) asked a question.

Excluded paths have no effect on an existing database, even after manual reindex and merge

MarkLogic Server 12.1.0, single-node, single-forest.

 

What we did:

1. Loaded ~1000 XML documents into a database with default indexing config.

2. Configured 5 index-exclusion-paths entries via the Management API.

3. Waited to see if the reindexer picked up the change automatically.

4. Manually triggered a reindex, then a database merge.

5. As a separate run, cleared the database, configured the same excluded

  paths on the empty database, then reinserted the same 1000 documents.

 

What we observed:

- After step 2, automatic reindexing did not trigger.

- After step 4 (manual reindex and merge), there was no effect - verified

 via before/after size metrics and via word search (content inside the

 excluded paths was still returned by cts:word-query).

- After step 5 (clear, configure, then reload), the universal index was

 reduced as expected: 71.5% smaller, with content size unchanged, and

 word search no longer matched content inside the excluded paths.

 

No data loss in either case.

 

Workaround: clear the database, apply the excluded-paths config first,

then load documents.

 

Question: Is there a supported way to apply excluded paths to an existing

database and have it actually take effect, without a full clear-and-reload?


  • ASHISH PATIL (Customer)

    Just learned that this has been accepted as a P1 bug. Looking forward to the fix patch.

    Selected as Best
  • Mads Hansen (Progress Software)

    Double check that your reindex test didn't have a mistake or issue with the XPath. I just did a quick test adding an excluded path on a 12.1.0 instance, and it triggered a reindex, and then my search for a word in an element matching that excluded path that previously returned the doc no longer did.

  • David Ennis (Clever Llamas)

    Hmm. Reindex enabled and setting the reindex to current timestamp should also trigger a full reindex.

  • James Kenwood (Progress Software)

    Hi Ashish, did Mad's response help answer your issue? If not please let us know. We could also probably handle this via a Support Ticket if needed. Feel free to reach out to me if you have any difficulties logging this.

  • ASHISH PATIL (Customer)

    Following up on this ticket, as requested we retested the behavior on MarkLogic 12.1.0, this time on two different systems with a higher configuration than our first attempt.

    There is definitely an issue with this feature. On both systems, adding excluded paths to a database that already holds documents triggered a reindex automatically, but the reindex never completed. It stayed stuck in a continuous merging / reindexing / refragmenting loop for more than 8 hours, on a database of only 1000 documents.

    • System 1: still stuck after more than 12 hours.
    • System 2: stuck for more than 8 hours, after which we cleared the database.

    A reindex/merge of 1000 documents should take minutes (seconds actually), not hours.

  • Mads Hansen (Progress Software)

    Can you provide examples of the excluded paths that were applied? It would be helpful if your provided details, in order to attempt to reproduce and understand the issue.

     

    For instance, do the paths have predicates, namespaces, wildcarded namespaces, do the paths match JSON or XML docs, etc? If there is an issue, it helps to provide sufficient information so that others can attempt to reproduce.

    • ASHISH PATIL (Customer)

      Here are the excluded paths we applied. The element names are replaced with unrelated ones, but the structure, depth and naming style are the same as the real ones.

       

      Paths applied (5):

       

      /book/chapter/short-chapter/illustration

      /book/chapter/collected-edition/volume/chapter/short-chapter/illustration

      /book/chapter/appendix

      /book/chapter/searchable-glossary

      /book/searchable-preface

       

      Your questions:

      - Predicates: none. They are plain element steps.

      - Namespaces: none. The documents have no namespace, and the paths use no prefixes.

      - Wildcards: none, in names or namespaces. There is no *, // or *:.

      - Document type: XML only, no JSON. Every document is an XML file with root element "book".

      - Path form: absolute from the root, 2 to 7 steps deep. Element paths only, with no attribute steps.

       

      How they were applied: a Management API PUT /manage/v2/databases/{db}/properties with Content-Type: application/xml, using <index-exclusion-paths> with one <excluded-path><excluded-path-expression> per path. It returned 204, and the properties read back with all 5 paths.

       

      About the data (1000 XML docs):

      - "chapter" is a large subtree that holds most of each document, with a lot of encoded text.

      - "appendix" and "searchable-glossary" are the biggest paths, about 25% and 41% of all document text.

      - "illustration" (under both "short-chapter" paths) is small in text but has a lot of child elements, about 140,000 across the set.

      - "collected-edition" and "volume" are the multi-part variants, so the second path exists in about 7% of the docs.

      - /book/searchable-preface matches nothing in this data set, so it should be harmless.

       

       

      • ASHISH PATIL (Customer)

        Also,

         

        The reindex on the DB is still running, about 20 days after we added the excluded paths (Oct 13 to Nov 2). It has never finished, on a database of only 1000 documents.

         

        The forest is still about 4.3 GB against a 169 MB baseline, and merges are still running. About 102,000 new stands have been created since our last check.

         

      • Mads Hansen (Progress Software)

        I'm not yet able to reproduce any of those issues. I created a handful of docs that would match the exclusion paths, and added them to the config. Word queries no longer matching and reindex didn't get "stuck".

         

        I know you had opened a Support ticket, but didn't see anything attached (ErrorLog.txt to see what progress was being made when reindexing or if maybe it's failing and retrying on something and spinning it's wheels).

         

        If you look at the Database Status tab and click the "Show Reindex" button at the top, what does it say it's indexing and the status of that?

10 of 11