Anthony Hashemi (Customer) asked a question.

On a couple of occasions where our Marklogic cluster has been under heavy load, it has resulted in a couple of large requests being stuck in a "cancelling" status for around 20 hours or so. Is this normal, and how can we handle it within a day?

On a couple of occasions where our Marklogic cluster has been under heavy load, it has resulted in a couple of large requests being stuck in a "cancelling" status for around 20 hours or so. 

 

During that period, the CPU of the EC2 instance one of the nodes stays at a higher baseline.

 

Is this normal, and how can we handle it within a day?


  • Mads Hansen (Progress Software)

    No, it's not normal. Sounds like something has gone wrong. Normally, when queries are in "cancelling" state, there is some rollback or cleanup happening, but it should not take hours.

     

    What sort of appserver is this happening on? HTTP, XDBC, ODBC?

  • Anthony Hashemi (Customer)

    Hi Mads,

     

    Thanks for the swift response. That was my understanding but the 2 requests I have seen, I believe were just querying the database so I assumed no cleanup would be necessary and indeed would be much quicker than 20 hours if so. I believe this is on HTTP.

     

    James Kenwood reached out and suggested I create a support case which I have done, so hopefully will get to the bottom of it there.

     

    I will update here when we get to the bottom of it.

     

    Thanks again,

    Anthony

     

     

     

  • Mads Hansen (Progress Software)

    On the host that is showing the "cancelling" request, run `sudo perf top` to see if it shows what it's doing. You could capture perf report and upload to the Support case. https://community.progress.com/s/article/using-perf-to-understand-performance

     

    If the output of perf looks like it's "stuck" performing gzip compression (i.e. top items showing time spent in slide_hash, crc32_z, longest_match), then you might try disabling gzip compression for the HTTP appservers by applying the "HTTPServer Disable Compression" trace event in the Diagnostics page (Groups > group_name > Diagnostics ) of the Admin UI.

     

     

  • Anthony Hashemi (Customer)

    Unfortunately we have not been able to reproduce the issue yet. We have observed 2 distinct occurrences, generally when the instances have been under sever,e unexpected load. If we have capacity, we will try to reproduce it in a lower environment, and follow those steps you outlined.

     

    Thanks,

    Anthony