OpenEdge 11.6 | How to find a memory leak on AppServer - Forum - OpenEdge Development - Progress Community

OpenEdge 11.6 | How to find a memory leak on AppServer

 Forum

OpenEdge 11.6 | How to find a memory leak on AppServer

This question is not answered

Hello All,

We're investigating a memory leak in our application.

Is there any utility or any setting which will report the memory leaks in the AppServer source code.

Any tips on how to tackle this kind of issues is appreciated!

Regards,

Atul Dalvi

All Replies
  • You've probably done a quick search before but that's how Progress suggest finding memory leaks

  • There are a few approaches you can take. These apply to all AVM/clients
     
    1. Use the DynObjects.* logging type to identify when OO and other objects are created and destroyed. Parse the logs generated to see what isn’t appropriately destroyed. There are knowledgebases and Communities posts on how to do this
    2. Do the same for Temp-Tables ; these entries also help you identify when you are unnecessarily passing temp-table data by-value (instead of by-reference)
    3. For AppServers, identify the start and end of a request (in activate and deactivate event procedures) the objects, persistent procedure and other stuff that’s in memory. Look at what’s increased in the deactivate, and decide whether that’s a leak. Use the SESSION handle’s FIRST-OBJECT, FIRST-PROCEDURE, FIRST-BUFFER, FIRST-TABLE and any other FIRST-* reference, and “walk the tree” by checking that reference’s NEXT-SIBLING until there are no more valid-handle/valid-objects.
     
    Notice that you will need to decide whether additional/remaining references at the end of a request are leaks or not – in some cases there will be objects cached deliberately, which should not be considered leaks. There are some conversations on Communities in this vein.
     
    For (3) above, you will have to add ABL code to gather the information; in newer versions of PASOE there are mechanisms to get some of this data without having to write ABL, though the analysis about leak-or-not will be the same.
     
    Hth,
    - peter
     
     
  • Classic appserver or PASOE?  How fast is the leak?  Can you quantify it?  Almost all apps leak to some extent, and what is more interesting is how fast (1 GB an hour/day/week).  A "leak" can also be defined in different ways.  (Eg. Peter gave the example of objects that are cached deliberately, which may appear like a leak until closer inspection.  Another example is the use of classes that have static member data.  Or even the runtime itself may "leak" if it accumulates a lot of r-code in memory, or chooses not to instantly release memory allocations back to the OS in favor of keeping them for some future reuse.)

    If there is in fact a measurable leak, then PASOE (if that is what we are talking about) will amplify if.  This is  because there are lots of ABL sessions that are hosted in a single process ( _mproapsv).

    This question just came up a week ago, you may want to look at this thread too:

    community.progress.com/.../57654

  • What suggested was what we did at a previous customer: we had a special program to testrun a procedure. Before the start of the program we created a fingerprint of all persistent procedures and tables currently active in your session. Then we ran the program, did an UNDO and made a second list of persistent stuff. We removed everything from the list that was already there (tip: store handle in a tt) and what remains is the stuff your program created but did not clean up.

  • Just a caveat when you do use the NEXT-SIBLING method: In some cases you might get a handle / object back, but when you do a VALID-HANDLE or VALID-OBJECT on it it will return FALSE. It typically happens when you've created something inside a widget-pool and destroy the widget-pool later, but nowhere there is an explicit DELETE on that handle / object. Personally I think this is a bug because that should've be cleaned up as part of the widget pool, but for now you need to ignore these invalid handles / object and do another NEXT-SIBLING (yes you can do that, even though they don't point to a valid handle / object). If you don't you'll simply stop scanning the rest of the chain.

  • That knowledgebase.progress.com/.../P133306 article and its associated script came in very handy for us just this Monday as we were chasing a ** stget out of storage error. Added the DynObjects.* loging and run the output through the script only to discouver some dynamic queries being created and never deleted... CREATE WIDGET-POOL is my new friend.

  • Hello Peter,

    Thanks for the reply. Actually, we have a PAS server specially running for the OpenEdge soap web services only. We have couple of OpenEdge soap services deployed on this PAS server and hundreds of programs running and linked under it. There are no OpenEdge classes involved in it only .p programs with Dataset handles, input-output Datasets, temp-tables, Dynamic Queries, buffers etc. I have below doubts on the options you suggested -

    1. Will the Dynamic Objects Logging work in this case ?

    2. Can you give some more details on the option (2) to identify temp-table entries. Any other logging type for temp tables ?

    3. Also how can we implement option(3). I have code for this but Its a session or request specific right ? Can't we implement this at very higher level as a common code for all the web services ? otherwise I have to modify main program of each web service to write this code.

    Thanks in advance.

  • DynObjects tracks the following docs.progress.com/.../Dynamic-object-logging.html

    The temp-table logging is described docs.progress.com/.../Temp-table-logging.html .

    These 2 should give you good insight .

    In PASOE you can always add activate and deactivate procedures, and those will run on every request (for SOAP, WEB, etc). That would be the place to add the checks IMO.

  • Thanks Peter.

  • Please note that if you can run the same application in 11.7.4 or later you can use the PASOE REST API to monitor for leaks.  The below article talks about using the ABL http client to call out to the REST API and includes references to code that will allow you to turn the getABLObjects functionality on/off, and another piece of code that can be used to fetch a list of leaked objects.

    000095195 - How to call into the OEManager's REST API for insight into PASOE?

    knowledgebase.progress.com/.../How-to-call-into-the-OEManager-s-REST-API-for-insight-into-PASOE

    The benefit of using this is that you don't need to change your application code or use parameters to generate a multi-million line log file that you have to then parse in order to compare the creates with the deletes to find the list of leaked objects.  This utility is simply turned on/off as desired and then it does the analysis and simply returns where the leaks are in your code.

    The only pain point I can think of is that you DO have to have the oemanager deployed to your pasoe instance.

    Kenneth S. McIntosh

    Principal Technical Support Engineer

     

     

  • Hello All,

    I tried DynObjects but in log report I did not find anything related to the dynamic objects as I don't have object programming in my code only input output temp tables and datasets and temp tables & datasets are simple not with handle object.

    What else I can check to find out this memory leak on AppSeerver ?

    Thanks in advance !!!

  • > On May 15, 2019, at 5:18 AM, atuldalvi123 wrote:

    >

    > only input output temp tables and datasets and temp tables & datasets are simple not with handle object

    based on what you have said, it is likely that you are creating tem-tables and dataasets that are not being deleted when you don't need them anymore.

    if you start client with

    -clientlog foo -logentrytype "temp-tables:4"

    then this code (admittedly crude) might be helpful and that you can use along with client-logging:

    /* this program matches up temp-table create and delete log messages.

    generated by the 4gl runtime clientlog facility.

    -clientlog foo.log -logentrytype "temp-tables:4"

    just cat your foo.log log files as generated by the 4GL into tt_all.log.

    version 6, gus bjorklund, december 15, 2014.

    */

    using Progress.Database.TempTableInfo.

    def var deb as logical no-undo initial false.

    def var input1 as char no-undo.

    def var input2 as char no-undo.

    def var data_op as char no-undo.

    def var data_name as char no-undo.

    def var data_path as char no-undo.

    def var data_line as char no-undo.

    def var n as int no-undo.

    def var m as int no-undo.

    def var nmatches as int no-undo.

    def var entries as int no-undo.

    /* here we collect info about the orphans

    a row is added for each create.

    the row is eliminated when we see a delete */

    def temp-table ttorphans no-undo

    field tt_op as character

    field tt_name as char

    field tt_path as char

    field tt_line as char

    index ix1 tt_name.

    /* here we collect tt name and number of creates

    and deletes for each one */

    def temp-table ttcounts no-undo

    field theName as char

    field path as char

    field creates as int

    field deletes as int

    index ix1 theName.

    /* read log entries for tt creates and deletes */

    input from tt_all.log.

    inloop:

    repeat:

    import unformatted input1.

    trim (input1).

    /* skip the stuff we don't want */

    if (index (input1, "TEMP-TABLE")

    if (index (input1, "Created")

    data_op = entry (11, input1, " ").

    data_name = entry (13, input1, " ").

    if (index (input1, ")") > 0) then do:

    input2 = substring (input1 , index (input1, ")") + 1).

    data_path = entry (2, input2, " ") no-error.

    data_line = entry (4, input2, " ") no-error.

    if deb then

    display data_op format "x(1)"

    data_name format "x(25)"

    data_path format "x(30)"

    data_line format "x(6)"

    .

    end.

    if (data_op <> "Created") and (data_op <> "Deleted") then next inloop.

    entries = entries + 1.

    if (entries mod 25000) = 0 then display entries.

    if (entries > 1000000) then leave.

    find ttcounts where theName = data_name no-error.

    if not available ttcounts then do:

    create ttcounts.

    assign theName = data_name

    path = data_path

    creates = 0

    deletes = 0

    .

    end.

    if (data_op = "Created") then creates = creates + 1.

    else deletes = deletes + 1.

    find first ttorphans where (data_name = tt_name) and (tt_op <> data_op) no-error.

    if available ttorphans then do:

    /* have matching tt , get rid of old, drop new */

    nmatches = nmatches + 2.

    delete ttorphans.

    next inloop.

    end.

    /* no match, so add a new record */

    create ttorphans.

    assign tt_op = data_op

    tt_name = data_name

    tt_path = data_path

    tt_line = data_line

    .

    end.

    message entries " records read," nmatches " matched".

    output to nodelete.txt.

    for each ttcounts where (creates > 0) and (deletes <> creates):

    put theName format "x(20)"

    (creates - deletes)

    skip.

    end.

    output close.

    n = 0.

    m = 0.

    output to nodelete2.txt.

    for each ttorphans:

    put tt_path format "x(30)"

    tt_name format "x(30)"

    skip.

    if tt_op = "Created" then n = n + 1.

    if tt_op = "Deleted" then m = m + 1.

    end.

    output close.

    message n " creates and " m " deletes don't match".

    def var tableCount as int no-undo.

    def var procName as char no-undo.

    def var infoHandle as handle no-undo.

    repeat n = 1 to TempTableInfo:TempTableCount:

    TempTableInfo:GetTableInfoByPosition (n, infoHandle, procName).

    display infoHandle:name procName.

    end.

  • You said you don’t have “object programming”, and only TT, DataSet parameters, no handles.  But previously you said you have dynamic queries and buffers. Those are tracked with the DynObjects logging.

  • I have a support case open with Progress at the moment regarding a memory leak in the _mproapsv.  I can give you more details if you need them; I think the SP will be available in 11.7.5.  Your issue may not be the same as ours.  It would be helpful if you gave us a lot more explanation about your problem.  How fast is the leak?  Can you quantify it?   What process name is actually leaking (_mproapsv.exe? java.exe? tomcat.exe?)

    One thing I would suggest doing is create a reproducible that you can play with *outside* of production.  You can run your repro at high speed to amplify the problem.  That may also allow you to isolate the portion of your code that leaks.  In the end you might be left with something that you can send over to Progress tech support.

    Are you running on windows or linux?  If windows then I would recommend a sysinternals tool:

    docs.microsoft.com/.../vmmap

    This tool is easy to use.  And should help isolate the memory leak to a certain degree, and also quantify it.  I suspect your leak will be seen as allocations in the native heap.

    Another thing I would do ... especially since your question hasn't been resolved after a month ... is to approach the PASOE problem from another angle.  You can install the "oemanager" API and use a REST interface to trim your inactive ABL sessions on a recurring schedule (eg. on the hour or half hour).  This is a very safe operation and could be enough to resolve a wide range of so-called "leaks".  (Depending on the definition of "leak".)  In the very least it will give you additional information to add to everything else that you know about your "leak".

    Trimming inactive ABL sessions can be accomplished via the OE manager rest interface. Here is a powershell script that shows how to trim inactive sessions from an ms-agent process:

    community.progress.com/.../36461

    (Posted by me on 26 Dec 2017 14:04, with lots of help from Irfan)

    When an inactive session is trimmed, PASOE will clear out all the memory that is directly associated with the session.  That may be half the battle.  When new PASOE client requests arrive, PASOE will create fresh new sessions to replace the ones that were trimmed.  

    Yet another thing you may want to look at is the tomcat manager console:

    httt://localhost:8810/manager/html

    The console may help in your troubleshooting, and may show when HTTP sessions are open for long periods of time.  I don't know about SOAP, but where APSV is concerned sometimes the HTTP sessions can become orphaned for long periods of time and hold onto resources in the MSagent, thereby causing "leaks".  This can be resolved administratively in the tomcat manager console by explicitly forcing the expiration of the HTTP sessions.

    Good luck isolating the leaks.  As a side, I will point out that even Java and .Net code can have "leaks" when items on the heap are rooted in memory and the garbage collector is unable to clear everything out.  It would be nice if PASOE had an operation that was analogous to forcing a GC.Collect() operation, which is something to help determine when there is truly a leak.  The closest thing I've found to this (in the world of PASOE) is to trim inactive ABL sessions via the OE manager REST api.

  • Hi,

    PASOE?

    Have you heard or worked with swagger?

    With swagger you can identify the program that is causing the memory leak as well as the line that is not deleting DATASET.

    Tell me if you want to know more than I can tell you.

    Regards