Investigating memory leaks - ABL memory profiler?

Posted by Lieven De Foor on 03-Jul-2018 09:44

Hi,

We're investigating a memory leak in our application causing .NET forms to not get destructed on close.

We've been able to gather a list of leaking objects using client logging with DynObjects and leakcheck.p (https://knowledgebase.progress.com/articles/Article/P133306).

While this shows you a list of what's leaking, it is not showing the cause of the leak i.e. the objects that are keeping the references and thus preventing garbage collection.

Explicitly deleting and/or disposing the Form and ToolbarsManager + related objects (Ribbon, Tools, etc.) to get the objects released causes the destructor to run but seems to cause the session to become unstable under some circumstances.

This explicit delete is also not attacking the problem at the root, but is rather symptom fighting. (in .NET you can't even do this and have to rely solely on the garbage collector to clean up objects).

By using the ANTS memory profiler we also see hybrid objects staying alive on the .NET side, while their ABL counterpart is gone (i.e. destructor has run). In fact we've seen cases where the ABL side apparently has no more/few leaks, where the .NET is leaking a lot.

Are there any tools/options to debug this on the ABL side as well?

Should we consider zombie objects (i.e. still living on the .NET side of the bridge, but gone in the ABL side) as OpenEdge bugs or not?

Any tips on how to tackle this kind of issues is appreciated!

All Replies

Posted by Laura Stern on 03-Jul-2018 09:59

It is not necessarily wrong that that the ABL side is gone but the object lives on the .NET side.  What I mean by that is, if you explicitly delete the object in the ABL, we will clean up our side and our references to the .NET part of the hybrid object.  But as you just said, in .NET, you cannot force the object to go away.  You can only free references and hope it gets garbage collected.  If there are still references on the .NET side, it will remain there.

Are you using Infragistics controls?  We have had many cases where the form is not GC'd in .NET because of references within Infragistics components.  In fact we've logged bugs with Infragistics, but they were not resolved.  

Also, have you forced .NET to do its garbage collection?  I wouldn't recommend putting this into the application.  It is just a diagnostic technique to make sure you are not seeing objects that appear to be leaking but just haven't been GC'd yet.  I believe the memory profiler will have a way of doing that.  To do it programmatically, you do this:

System.GC:Collect().

System.GC:WaitForPendingFinalizers().

System.GC:Collect().

What do you mean by unstable? Maybe you have gone too far in your attempt at cleaning up.  Calling Dispose and doing the DELETE OBJECT on the ABL side is often enough.  And you have to make sure you do this in the Closed event handler, not the Closing event handler.

Posted by Lieven De Foor on 03-Jul-2018 10:33

Hi Laura,

Yes, we are using Infragistics controls, and yes we've figured that these might be the main cause of issues.

We're using Infragistics controls and UltraToolbarsManager extensively and reading articles like this one isn't really helping our mood: subjectively.blogspot.com/.../importance-of-recycling-memory.html

By unstable I mean session crash in certain cases, possibly by cleaning up too rigorously... or by just hitting a bug in Infragistics

Reading this KB article, am I right to conclude that references on both sides of the bridge can keep objects alive, and that cleaning up on either side can make the object available for garbage collection, assuming no references are kept on the other side? I've seen the occasional error where the .NET side expected something to be still there on the ABL side after a DELETE OBJECT, but can't readily reproduce so I can't provide the error number...

If there are still references on the .NET side that we can't reach (bugs in Infragistics?), then we will never be able to have the objects properly deleted...

I've taken the example of https://knowledgebase.progress.com/articles/Article/000052860 and ran that through the ANTS memory profiler, which clearly shows objects (not only Infragistics) staying around, while the destructor gets run on the ABL side. Is there any update on this bug (PSC00311825)?

Creating the same in c# however, and having the GC run spontaneously, does not show these left-overs.

Is this hinting to an OpenEdge bug instead of an Infragistics bug?

We will experiment with explicitly calling the GC.

Thanks

Posted by Laura Stern on 03-Jul-2018 10:59

Yes, that article sounds like exactly what we were running into.  When we reported it to Infragistics, they claimed that there was no bug.  

Yes, references on either side of the bridge will keep an object alive.  If all references go away, the object will be garbage collected.  In all the cases I've looked at, there were no references left from the ABL side.  If I had found one, the bug has already been fixed. And you're correct, if there are still references on the .NET side, it's possibly tough to make the object go away.  But I did find that calling Dispose() and doing DELETE OBJECT on the form was usually enough.  We did make a change in 11.7.2 such that we will internally call Dispose() on the form once it is closed if the ABL has not done so.  So DELETE OBJECT would be enough.  Did you actually try just that?

Posted by jquerijero on 03-Jul-2018 11:16

[quote user="Lieven De Foor"]

Hi Laura,

Yes, we are using Infragistics controls, and yes we've figured that these might be the main cause of issues.

We're using Infragistics controls and UltraToolbarsManager extensively and reading articles like this one isn't really helping our mood: subjectively.blogspot.com/.../importance-of-recycling-memory.html

By unstable I mean session crash in certain cases, possibly by cleaning up too rigorously... or by just hitting a bug in Infragistics

Reading this KB article, am I right to conclude that references on both sides of the bridge can keep objects alive, and that cleaning up on either side can make the object available for garbage collection, assuming no references are kept on the other side? I've seen the occasional error where the .NET side expected something to be still there on the ABL side after a DELETE OBJECT, but can't readily reproduce so I can't provide the error number...

If there are still references on the .NET side that we can't reach (bugs in Infragistics?), then we will never be able to have the objects properly deleted...

I've taken the example of https://knowledgebase.progress.com/articles/Article/000052860 and ran that through the ANTS memory profiler, which clearly shows objects (not only Infragistics) staying around, while the destructor gets run on the ABL side. Is there any update on this bug (PSC00311825)?

Creating the same in c# however, and having the GC run spontaneously, does not show these left-overs.

Is this hinting to an OpenEdge bug instead of an Infragistics bug?

We will experiment with explicitly calling the GC.

Thanks

[/quote]

The result of PSC00311825 was the addition of  THIS-OBJECT:ComponentsCollection:ADD(THIS-OBJECT:components) to the PDSOE form template after the InitializeComponent().

Based on our testing using JetBrains, it did help some to reclaim leaked memory, but it did not remove all the memory leaks, for example, System.String references were still not being cleaned up correctly, which for UltraToolbarManager control could be significant.

We are now testing 11.7.3, and the memory leak issue is showing up again.

Posted by Lieven De Foor on 04-Jul-2018 07:44

the code jquerijero mentions (simplified/extracted):

CLASS MyForm INHERITS Progress.Windows.Form:

    DEFINE PRIVATE VARIABLE components AS System.ComponentModel.IContainer NO-UNDO.

    DEFINE PRIVATE VARIABLE ultraToolbarsManager1 AS Infragistics.Win.UltraWinToolbars.UltraToolbarsManager NO-UNDO.

    ...

    CONSTRUCTOR MyForm():

        ...

        InitializeComponent()

        THIS-OBJECT:ComponentsCollection:Add(THIS-OBJECT:components).

    END CONSTRUCTOR.

    METHOD PRIVATE VOID InitializeComponent():

        THIS-OBJECT:components = NEW System.ComponentModel.Container().

        ...

        THIS-OBJECT:ultraToolbarsManager1 = NEW      Infragistics.Win.UltraWinToolbars.UltraToolbarsManager(THIS-OBJECT:components).

    END METHOD.

END CLASS.

I have some questions about:

  • Is there some mechanism working behind the scenes here to dispose all components?
  • ComponentsCollection is a new property on Progress.Windows.Form, but it is undocumented. What is the use?

The code that Visual Studio generates is similar, but not quite the same:

partial class Form2
{
    private System.ComponentModel.IContainer components = null;

    protected override void Dispose(bool disposing)
    {
        if (disposing && (components != null))
        {
            components.Dispose();
        }
        base.Dispose(disposing);
    }    

    private void InitializeComponent()
    {
        this.components = new System.ComponentModel.Container();

        this.ultraToolbarsManager1 = new Infragistics.Win.UltraWinToolbars.UltraToolbarsManager(this.components);

    }    

}

  • Is the ABL code equivalent to the c# Dispose override?
  • Why can't the protected Dispose method be overridden in ABL? I get a compile error that it is final, though I can do this in c#. How come?   

Posted by Laura Stern on 05-Jul-2018 08:02

The code in Progress.Windows.Form does the equivalent of what C# is doing.  i.e., When Dispose is called on the form, it runs the components collection and calls dispose on each item in there.

I will address the Dispose/Final question under your other post.

But bottom line - why do you need to override the Dispose  method?  Clearly you are trying to resolve your memory leak issue.  What do you intend to do in the override that you think will help?

Posted by Lieven De Foor on 06-Jul-2018 08:45

Ensuring Dispose() gets called explicitly for MdiChildren (for which it is not called automatically) and child controls of a UserControl (for which it should work automatically, perhaps this is a bug in Progress.Windows.UserControl:Dispose()?) seems to fix most of our problems, assuming all classes have the "components" IContainer added to ComponentsCollection.

We did not need to add any explicit DELETE OBJECT statements, for now...

On the other hand we found something very alarming: apparently the client log setting influences when the destructor of our forms and child controls/components fires.

With client log on (at least 4GLTrace level basic) everything works as expected.

With client log turned off, we do NOT get the destructor of our forms and child controls (we get it at the close of the session).

This means that no forms/controls get cleaned up completely...

I've added this finding to a technical support case to investigate our memory issues.

Posted by Lieven Cardoen on 06-Jul-2018 09:15

I work together with Lieven De Foor. Very strange that software behaves differently when client logging is ON or OFF. If we turn it on, then most of our memory leak problems are gone.

It's not an option turning logging always on just to prevent memory leaking.

Turning client logging off, forms and the controls on it are not cleaned up. Working with the Infragistics ToolbarsManager means that memory starts leaking very fast ;-). 

Lieven Cardoen

Posted by Laura Stern on 09-Jul-2018 08:50

Glad you hear you mostly solved your problem. Regarding child controls of a UserControl, I would think that the Dispose of the UserControl itself should be taking care of that.  Perhaps it does not have the Components mechanism the way a form does?  I didn't check to see how that works.  Please log a bug for this if you feel there is one.

I'm finding it very difficult to fathom how logging could be affecting this.  But if you have a reproducible case, please log a bug.    

Posted by Lieven De Foor on 10-Jul-2018 02:16

We could hardly believe it either Laura, but that's what we saw.

We're preparing some things for technical support...

Posted by jquerijero on 10-Jul-2018 13:48

[quote user="Lieven De Foor"]

We could hardly believe it either Laura, but that's what we saw.

We're preparing some things for technical support...

[/quote]

BTW, what Progress version are you running your tests?

Posted by Lieven De Foor on 11-Jul-2018 02:11

[quote user="jquerijero"]

BTW, what Progress version are you running your tests?

[/quote]
11.7.2

Posted by jquerijero on 11-Jul-2018 09:41

[quote user="Lieven De Foor"]

jquerijero

BTW, what Progress version are you running your tests?

11.7.2

[/quote]

OK, thanks. I'm evaluating 11.7.3 for our next release, and we have an ongoing support case regarding memory leak.

Posted by Lieven De Foor on 24-Jul-2018 06:21

OpenEdge bugs OCTA-7011 (destructors only called when client log is active) and OCTA-7102 (child components of UserControl not disposed when UserControl is disposed) are confirmed and under investigation...

Posted by Lieven De Foor on 06-Nov-2018 15:09

OCTA-7011 (destructors only called when client log is active) -> fixed in 11.7.3

OCTA-7102 (child components of UserControl not disposed when UserControl is disposed) -> fixed in 11.7.4

This thread is closed