We've deployed PASOE to production and I've been monitoring for a couple days. We are using 11.7.4 on windows.
One thing that really concerns me is the memory usage of the _mproapsv processes. After they are freshly started, they are only about 100 MB. But once they start being used for a day, they can grow quite large, and don't seem to be under control. For example, I have one in production right now that has 1.5 GB in resident memory, which is a lot considering the whole server only has 16 GB of physical RAM and there are lots of other ABL applications on the same server.
Here is a screenshot of that agent and its RAM:
The process as a whole has been running for a couple days, and seems greedy on memory. For this reason I've tried to trim its internal ABL sessions with REST, via powershell:
Invoke-RestMethod -Method Delete -Uri ('http://localhost:' + $portnumber.ToString() + '/oemanager/applications/' + $ablapp + '/' + $agent.agentId + '/sessions') -Credential $cred
You can see in my screenshot above that the session start times for the four new sessions are from moments ago (2018-11-21T14:00:00:00Z), while the agent process has been running much longer - for two days. My hope was that by trimming the inactive sessions, the resident memory would drop back to what it was at the time an agent is freshly started (ie... 100 MB rather than 1.5 GB). But the resident memory of that process doesn't change substantially after trimming all the inactive sessions. We stay at 1.5 GB resident! Notice that the four new ABL sessions are only using about 10 MB each, so I'm not sure how to account for the resident process size of 1.5 GB.
I saved a dump file that comes out to 1.6 GB, and compresses to about 200 MB.
Is this expected behavior? Do a handful of ABL sessions (that are normally inactive) need to consume that much RAM? What is really scary is that I noticed the default config setting for ABL sessions per agent is 200! If 5 to 10 ABL sessions can cause the process to grow to 1.5 GB, I'd really hate to see what would happen with 200 ABL sessions in a single process!
I see that it can be an easy thing to use OEE to start a new agent, and then kill off the old one. But that seems like a fairly drastic operation and I wouldn't think it necessary unless there was something that was going out-of-control within the _mproapsv processes. Also, if any session within the agent is continuously busy on a long-running operation, it becomes infeasible to attempt killing the agent as a whole.
Another update, I was able to force the agent processes to shrink the private bytes ("commit size") in some simple scenarios. First I would allow a session to leak a lot of memory, then I would trim the inactive ABL sessions out of the PASOE agent. After doing this, the hosting agent process will shrink. These are short-lived agents with a very artificial and simple memory leak (basically a persistent procedure, that has a nested LONGCHAR with a lot of text in it).
In this simple scenario my agent processes go back to about an 80 MB "commit size".
In production I can do the same thing and try to trim all inactive ABL sessions out of a PASOE agent that is running at 1.5 GB, but I do *not* get all the way down to 80 MB again. The routine only gets us down to a 1 GB commit size or so. This may be either because there is a surplus of memory in production, or because there is a leak in the process (which is outside of the trimmed ABL sessions themselves).
I'm not saying this is your cause, but Mike Banks said at the PASOE workshop I attended last month that this is something they've seen quite a bit with PASOE in production environments. As far as I remember the main culprit they've found is memory leaks within the code executed on the AppServer. You don't notice the leaks in classic AppServer as the agent shuts down at the end of processing. With PASOE sharing so much more, and agents sticking around so much longer, memory leaks become an issue.
They suggested using Dynamic Object logging to look for objects that are created and not cleaned up as a starting point for finding these leaks.
This KB applies: knowledgebase.progress.com/.../P133306
Obviously bare in mind (as I'm sure you would), that some objects shouldn't be cleaned up!
It may also be a good idea to raise a case with tech support on this.
When you refer to "memory leaks within code executed on the AppServer" then I'm assuming you mean custom ABL code. (ie. not the Progess native/internal stuff).
I can trim all inactive appserver sessions (my ABL sessions) and the OEE console will no longer show them listed within that agent. What doesn't make sense is that once my sessions are gone, and fresh sessions take their place, the working set of the process doesn't drop substantially. My understanding is that performing this operation should eliminate any reason for pointing fingers at my custom ABL code, because those ABL sessions, along with related ABL memory, should not exist anymore.
Yes I'm referring to custom code. I'll have to defer to those with more knowledge of the subject as I'm just regurgitating something that came up in the workshop! Essentially it's to do with the way PASOE reuses resources that the classic appserver doesn't. As a result code that leaks memory can hang around more easily than it would in the classic appserver.
Maybe PASOE is not programmed to return freed memory back to the operating system, and therefore continues to show the highest ever memory size indefinitely, even if it is currently using less? (Just speculating. I have no idea what those figures are based on.)
It could even be that the agent already deallocated the memory, but the OS didn't fully release it yet due to the related kernel functions being as lazy as possible.
That would explain why Virtual Size is close to zero, while resident size is not.
This is assuming what's reported matches Linux convention, where Resident Set is the physical ram used and Virtual Set is the virtual memory pages requested by the process. Which may be in resident memory, mapped file or swap space.
An assumtion that's immediately challenged by the fact that Windows uses different terminology, thus confusing what's actually being reported to start with.
An update: I was able to get the resident size (working set) to go back down to 100 MB if I run another greedy program that puts pressure on the memory of the OS. This is a relief, and I suppose that most of my concerns will be mitigated by having an extremely large pagefile. But I don't think it is the whole answer. IE. If I put pressure on memory then the working set drops, however I see that the "commit size" is never decreasing. That means we will probably fill up the pagefile at some point, no matter how large it may be.
It was the writing of the process memory-dump that caused my concerns. The size of that memory-dump corresponds to the "commit size" of the process. The act of writing the memory dump will transfer *all* committed memory to a file (ie. both the resident stuff or the stuff that has been swapped to the pagefile). It is pretty clear that these memory dumps are unreasonably big, and the committed memory is never getting released (even when the system is under memory pressure, and the ABL sessions have all been trimmed from the agent).
Since it has only been a few days, I will make sure there is plenty of space in the pagefile and continue to monitor. Perhaps the "commit size" of the agent process will stop growing. We should be fine as long as it stops growing. I am still assuming that the problem is *not* with our custom code, because the "commit size" is large even after trimming all of the ABL sessions (which are responsible for hosting our custom code ).
Note that if you are connected to a database using shared memory, then the database cache and other shared data structures show up in the process virtual and resident sizes. Also any dll’s used by the process, other shared code, and shared procedure libraries.
Given that our MS-Agent processes in production continue to grow in memory usage, over an extended period, I'm opening a support case.
I don't suppose there is any way for a customer to independently investigate the contents of a memory dump?
It would be really nice to understand why it is that some of our PASOE-ABL-applications have agents that don't leak memory, and other applications leak rapidly.
(I was able to confirmed that it was a genuine leak by putting pressure on the system memory as a whole, and observing that none of the committed/private memory was ever returned to the OS. Even when all few existing sessions were trimmed, the agent did not give up any of its memory).
I wish Irfan was still monitoring these forums... He always had a lot of practical suggestions for troubleshooting PASOE... I suppose everyone at Progress is busy working on the OE 12 release.
I think I will open a support case because it is clear that there is a leak within the ms-agent process. One thing that we started doing is trimming inactive ABL sessions out of the agents every hour. My understanding is that this should absolve us of blame for leaks. Because memory that is outside the boundaries of CURRENT sessions is not reachable and not our responsibility if we can't manage it anymore.
Does that sound right? Shouldn't Progress be responsible for reclaiming all memory used by any individual ABL session that has been trimmed? Or are we responsible for making sure the ABL session memory is clean and spotless, before the session is trimmed. It seems unreasonable for an ABL programmer to be responsible for cleaning up everything, eg static members of classes, etc.
If you terminate a ABL session in an MS-Agent then the memory allocated to that session goes back to the MS-Agent. The memory goes back to the OS only when the agent is terminated.
But if you have figure out why is your session taking up so much memory then you can API's like session stacks and ABLObjectsReport to see which objects lying around and not getting cleaned-up.
I would not worry if it took up a huge memory used by application and stay there. Instead of releasing the memory to the OS and asking again every time, the agent holds the memory and use it for the next requests. Attention is required if the memory is constantly growing and causing performance impact.