I discovered some new things about the way that PASOE uses memory and I thought it would be helpful to others.  The product consumes most of its memory via the "msagents" that are named _mproapsv.  Depending on your workload, these can legitimately consume a ton of memory.  However there may be times when the memory usage seems unreasonable. 

Take the following example (click to open image):

This is a PASOE instance on Windows. Notice that the process with PID #3780 is ~2.3 GB!  However I can examine the list of ABL sessions and see that there is only one session and the session memory that is listed is only 340 KB.

So the question is why the OEE "instance control" console is saying that the process is using 2.3 GB in its working set???

Based on some preliminary investigation, this 2.3 GB is almost certainly a high water mark of some kind.  It is the point of time when the process had used the most memory to satisfy the needs of all active ABL sessions.  To be more specific, the memory seems to consist almost entirely of r-code that was used by all of the ABL sessions at a certain moment in time.  R-code is quite *large*, and some sessions can retrieve quite a lot of it from disk.  The r-code isn't shared across sessions either, from what I understand, so if multiple sessions are running the same heavy workload of programs, there will probably be lots of r-code duplication in memory.

The good news is that the memory growth will (probably) level out at some point and the existing memory allocations will (probably) be re-used again.  The allocations aren't released to the OS, but they seem to be made available to new ABL sessions that are created in the future after others have been trimmed away.

Another important fact that I've discovered is that these memory allocations are not required to remain in physical memory (ie. "working set").  If there is memory pressure from other processes, then much of the msagent will swap itself to the page file!  This was a big relief....  It doesn't take that many 2GB processes before you run out of all your memory!

In the next image I show what happened in my testing when I applied memory pressure on the server.  Remember that the PID #3780 was consuming 2.3 GB of "working set"!  I ran a program that simply consumed 25 GB of the server (out of 32 GB).  This applied pressure on all other processes, especially msagents.

That is what you see here. (click to open)

After applying pressure, the agent process with PID #3780 released more than half of the physical memory that it had been consuming.  You can see that the "working set" in the task manager above is now only half of the "commit size".  The memory was swapped to the page file.

If I switch back over to OEE again, the "Memory Use" column for this agent has now dropped as well (see that it is showing up as ~730 MB below)

Finally, if I then click on that PID then I get the additional details that you see below.

A couple things are important to point out.  First of all the "Memory Use" column in OEE corresponds to "working set", not the "commit size".  The "commit size" obviously remains the same - except for the fact that half of it is now living on disk in the page file.  It would be super nice if the OEE console would show both the physical memory use (as it does today) and the "commit size" which is the amount of memory that the process would consume if everything was swapped back into memory again. 

Another thing that would be super nice is if the OEE console would tell us how much extraneous memory the agent has allocated but is not using at the moment.  Given that the agent has its own pool of allocated memory, and it doesn't quickly release the memory back to the operating system, it should be able to tell us how much of that pool is unused at any given moment.  This information would be extremely helpful when distinguishing between a real PASOE memory leak, and an overly aggressive allocation of memory for use at a later time (ie. not a real leak)

If the agent could actually tell us how much allocated memory is not in use then, on that same note, it should also allow us a way (perhaps via oemanager/REST) to explicitly release that allocated memory which is not in use.  Better yet, that should be done automatically as a part of certain other operations, like trimming idle sessions.

A final thing that would be nice is if the "virtual size" was removed from the last screenshot above.  The virtual memory size of a Windows process is normally a much, much larger number than either the "working set" or the "commit size".  I cannot imagine what the OEE console could be referring to (.. and 0.2 Kbytes is just not worth mentioning).  I would guess that is probably not a meaningful number.  Perhaps on a Linux server this would result in a more meaningful calculation that would actually display the virtual size of the process.  Even so, the virtual memory size on the Windows platform is not normally helpful or meaningful.  The "commit size" (or "private bytes") would be much more valuable to us, together with the "working set".

I hope this is clear and is helpful to others who observe surprisingly high amounts of memory usage by PASOE agents.  The reason for that high memory usage is probably due to the combination of factors: (1) that PASOE is pooling memory for a later time, and (2) when PASOE does use memory, it uses a LOT of it - since the r-code is quite large and it is probably duplicated across multiple ABL sessions.  The good news is that this memory appears to be easily swapped out to disk as needed. And when the time comes for that to happen, the few extra gigabytes that get sent to the swap-file will probably not be missed and probably won't even cause very much thrashing (as was the case with my 2.5 GB msagent process).