Hello All,
We're investigating a memory leak in our application.
Is there any utility or any setting which will report the memory leaks in the AppServer source code.
Any tips on how to tackle this kind of issues is appreciated!
Regards,
Atul Dalvi
You've probably done a quick search before but that's how Progress suggest finding memory leaks
Classic appserver or PASOE? How fast is the leak? Can you quantify it? Almost all apps leak to some extent, and what is more interesting is how fast (1 GB an hour/day/week). A "leak" can also be defined in different ways. (Eg. Peter gave the example of objects that are cached deliberately, which may appear like a leak until closer inspection. Another example is the use of classes that have static member data. Or even the runtime itself may "leak" if it accumulates a lot of r-code in memory, or chooses not to instantly release memory allocations back to the OS in favor of keeping them for some future reuse.)
If there is in fact a measurable leak, then PASOE (if that is what we are talking about) will amplify if. This is because there are lots of ABL sessions that are hosted in a single process ( _mproapsv).
This question just came up a week ago, you may want to look at this thread too:
What [mention:9e4ee96fac634b8f91b580e1fb4f7e71:e9ed411860ed4f2ba0265705b8793d05] suggested was what we did at a previous customer: we had a special program to testrun a procedure. Before the start of the program we created a fingerprint of all persistent procedures and tables currently active in your session. Then we ran the program, did an UNDO and made a second list of persistent stuff. We removed everything from the list that was already there (tip: store handle in a tt) and what remains is the stuff your program created but did not clean up.
Just a caveat when you do use the NEXT-SIBLING method: In some cases you might get a handle / object back, but when you do a VALID-HANDLE or VALID-OBJECT on it it will return FALSE. It typically happens when you've created something inside a widget-pool and destroy the widget-pool later, but nowhere there is an explicit DELETE on that handle / object. Personally I think this is a bug because that should've be cleaned up as part of the widget pool, but for now you need to ignore these invalid handles / object and do another NEXT-SIBLING (yes you can do that, even though they don't point to a valid handle / object). If you don't you'll simply stop scanning the rest of the chain.
That knowledgebase.progress.com/.../P133306 article and its associated script came in very handy for us just this Monday as we were chasing a ** stget out of storage error. Added the DynObjects.* loging and run the output through the script only to discouver some dynamic queries being created and never deleted... CREATE WIDGET-POOL is my new friend.
Hello Peter,
Thanks for the reply. Actually, we have a PAS server specially running for the OpenEdge soap web services only. We have couple of OpenEdge soap services deployed on this PAS server and hundreds of programs running and linked under it. There are no OpenEdge classes involved in it only .p programs with Dataset handles, input-output Datasets, temp-tables, Dynamic Queries, buffers etc. I have below doubts on the options you suggested -
1. Will the Dynamic Objects Logging work in this case ?
2. Can you give some more details on the option (2) to identify temp-table entries. Any other logging type for temp tables ?
3. Also how can we implement option(3). I have code for this but Its a session or request specific right ? Can't we implement this at very higher level as a common code for all the web services ? otherwise I have to modify main program of each web service to write this code.
Thanks in advance.
DynObjects tracks the following docs.progress.com/.../Dynamic-object-logging.html
The temp-table logging is described docs.progress.com/.../Temp-table-logging.html .
These 2 should give you good insight .
In PASOE you can always add activate and deactivate procedures, and those will run on every request (for SOAP, WEB, etc). That would be the place to add the checks IMO.
Thanks Peter.
Please note that if you can run the same application in 11.7.4 or later you can use the PASOE REST API to monitor for leaks. The below article talks about using the ABL http client to call out to the REST API and includes references to code that will allow you to turn the getABLObjects functionality on/off, and another piece of code that can be used to fetch a list of leaked objects.
000095195 - How to call into the OEManager's REST API for insight into PASOE?
knowledgebase.progress.com/.../How-to-call-into-the-OEManager-s-REST-API-for-insight-into-PASOE
The benefit of using this is that you don't need to change your application code or use parameters to generate a multi-million line log file that you have to then parse in order to compare the creates with the deletes to find the list of leaked objects. This utility is simply turned on/off as desired and then it does the analysis and simply returns where the leaks are in your code.
The only pain point I can think of is that you DO have to have the oemanager deployed to your pasoe instance.
Hello All,
I tried DynObjects but in log report I did not find anything related to the dynamic objects as I don't have object programming in my code only input output temp tables and datasets and temp tables & datasets are simple not with handle object.
What else I can check to find out this memory leak on AppSeerver ?
Thanks in advance !!!
> On May 15, 2019, at 5:18 AM, atuldalvi123 wrote:
>
> only input output temp tables and datasets and temp tables & datasets are simple not with handle object
based on what you have said, it is likely that you are creating tem-tables and dataasets that are not being deleted when you don't need them anymore.
if you start client with
-clientlog foo -logentrytype "temp-tables:4"
then this code (admittedly crude) might be helpful and that you can use along with client-logging:
/* this program matches up temp-table create and delete log messages.
generated by the 4gl runtime clientlog facility.
-clientlog foo.log -logentrytype "temp-tables:4"
just cat your foo.log log files as generated by the 4GL into tt_all.log.
version 6, gus bjorklund, december 15, 2014.
*/
using Progress.Database.TempTableInfo.
def var deb as logical no-undo initial false.
def var input1 as char no-undo.
def var input2 as char no-undo.
def var data_op as char no-undo.
def var data_name as char no-undo.
def var data_path as char no-undo.
def var data_line as char no-undo.
def var n as int no-undo.
def var m as int no-undo.
def var nmatches as int no-undo.
def var entries as int no-undo.
/* here we collect info about the orphans
a row is added for each create.
the row is eliminated when we see a delete */
def temp-table ttorphans no-undo
field tt_op as character
field tt_name as char
field tt_path as char
field tt_line as char
index ix1 tt_name.
/* here we collect tt name and number of creates
and deletes for each one */
def temp-table ttcounts no-undo
field theName as char
field path as char
field creates as int
field deletes as int
index ix1 theName.
/* read log entries for tt creates and deletes */
input from tt_all.log.
inloop:
repeat:
import unformatted input1.
trim (input1).
/* skip the stuff we don't want */
if (index (input1, "TEMP-TABLE")
if (index (input1, "Created")
data_op = entry (11, input1, " ").
data_name = entry (13, input1, " ").
if (index (input1, ")") > 0) then do:
input2 = substring (input1 , index (input1, ")") + 1).
data_path = entry (2, input2, " ") no-error.
data_line = entry (4, input2, " ") no-error.
if deb then
display data_op format "x(1)"
data_name format "x(25)"
data_path format "x(30)"
data_line format "x(6)"
.
end.
if (data_op <> "Created") and (data_op <> "Deleted") then next inloop.
entries = entries + 1.
if (entries mod 25000) = 0 then display entries.
if (entries > 1000000) then leave.
find ttcounts where theName = data_name no-error.
if not available ttcounts then do:
create ttcounts.
assign theName = data_name
path = data_path
creates = 0
deletes = 0
.
end.
if (data_op = "Created") then creates = creates + 1.
else deletes = deletes + 1.
find first ttorphans where (data_name = tt_name) and (tt_op <> data_op) no-error.
if available ttorphans then do:
/* have matching tt , get rid of old, drop new */
nmatches = nmatches + 2.
delete ttorphans.
next inloop.
end.
/* no match, so add a new record */
create ttorphans.
assign tt_op = data_op
tt_name = data_name
tt_path = data_path
tt_line = data_line
.
end.
message entries " records read," nmatches " matched".
output to nodelete.txt.
for each ttcounts where (creates > 0) and (deletes <> creates):
put theName format "x(20)"
(creates - deletes)
skip.
end.
output close.
n = 0.
m = 0.
output to nodelete2.txt.
for each ttorphans:
put tt_path format "x(30)"
tt_name format "x(30)"
skip.
if tt_op = "Created" then n = n + 1.
if tt_op = "Deleted" then m = m + 1.
end.
output close.
message n " creates and " m " deletes don't match".
def var tableCount as int no-undo.
def var procName as char no-undo.
def var infoHandle as handle no-undo.
repeat n = 1 to TempTableInfo:TempTableCount:
TempTableInfo:GetTableInfoByPosition (n, infoHandle, procName).
display infoHandle:name procName.
end.
You said you don’t have “object programming”, and only TT, DataSet parameters, no handles. But previously you said you have dynamic queries and buffers. Those are tracked with the DynObjects logging.
I have a support case open with Progress at the moment regarding a memory leak in the _mproapsv. I can give you more details if you need them; I think the SP will be available in 11.7.5. Your issue may not be the same as ours. It would be helpful if you gave us a lot more explanation about your problem. How fast is the leak? Can you quantify it? What process name is actually leaking (_mproapsv.exe? java.exe? tomcat.exe?)
One thing I would suggest doing is create a reproducible that you can play with *outside* of production. You can run your repro at high speed to amplify the problem. That may also allow you to isolate the portion of your code that leaks. In the end you might be left with something that you can send over to Progress tech support.
Are you running on windows or linux? If windows then I would recommend a sysinternals tool:
This tool is easy to use. And should help isolate the memory leak to a certain degree, and also quantify it. I suspect your leak will be seen as allocations in the native heap.
Another thing I would do ... especially since your question hasn't been resolved after a month ... is to approach the PASOE problem from another angle. You can install the "oemanager" API and use a REST interface to trim your inactive ABL sessions on a recurring schedule (eg. on the hour or half hour). This is a very safe operation and could be enough to resolve a wide range of so-called "leaks". (Depending on the definition of "leak".) In the very least it will give you additional information to add to everything else that you know about your "leak".
Trimming inactive ABL sessions can be accomplished via the OE manager rest interface. Here is a powershell script that shows how to trim inactive sessions from an ms-agent process:
community.progress.com/.../36461
(Posted by me on 26 Dec 2017 14:04, with lots of help from Irfan)
When an inactive session is trimmed, PASOE will clear out all the memory that is directly associated with the session. That may be half the battle. When new PASOE client requests arrive, PASOE will create fresh new sessions to replace the ones that were trimmed.
Yet another thing you may want to look at is the tomcat manager console:
httt://localhost:8810/manager/html
The console may help in your troubleshooting, and may show when HTTP sessions are open for long periods of time. I don't know about SOAP, but where APSV is concerned sometimes the HTTP sessions can become orphaned for long periods of time and hold onto resources in the MSagent, thereby causing "leaks". This can be resolved administratively in the tomcat manager console by explicitly forcing the expiration of the HTTP sessions.
Good luck isolating the leaks. As a side, I will point out that even Java and .Net code can have "leaks" when items on the heap are rooted in memory and the garbage collector is unable to clear everything out. It would be nice if PASOE had an operation that was analogous to forcing a GC.Collect() operation, which is something to help determine when there is truly a leak. The closest thing I've found to this (in the world of PASOE) is to trim inactive ABL sessions via the OE manager REST api.
Hi,
PASOE?
Have you heard or worked with swagger?
With swagger you can identify the program that is causing the memory leak as well as the line that is not deleting DATASET.
Tell me if you want to know more than I can tell you.
Regards
Hello Daniel,
No, I have not worked with swagger.
Can you give some more details on it?
And as I said I have few datasets defined in my programs but those are normal ones, no dynamic handles for datasets or output parameter handles.
How swagger will help into this?
Hi,
First sorry for my English. I'm using google translator and I hope you can understand.
I believe that with swagger you will be able to identify the program that is causing the memory leak and know which DATASET is not being removed.
I solved my memory leak problem with swagger help where the programs were adjusted in front of your report.
The following is the step-by-step guide for you to enable and use swagger
- Step 1 - Enable OEmanager
cd /usr/wrk/instance
bin/tcman.sh undeploy -u tomcat:yourpasswod oemanager - Password default - tomcat
bin/tcman.sh deploy -u tomcat:yourpassword -a oemanager /$DLC/servers/pasoe/extras/oemanager.war
vi webapps/oemanager/WEB-INF/oemgrSecurity-container.xml
//remove comments in line 35 <!-- Access to SwaggerUI. Disabled by default, user has to uncomment the below line to enable it -->
:wq
Sign in - hostname:port-instcance/manager/ - do not forget / at the end
/oemanager - Stop and Start
- Step 2 - Collect Agent ID
Sign in - hostname:port-instcance/oemanager/ - do not forget / at the end - Usr: tomcat / Pwd: your password
Click "Get / application/{appName}/agents Get Agents"
Click "Try it out"
Inform Instance name and execute
copy "agentID"
- Step 3 - Enable agent for collection
Click " PUT /aaplication/{appName}/agents/{agentID}/ABLObjects/status Enable/Disable ABLObjects and click “Try it out”
Inform instance name and agentID
Change for
{
"enable" : "true"
}
And Execute
Response Body will be changed for: "result": true
- Step 4 - Run your program or your system normally so that the collection is done
- Step 5 - Analyze collection
Click: GET / applications/{appName}/agents/{agentID}/ABLObjects Get ABLObjects Report and click in “Try it out
The screen below will show the handle used;
Do it Download
- Step 6 - Run again your program or your system normally so that the collection is done
- Step 7 - Analyze collection again
Click: GET / applications/{appName}/agents/{agentID}/ABLObjects Get ABLObjects Report and click in “Try it out
The screen below will show the handle used;
Do it Download
- Step 8 - Check the reports
If the handle of the first scan remains, it means that the program is not deleting the memory leak. It should be reviewed and corrected.
If it disappears and generates another, the program is correct without the addition of memory leak.
If the number of rows in the report increases, it means that there is memory leak and must be corrected. The handleID should always be refreshed.
With this tool it is possible to evaluate if the program is having memory leak and correct it. So the memory consumption will drop successfully.
- Step 9 - Disable agent for collection
Click " PUT /aaplication/{appName}/agents/{agentID}/ABLObjects/status Enable/Disable ABLObjects and click “Try it out”
Inform instance name and agentID
Change for
{
"enable" : "false"
}
And Execute
END Step
I hope you understand and if you have any questions, I am at your disposal.
Regards
Hello,
Where I can set this -clientlog ? My client is not an OpenEdge client, its a .net client and they call the soap services developed in OpenEdge.
We just accept inputs from them, process those inputs in OpenEdge and returns back the output in datasets or temp table.
Hello,
My client is also dotnet.
When you call the dotnet client, it calls a .p program in OpenEdge?
Does 11.6 support swagger?
I'm in version 11.7
See if you can enable it in version 11.6
Swagger is only available in 11.7.3+. If you are on 11.6, you will not find the API's to find the leaks. They are only available from 11.7.2+ onward. I think the choice for you would be enable DynamicObjects logging in the PASOE logEntryTypes and debug which objects are leaking from the agent log.
Wow
Hello All,
On the AppServer source code, we have a code structure like 1 include that contains all its temporary table definition and related dataset definitions. We use it wherever we need it and declare the input or output parameters for a dataset without a by-reference or BIND.
Will this also be the reason for the memory leak ?
e.g. Sample.p
{ttCust.i}
define output parameter dataset for dscust.
-
-
-
run CreateData.p(outputDATASET dsCust).
-
-
-
run UpdateData.p(input-output DATASET dsCust).
-
CreateData.p
{ttCust.i}
define output parameter dataset for dscust.
-
-
-
Thanks in advance.
Not sure what you're asking now. Statically defined temp-tables and datasets should not be leaking. If there is a leak there, it would be a bug in the AVM. But right now, we are not aware of such a bug. It doesn't matter how you pass them around as parameters. It is only if you create the TTs or Datasets dynamically that you would be responsible for deleting them and there could be a leak there.
There's been a lot of information provided in this thread. I think you need to follow up on one or more of the methods provided and then provide more concrete data as to what you tried and what the results were.
Yes! Just passing static dataset around should not be a / the problem ...
How do you know that you have a leak on AS? Just a feeling?
From what you are writing there may be some potential problems:
- Soap and XML. You parser may leave X-DOC, X-NODEREF ... in memory.
- Persistent procedures stay in memory...
How do you pass the datasets to your .Net frontend? C# OpenClient Proxy? Do you call "sample.p" directly from .Net or do you have some kind of "service adapter"?
To see what's going on the appserver (or even on a client) I use a simple tool.
- 1) Scan the SESSION on the Appserver and collect objects that stay in memory. Pass the data as a temp-table to the client.
- 2) Do something like calling your sample.p on the appserver
- 3) again 1)
Normally the result of 1) and 3) should show the same objects. If the object list grows then you may have a problem.
/* MemoryChecker.i/
DEFINE TEMP-TABLE ttDynObject NO-UNDO
FIELD ObjectNumber AS INTEGER LABEL "Number"
FIELD ObjectHandle AS HANDLE LABEL "Handle"
FIELD ObjectType AS CHARACTER FORMAT "x(32)" LABEL "Type"
FIELD ObjectName AS CHARACTER FORMAT "x(80)" LABEL "Name"
INDEX pix IS PRIMARY UNIQUE ObjectNumber.
/* MemoryChecker.p */
{MemoryChecker.i}
DEFINE OUTPUT PARAMETER TABLE FOR ttDynObject.
DEFINE VARIABLE hObject AS HANDLE NO-UNDO.
DEFINE VARIABLE oObject AS Progress.Lang.Object NO-UNDO.
DEFINE VARIABLE iSeq AS INTEGER NO-UNDO.
hObject = SESSION:FIRST-PROCEDURE.
RUN addWidgets(hObject).
hObject = SESSION:FIRST-DATASET.
RUN addWidgets(hObject).
hObject = SESSION:FIRST-BUFFER.
RUN addWidgets(hObject).
hObject = SESSION:FIRST-QUERY.
RUN addWidgets(hObject).
hObject = SESSION:FIRST-DATA-SOURCE.
RUN addWidgets(hObject).
hObject = SESSION:FIRST-CHILD.
RUN addWidgets(hObject).
/*
hObject = SESSION:FIRST-SERVER.
RUN addWidgets(hObject).
*/
oObject = SESSION:FIRST-OBJECT.
DO WHILE VALID-OBJECT(oObject):
CREATE ttDynObject.
ASSIGN iSeq = iSeq + 1
ttDynObject.ObjectNumber = iSeq
ttDynObject.ObjectType = oObject:GetClass():TypeName
ttDynObject.ObjectName = oObject:ToString()
oObject = oObject:NEXT-SIBLING.
END.
PROCEDURE addWidgets:
DEFINE INPUT PARAMETER phObject AS HANDLE NO-UNDO.
DO WHILE VALID-HANDLE(hObject):
CREATE ttDynObject.
ASSIGN iSeq = iSeq + 1
ttDynObject.ObjectNumber = iSeq
ttDynObject.ObjectHandle = hObject
ttDynObject.ObjectType = hObject:TYPE
ttDynObject.ObjectName = hObject:NAME.
hObject = hObject:NEXT-SIBLING.
END.
END.
/* Client prog */
{MemoryChecker.i}
RUN MemoryChecker.p ON SERVER <Appserver> (OUTPUT TABLE ttDynObject).
Then you can load the temp-table into a browse or better a .Net grid with grouping. If you display the number of records of ttDynObject it's easy to see if something goes wrong.
All of this only makes sense if the client talks to the same appserver agent for every request.
- Start the appserver with a single agent (in development).
- Make the appserver agent stateless bound for the time of checking. (Start a .p o the appserver persistent and delete it when you are done).
Good luck...
Thomas