Memory-leak? - Forum - Corticon - Progress Community

Memory-leak?

 Forum

Memory-leak?

  • I'm using the Corticon 5.5 for java on Windows-2012. It seems that during each DS-call, the memory-usage for java increases, but afterwards does not decrease. During each call about 100,000 entity-instances are created. So after about 30 DS-calls the max (16Gb) is reached. Last step of each flow is to persist the created entities via a ServiceCallOut.

    Do i need tot explicitly remove the created entities in the SCO?

    In BRMS.properties i use the setting for not returning the created entities:

    com.corticon.server.restrict.response.rulemessages.only=true

    Greetings,

    Bartel

  • Hi Bartel,

    Have you tried to install the 5.5.1 update yet? It fixes a lot of (smaller) issues with 5.5.

    -Gertjan

  • Are you saying the process grows to 16gb and then you get an out-of-memory exception?
     
    Jim
     

    James Arsenault  |  Product Development  |  Progress Software
    email: james.arsenault@progress.com |  direct: 781-280-4934

  • Hi Bartel,

    What version of the JRE are you using? Is it the standard 1.7 that came with 5.5? Did you make any changes to startServer.bat or jvm.properties?

    Gertjan

  • Process grows to almost 16gb and then my DS-call times out (com.corticon.server.execution.queue.timeout=1800000). There's no memory-exception. Using Corticon 5.4.0.3 the proces grew up to 11gb after many calls and then stabilized.

    With the help of Gertjan i installed 5.5.1-bugfix, but that did not make a difference.

    Bartel

  • I'm using java1.8. I will try using the standard 1.7 again and see what happens.

    Bartel

  • Have you tested with your SCOs disabled? If there is a memory leak in them you'll need to profile your code to identify the cause.

    Are you using EDC? Is it possible you're inadvertently running some query which is retrieving massive amounts of data?

    James Arsenault  |  Product Development  |  Progress Software
    email: james.arsenault@progress.com |  direct: 781-280-4934

  • My SCO runs several queries in order to create the required entities. One of those queries has a massive resultset (about 100000 rows). After the resultsets are processed (creating entities) the queries and database-connection are closed.  With Corticon 5.4.0.3 i did use a combination of EDC ans SCO for database-actions. Since Corticon 5.5.1. i perform the database-actions via my SCO exclusively and do not use EDC any more. As a kind of test i run many DS and after half a day CorticonServer-console showed the following log-enties :

    Exception in thread "catalina-exec-2"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "catalina-exec-2"

    Exception in thread "Timer-9" Exception in thread "Timer-10"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-9"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-10"

    Exception in thread "Timer-12"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-12"

    Exception in thread "Timer-13" Exception in thread "Timer-14"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-13"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-14"

    Exception in thread "Timer-11"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-11"

    Exception in thread "Timer-15"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-15"

    Exception in thread "Timer-16"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-16"

    Exception in thread "Timer-17"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-17"

    Exception in thread "Timer-0" Exception in thread "Timer-1"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-0"

    Exception in thread "Timer-3"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-1"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-3"

    Exception in thread "Timer-2"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-2"

    Exception in thread "Timer-8"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-8"

    Exception in thread "Timer-6" Exception in thread "Timer-7"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-6"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-7"

    Exception in thread "Timer-4" Exception in thread "Timer-5"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-4"

    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-5"

    Could it be that the thread for each DS-call does not end?

  • Any thing is possible but I don't think that's the cause. I would first look to see if your SCO is keeping a reference to the data it queries or the entities it creates in some way which persists between executions. The cumulative effect of this could be leading to the OOM error. Do you have access to, or experience with, a profiler? That would most likely point to the cause.

    James Arsenault  |  Product Development  |  Progress Software
    email: james.arsenault@progress.com |  direct: 781-280-4934

  • Hi Bartel,

    Is the use of Java 1.7 (instead of Java 1.8) causing a slower increase of used memory?

    Did you add a RuleSheet to the RuleFlow yet, that clears the created instances after the SCO has performed the SQL INSERT?

    - Gertjan

  • I already checked my SCO-class. It has no members that contains the queried data.

    Altough i have no experience i will try a profiler. I'm leaning towards VisualVM ,but perhaps you do have a special recommendation?

  • Hi Gertjan,

    Yes i tried both, but with no effect. I removed the created entities explicitly as a final step in my SCO. Total time for one DS-call increased with 2 minutes, but it did not have benefits in memory-usage.

    Bartel

  • Personally I like YourKit for java profiling.

    James Arsenault  |  Product Development  |  Progress Software
    email: james.arsenault@progress.com |  direct: 781-280-4934

  • When looking at Java memory usage you need to be careful how to interpret the process size. Excluding actually hitting an OOM condition, the process size does not definitively identify how much memory is used. It's all subject to Java garbage collection. You could have a large process size but the heap usage is low; that just means Java hasn't freed the unused heap space.

    James Arsenault  |  Product Development  |  Progress Software
    email: james.arsenault@progress.com |  direct: 781-280-4934

  • Gertjan visited me in order to fix this problem. He re-compiled all of the ruleflows leaving the EDC-settings empty. The result of this is that i'm happy to have one CorticonServer-machine that keeps on running all my decisionservices without any problem. This happens to be the only server that runs on dedicated hardware and OS is Windows7-professional. All my other servers still show the memory-problem when running the most demanding decisionservice repeatedly. Those servers are virtual (ESX-server-pool) and OS is Windows-server2012.

    So our next plan is to replace all virtual servers by dedicated hardware.

    Bartel