11.7.4 PAS related process takes all CPU with leaking idle session threads

PAS related process takes all CPU with leaking idle session threads
PASOE instance generates a large amount of "idleSessionWatchdog" threads. 
Even when there is no client activity on the PASOE instance, the number of idle session threads was growing. 

When PASOE is a Windows Service, the ‘WmiPrvSE.exe’ process eventually shows high CPU usage.

• It’s not possible to stop the PAS using tcman stop -F, it never ends, 
• The PASOE Tomcat/Java process process has to be killed from the Task Manager.

A jstack thread dump of the PASOE Tomcat/Java process with jstack -l JAVA_PID > jstack.out shows the idleSessionWatchdog thread count growing

The following settings in openedge.properties have no impact on idle sessions cleanup: Idle sessions are still not cleaned up when the idleSessionTimeout property is set to zero so that Tomcat uses the value of the session-timeout property configured in web.xml, which defaults to 30 minutes.
 

The creation of the idleSessionWatchDog threads is a consequence of changes in OpenEdge 11.7.4 to make the idleSessionTimeout property work for Java OpenClient transports.
 
In prior versions, Idle sessions created by PASOE transports that use the embedded Java OpenClient are not automatically shut down by the idle resource manager.  

• These sessions always appear in the "RESERVED" state because Idle sessions are created by the Java OC session pool manager
• These sessions cannot be terminated by session manager, which only shuts down AVAILABLE sessions
• In order to reap sessions that were owned by the internal java openclient interface, the java openclient idle session watchdog for PASOE was enabled with a default timeout value of 15 seconds. 
• This allows the cleanup of unused sessions owned by the Java OpenClient based transports.
• The idle session watchdog threads are shut down when the OpenClient session pool is destroyed, as a consequence of successful Java OpenClient appobject destruction. 

However this change exposed leaking OC session pools, in the REST adapter.

This issue was replicated by forcing the agent to run out of database connections.

• Handling the errors in the REST adapter caused the Java OpenClient objects to be recycled. 
• When this happens, the idle connection watchdog thread in the Java OpenClient session pool is not stopped when the session pool was released, causing the buildup of watchdog threads. 
• Because the watchdog thread maintained a reference to the session pool object, this object was also not garbage collected.

Upgrade to OpenEdge 11.7.5. 
Upgrade to OpenEdge 12.1 or later.

The fix implemented the following:

• Clean up the watchdog thread when the session pool object is released
• Not recycle the OC appobject in the REST Adapter when handling Agent errors.
• Remove SessMgr sessions from the session pool when an error occurs.
• Use the multi-user form of the OC Connection object when instantiating the OERealm SPAClient

Until the fix is available, use the workaround below.

Set -DPROGRESS.Session.idleConnectionTimeout=-1
The value being set is “-1” (minus one)

WINDOWS: add the following line to the PASOE instance openedge_setenv.bat :

rem Now add those as Java system properties to JAVA_OPTS environment variable
set _oeopts=-Dpsc.as.oe.dlc="%DLC%"
set _oeopts=%_oeopts% -Dpsc.as.oe.wrkdir="%WRKDIR%"
set _oeopts=%_oeopts% -Dlogback.ContextSelector="JNDI"
set _oeopts=%_oeopts% -Dlogback.configurationFile="file:///%CATALINA_BASE%/conf/logging.xml"
set _oeopts=%_oeopts% -DPROGRESS.Session.NetworkBufferSize="60000"
rem new line to add:
set _oeopts=%_oeopts% -DPROGRESS.Session.idleConnectionTimeout=-1

UNIX: add the following line to the PASOE instance openedge_setenv.sh :

_oeopts="${_oeopts} -DPROGRESS.Session.idleConnectionTimeout=-1"

This line should be inserted before the line:

JAVA_OPTS="${JAVA_OPTS} ${_oeopts}" ; export JAVA_OPTS

The instance must be restarted in order for this change to take effect.