Does anyone understand the "sessions" entities dis

Posted by dbeavon on 16-Nov-2018 23:49

Hi, We are preparing to use PASOE in production for the first time next week.  It has been a bit of a learning curve coming from "classic" appserver.  But after a year of playing around with PASOE in our pre-production environments, I think I'm finally comfortable moving ahead.

For administration and troubleshooting we will rely heavily on both the tomcat ("manager") webapp, and the OEE console.

Within the OEE console, the screen I use the most is the "agents" circled in blue below.  It is extremely helpful.  But a lot of the time I try to avoid the OEE "sessions" screen (circled in red) because it often seems to contain inaccurate or misleading data.   There may be multiple factors (specific to our usage patterns) that cause this data to appear the way it does.  Our appserver activity primarily consists of SESSION_FREE connectivity over APSV, and we have HTTP sessions enabled.  Perhaps the screen isn't designed for that combination of factors.

Below is a screenshot showing an entire table of misleading or incorrect data.  First I should say that the "Session ID" (red arrow) is SOMETIMES meaningful and will sometimes correspond to a tomcat HTTP session that can be found in the tomcat-manager.  But if I am NOT able to find the "Session ID" on the tomcat-manager screens then I basically assume that the OEE "sessions" have become confused and probably has a bunch of orphaned records in it's internal memory that it is unable to rid itself of.

...

I should note some other details that cause me to doubt the the table of misinformation that is shown above.  I can go to the tomcat-manager page and invalidate/expire HTTP sessions at any time.  When I do this, the related "Session IDs" in the screen above will be cleared away.  But the doubtful stuff is the cruft that is left over and can't be explained.  Similarly, if the "Session ID" in the table above is valid (and corresponds to a legitimate session that is listed in the tomcat-manager) then the "stop" button will work properly and I can stop that sesson.  But where a cruft session is concerned (one that isn't represented in tomcat-manager) the "stop" button doesn't do anything.  I can click it repeatedly for that type of session and nothing will happen.

Finally I should mention that if I use the tomcat-manager to invalidate/expire all HTTP sessions, and if I also kill all the MS-Agent processes for the ABL application, then the cruft sessions should be totally orphaned without anything connected to them on any side.  Yet they continue to remain visible in the OEE "sessions" list.  The only way to clear them out once-and-for-all is to stop the entire PASOE instance and restart.

Fortunately for me, I have not found that many reasons for using this "sessions" administration screen in the OEE console.  But because it is the very first menu item in the ABL application window, it seems to me that it should work a little better than it does.  I'd like to continue avoiding it, but at the same time it seems so prominent that it is hard to ignore.  It is hard to explain to other PASOE users that they should avoid it, and they should instead rely on the tomcat-manager session information.

If anyone has a good understanding of this screen, or why there would be items in this table that have no relationship to tomcat sessions, then I would greatly appreciate some insight into what is going on here.  The contents of the screen will grow until we eventually restart PASOE.  Something ain't right.  And while I'm asking, what is the "session pool ID"  that is shown in this table?  I haven't found any google hits for that at all.  I'm wondering if that might indicate that we may be using PASOE in a non-standard way that might be creating unusual problems.  I am open to any and all suggestions.  Thanks in advance.

Posted by dbeavon on 18-Dec-2018 03:34

I logged a support case and the final result was a KB article.  000093392

The KB article explains that the sessions are not specific to any particular ABL application.

The PASOE session page under OEM "does NOT distinguish which app the session is trying to access."

See...

knowledgebase.progress.com/.../pasoe-session-page-under-oem-is-showing-all-oeabl-sessions

I think it is silly to have a "sessions" link within a *particular* ABL application if the link is going to display *all* the sessions across the entire tomcat instance.  A better approach for monitoring sessions is to use the tomcat "manager" webapp instead.  I would ignore the misleading information in OEE/OEM and just use the corresponding details from tomcat itself.  That information is a lot more reliable and authoritative.

All Replies

Posted by dbeavon on 07-Dec-2018 22:47

Another thing I've noticed about this "sessions" console in OEE is that, even though the link to it is available within the context of an ABL Application, the sessions that are listed in the screen aren't exclusive to any particular ABL Application.  The same list of sessions seems to be displayed within the "sessions" screen for EVERY ABL application.

So the "sessions" seem to be top-level session-manager entities.  They do not seem to be associated with any particular ABL application.  In the very least, this appears to be a bad user interface design.  These sessions aren't attributed to a particular ABL application and they probably shouldn't be listed in a way that implies something different.  Also, if the information is overly technical/internal (as it appears to be), then it should be hidden or summarized, and expanded only when absolutely necessary.  Today it is presented very prominently in the very first link that is shown within an ABL application.

Here is another new link to a KB about the "sessions" console.  I suspect another PASOE customer was confused about the details in the "sessions" console as well:

knowledgebase.progress.com/.../sessionmanager-reports-session-state-as-reserved

I may need to open a support case because the list of these unexplained "sessions" continues to grow.  

The only way I know to reset them is to bounce the whole PASOE instance.

Posted by rkumar on 10-Dec-2018 14:16

Hi David,

If you execute the following oemanager REST API, what information do you get?

http://<localhost>:8810/oemanager/applications/<appName>/sessions

Please change localhost and appName as appropriate.

Rohit.

Posted by dbeavon on 10-Dec-2018 16:03

Hi Rohit,

Thanks for the response.

The REST results for path you pointed me to are below.  The sessions appear to be very similar to the confusing list of sessions that is shown in the OEE console (OEE>ABL-APP>sessions).   

Just as before, the list of results are the same, no matter which of our ABL applications are used for the <appName> component.  In other words, these sessions don't seem to be directly related to any particular ABL application.  Ideally they would be a top-level link in the OEE user interface (or not listed at all) instead of giving us the false impression that they are related to one of our ABL applications in PASOE.

{
    "result": {
        "OEABLSession": [
            {
                "adapterType": "APSV",
                "bound": false,
                "clientConnInfo": null,
                "sessionID": "2FAAAC6F8B8A035179AB44A3AE269289A94FCDF9BAA7.oepas1",
                "sessionPoolID": "gQRcPF13SE2VpQAokaWlDQ",
                "sessionType": "SESSION_FREE",
                "agentID": "",
                "elapsedTimeMs": 6279249,
                "agentConnInfo": null,
                "lastAccessStr": "2018-12-10T08:50:53.030-0500",
                "ablSessionID": "",
                "requestID": "",
                "requestState": "READY",
                "sessionState": "AVAILABLE"
            },
            {
                "adapterType": "APSV",
                "bound": false,
                "clientConnInfo": null,
                "sessionID": "7587F4AF8EA5705F9E5308EDDF8AA2002F5CCF341757.oepas1",
                "sessionPoolID": "gQRcPF13SE2VpQAokaWlDQ",
                "sessionType": "SESSION_FREE",
                "agentID": "",
                "elapsedTimeMs": 1313302,
                "agentConnInfo": null,
                "lastAccessStr": "2018-12-10T10:13:38.977-0500",
                "ablSessionID": "",
                "requestID": "",
                "requestState": "READY",
                "sessionState": "AVAILABLE"
            },
            {
                "adapterType": "APSV",
                "bound": false,
                "clientConnInfo": null,
                "sessionID": "72AFA729EDABBBBCD847CD9F60BB95D42EBB9ECD4161.oepas1",
                "sessionPoolID": "gQRcPF13SE2VpQAokaWlDQ",
                "sessionType": "SESSION_FREE",
                "agentID": "",
                "elapsedTimeMs": 5311384,
                "agentConnInfo": null,
                "lastAccessStr": "2018-12-10T09:07:00.895-0500",
                "ablSessionID": "",
                "requestID": "",
                "requestState": "READY",
                "sessionState": "AVAILABLE"
            },
            {
                "adapterType": "APSV",
                "bound": false,
                "clientConnInfo": null,
                "sessionID": "221A6CDE0EB255591E8FA108B4EB03F10D89401FCD76.oepas1",
                "sessionPoolID": "gQRcPF13SE2VpQAokaWlDQ",
                "sessionType": "SESSION_FREE",
                "agentID": "",
                "elapsedTimeMs": 17276576,
                "agentConnInfo": null,
                "lastAccessStr": "2018-12-10T05:47:35.703-0500",
                "ablSessionID": "",
                "requestID": "",
                "requestState": "READY",
                "sessionState": "AVAILABLE"
            },
            {
                "adapterType": "APSV",
                "bound": false,
                "clientConnInfo": null,
                "sessionID": "C6B464E7EF8A197699065379D5785619C9B9D83C932F.oepas1",
                "sessionPoolID": "gQRcPF13SE2VpQAokaWlDQ",
                "sessionType": "SESSION_FREE",
                "agentID": "",
                "elapsedTimeMs": 1042918,
                "agentConnInfo": null,
                "lastAccessStr": "2018-12-10T10:18:09.361-0500",
                "ablSessionID": "",
                "requestID": "",
                "requestState": "READY",
                "sessionState": "AVAILABLE"
            },

            ....


        ]
    },
    "outcome": "SUCCESS",
    "versionStr": "v11.7.4 ( 2018-10-10 )",
    "errmsg": "",
    "versionNo": 1,
    "operation": "GET CLIENT SESSIONS"
}

Any idea what this data is telling me?  I noticed that there are ~500 entries like the ones shown above.  About 90% of them have a value for "lastAccessStr" that has today's date.  This is a bit encouraging.  But some of them are a couple days old.

Most but not all of them are related to tomcat sessions (http://localhost:8810/manager/html/sessions ) based on a common sessionID.  

But as I mentioned earlier, I can forcefully cause the tomcat sessions to expire, and it won't necessarily remove the sessions from this list in the OEE console.

Any tips would be appreciated.  Tomcat's resident memory is growing to about 1 GB and I'm a bit suspicious of these "sessions" (assuming they are part of the session manager that is loaded within tomcat itself).  I've found no documentation to explain these details that are shown in the OEE console (and now available via REST too).

Thanks, David

Posted by Irfan on 10-Dec-2018 17:24

Hi David,

Based on the metrics, it looks like your client connections are not disconnected. I can say that as your agent information is null/empty. You can confirm that by looking at your localhost access logs. In the access log, for APSV client you should see the "2FAAAC6F8B8A035179AB44A3AE269289A94FCDF9BAA7.oepas1" ended as below(which means it is releasing the connection)

"POST /apsv?CONNHDL=A079394387185ACB613F5F00551BA39CB9BD9B02A411.oepas HTTP/1.1" 200 249 6 thd-5 -.

You can see the last column has the value "-". For the requests that are running remote procedures should have some session-id. So you can confirm based on this information in the logs.

Posted by dbeavon on 10-Dec-2018 23:09

Hi Irfan,  

Is this list of session data from the session manager, or from somewhere else? You say that the agent information is null/empty.  Is that common?  Does that always happen between session-free calls, for example?

Also, what would you expect the behavior to be if I were to expire the related session ID's from the tomcat manager (at http://localhost:8810/manager )?  Should the sessions go away (the ones that are listed within the OEE console)?

My experience is that this list of session data does not get cleared away, even after all the related session ID's are expired in the tomcat manager, and all the related MS-agent processes are ended as well.  Can you point me to any documentation about this list of session data, so I understand what it is telling me?

Thanks,

David

Posted by Irfan on 11-Dec-2018 00:48

These are the list of sessions available in session manager. Agent information is null or empty, if we just have connections and no requests running. If we have something running in the ABL session then we would see something like this

{
  "versionNo": 1,
  "outcome": "SUCCESS",
  "errmsg": "",
  "versionStr": "v12.0.0 ( 2018-12-10 )",
  "operation": "GET CLIENT SESSIONS",
  "result": {
    "OEABLSession": Array[1][
      {
        "elapsedTimeMs": 18452,
        "clientConnInfo": {
          "elapsedTimeMs": 18507,
          "executerThreadId": "thd-2",
          "reqStartTimeStr": "2018-12-10T19:45:17.572-0500",
          "requestProcedure": "server2.p",
          "clientName": "127.0.0.1",
          "requestID": "sgHO5E+PuIxJFO9skIEEEw",
          "httpSessionId": "",
          "requestUrl": "http://localhost:8080/apsv",
          "adapterType": "APSV",
          "sessionID": "1CE3CBD2F9ED87A27BDC1183FA0643F9D252128A1009.myinst24"
        },
        "ablSessionID": "",
        "lastAccessStr": "2018-12-10T19:45:17.623-0500",
        "agentConnInfo": {
          "connID": "oiUiEk3ZTiu60re8k2EQog",
          "connPoolID": "aFbj5MzDSPKp-f071VICZw",
          "agentAddr": "oelxdev11/172.16.21.106:62012",
          "agentID": "Adiobc3rTAaiu9AcGxTYGA",
          "localAddr": "/172.16.21.106:45271",
          "state": "RESERVED"
        },
        "requestID": "",
        "requestState": "RUNNING",
        "sessionPoolID": "8tjkNFCpToauGfiDmrsrkA",
        "sessionType": "SESSION_MANAGED",
        "agentID": "Adiobc3rTAaiu9AcGxTYGA",
        "sessionState": "RESERVED",
        "adapterType": "APSV",
        "bound": false,
        "sessionID": "1CE3CBD2F9ED87A27BDC1183FA0643F9D252128A1009.myinst24"
      }
    ]
  }
}

Posted by dbeavon on 11-Dec-2018 03:43

I suspect you are right that the sessions will look this way in certain common scenarios (where session-free  connections were made using http tomcat sessions, but no requests are currently underway.).

What bothers me is when I go and manually expire all the tomcat http sessions but this list of session manager sessions remains as-is.  Ie. Given that the "sessionID" property - in the session manager list - appears to be a reference to the tomcat session id, then you would think that this list of session manager sessions would immediately become empty as a result of expiring tomcat sessions.  That doesn't always seem to be the case.  I can't find a pattern.

What would the expected behavior be after I expire the related session ID's from the tomcat manager (at http://localhost:8810/manager )?  Should the sessions go away (the ones that are listed within the OEE console)?

There are still Other things are confusing to me in this oee console.. why do the same list of session manager sessions appear under every abl application?  Is this user interface a work-in-progress?

Thanks for helping me understand this better.  I am worried about the day that this sessions list jumps to thousands of entries, or the tomcat memory usage grows to multiple gb.  Hopefully by that time I will have a better grasp on what I'm looking at in the oee management console.

Posted by rkumar on 11-Dec-2018 05:51

Someone from PASOE team can answer the question on the expected behavior when one expires the session IDs from Tomcat manager.

OEM/OEE uses the Tomcat Manager API to fetch the list of sessions. I have registered the issue with OEM/OEE listing out the same sessions under every ABL app.

Posted by Matt Baker on 11-Dec-2018 14:01

As Rohit pointed out on the other forum question, this is a question for PASOE.  If you haven't already, please open a support case.  This may need a bug logged with the /oemanager web application where OEM/OEE gets its data from.

Posted by dbeavon on 18-Dec-2018 03:34

I logged a support case and the final result was a KB article.  000093392

The KB article explains that the sessions are not specific to any particular ABL application.

The PASOE session page under OEM "does NOT distinguish which app the session is trying to access."

See...

knowledgebase.progress.com/.../pasoe-session-page-under-oem-is-showing-all-oeabl-sessions

I think it is silly to have a "sessions" link within a *particular* ABL application if the link is going to display *all* the sessions across the entire tomcat instance.  A better approach for monitoring sessions is to use the tomcat "manager" webapp instead.  I would ignore the misleading information in OEE/OEM and just use the corresponding details from tomcat itself.  That information is a lot more reliable and authoritative.

Posted by Irfan on 18-Dec-2018 12:38

Hi David,

The sessions API give the sessions information for each ABL Application, but not for each OEABL WebApp. So if you have more than 1 OEABL Web App deployed for an ABL Application, then the sessions API will show you all the sessions running in a given ABL Application.

Also, if you have session hanging and if you were trying to terminate the session either with terminate session API using oemanager or by tomcat manager then it has to expire. If it did not then it is a bug.

Posted by dbeavon on 18-Dec-2018 13:32

Hi Irfan, thanks for your input.  I think you may have it backwards.  The tomcat instance knows which webapp the session is connected to, and is able to associate the HTTP session with a related *webapp*.    This information is available in the tomcat "manager" webapp.  But when the OEM console tries to enumerate the sessions based on the ABL application, then the underlying tomcat instance will not directly help to produce that list, hence the misinformation that is shown in the OEM console.  If you view the sessions, in the OEM console (within an ABL application) they are not limited to any single ABL application nor any single webapp.  Please see that KB for more explanation.

In our configuration we have only one webapp for each ABL application.  But the outer PASOE instance has multiple of these webapp/ABL-app pairs.  The sessions that are listed in the OEM console are *never* specific to a single webapp, nor a single ABL application.  The sessions are a combination of *all* the sessions across the entire tomcat instance.  You might want to set up this configuration to see for yourself.  I'd also suggest connecting from a large number of long-lived state-free client sessions, so that the problem is accentuated.

In order to manage our sessions, I'll probably continue to rely on the "manager" webapp in tomcat.

Thanks, David

This thread is closed