Active client count increasing in state-free appserver (clas

Posted by dbeavon on 11-Apr-2017 08:30

This is largely a repeat of another posting from a year ago: https://community.progress.com/community_groups/openedge_development/f/19/t/24276

I think that user received about half of the answers I was looking for. 

I am using .Net open clients (Progress version 10.2B) against a state-free classic application broker, which the clients are directed to via Name server. While checking out the very high number of client connections on the server , we found out that active client number is increasing even though there are very few *agents* are actually processing requests . For example active client now is 1000 only only two agents are in use. When I checked connection to broker using asbman with -clientdetail all option, I found out that all these unused connections are having agent PID and agent port as NULL and connection state is connected. For those connections where agents are processing requests, agent port and PID is having valid value and connection state is Sending. I am not able to find out reason why these connections are not getting disconnected.

Based on the prior forum answers by "lecuyer" I wanted clarifications for some follow-up questions:

By default, is it true that a client will continue to consume client connections (will use "SESSION" terminology going forward) indefinitely (when it can't find an available one that is not in use)?  IE sessions will never time out or be closed after lack of use?  And the only way to clean up the connections is to close the client application?

  (I may be confused by the "proper" behavior, as intended by "PROGRESS.Session.connectionLifetime" 

   - in our experience we are not seeing sessions get closed).

Lecuyer says the "size of the (session) pool should roughly correspond to the maximum number of simultaneous requests from that client on that server".  IE.  It sounds like it is not straight-forward to predict how many client sessions should be used by a given application, even if the application only uses a single shared state-free connection to the broker.  Am I understanding that correctly?  It sounded like the unpredictable nature of these sessions is compounded when using persistent procedures over state-free appservers, and when using async or multi-threaded calls.

Is there any way to *disconnect* these client sessions from the broker side (as discovered via asbman -clientdetail)?  I am not seeing asbman commands for disconnecting individual state-free client connections.  Of course we can always trim agents, but that doesn't affect the connected clients.  Similarly, we can stop the entire broker, but that is disruptive to client applications and causes them to throw exceptions and die.

From  the .Net client app side is there any way for us to discover the number of client sessions being used within a process (similar to the client-side equivalent of asbman -clientdetail)?  Or how long they have been in use?

Thanks for help with these questions.  We have been successfully using state-free appserver for over a year but the client sessions are not as well-behaved as we would like them to be, especially now that we are using them more than before.

(PS. we do try to follow the common best-practices for disposing IDisposables, etc.  The only thing we know of that is deliberately held open thru-out the lifetime of the application is the State-Free Connection object).

Posted by Brian K. Maher on 14-Apr-2017 11:02

Hi David,
 
Thanks for the info.  Honestly, I think the real answer if to fix the client side code so it handles the connection pooling more efficiently.  I mentioned to you the other day when we talked that you may want to consider capping the max connections in the client app and using the WaitIfBusy property so that if a thread requests a connection to run something and all are busy that the thread will block.  You may want to give that some thought.
 
As far as I am aware, there is no way to disconnect the connections from the broker side and doing so would wreak havoc on the connection pools in each client application and causing you more problem because the pool would give our a connection that it doesn’t know has been broken and the app would try to make a call on it and end up throwing an exception.
 
Brian

All Replies

Posted by dbeavon on 11-Apr-2017 14:35

Is there any documentation for the public API of the ".Net  Open Client"?  In particular I'm looking for SessionPool methods (public) so that I can troubleshoot the types of issues I had described above.  As near as I can tell, there is no way on the client-side to enumerate the State-Free sessions associated with a state-free/session-free appserver connection.  I need to enumerate them to examine for resource leaks (ie bugs in either our custom code or in the Progress open client itself.)

I have been able to poke around by trial-and-error and found that I could loop thru existing sessions like so:

        

   for (int LoopCount = 0;

LoopCount < StateFreeConnection.GetSessionPool().availableSessions();

LoopCount++)

           {

               MySessionsReserved.Add( StateFreeConnection.GetSessionPool().reserveSession());

           }

 

But the troubling thing is that, while the reserveSession() method seems to give me the details about one of the sessions, it also manipulates the SessionPool at the same time, thereby changing state.  I don't want to change anything, just iterate and observe the contents of the SessionPool.  No other public method seems to return Session objects from the SessionPool.  Any help would be appreciated.

...Another thing that is troubling to me is that there appears to be a bug in the .Net open client related to the ConnectionLifetime property.  I have heard it said that by default the open client is supposed to release sessions after five minutes and free up resources.  But what I'm actually observing is that the default behavior is to never release resources.

Posted by Brian K. Maher on 11-Apr-2017 21:30

David,
 
By default .NET Open Client, when used in session-free mode, has its connection pool set to grab a new connection if none are available in the pool and to not release them.
 
You can change this by setting several properties.
 
It is also important to ensure that you are creating connections properly.  For session-free you should instantiate and configure only one Connection object then you need to pass that one object to each AppObjects constructor (this ensures there is only one connection pool).  Failure to do this will create multiple pools, even if you instantiate new Connection objects that have the exact same settings).
 
Brian

Posted by Peter Judge on 11-Apr-2017 21:31

There is this doc documentation.progress.com/.../using-the-open-client-.net-openapi-to-directly-a.html but I don’t know if that has the level of detail you’re after.
 

Posted by dbeavon on 12-Apr-2017 07:39

Brian,  Thanks for the feedback.  What you described is exactly what I'm seeing.  The number of sessions in the pool for the connection is increasing to about 5-10, based on various hard-to-identify factors (timing issues, agent availability, using persistent procs, and rare sections of the app that need concurrent sessions).  Thereafter the size of the session pool remains steady and doesn't seem to grow forever.  It was very hard to troubleshoot this stuff, since the API that digs into the session pool is predominantly private/protected (not public).

The session pool grows bigger than we would have expected.  And it would be better if the sessions would somehow clean themselves up over time, and disconnect from the broker.  The vast majority of our app is single-threaded and only needs a single available session.    If you multiply 5-10 greedy state-free client connections to the broker by 500 .Net users, it becomes a resource problem on the server.

As I mentioned in my original post, we are careful to ensure that the application only uses a single shared state-free connection to the broker, and for other components like app-objects and proc-objects we follow common best-practices for disposing IDisposables.

I think I'd like to submit a bug about this and get a KB since the default behavior is not correct- at least not according to the documentation that I've found.  I suspect the default behavior on the .Net side is probably not behaving as well as an ABL or Java client either, or I probably would have seen other people complaining about similar issues on the broker side.  Here is what I found.

.NET Open Clients

https://documentation.progress.com/output/ua/OpenEdge_latest/index.html#page/dvnet/progress.session.connectionlifetime.html

PROGRESS.Session.ConnectionLifetime - Session model:Session-free only

The maximum number of seconds that a given connection can be used before it is destroyed. Connections whose lifetime exceeds the specified value are destroyed as they become available...

Default: 300

After a day of experimentation, I was able to adjust ConnectionLifetime to work the way it should have worked by default.

Is there a deliberate reason that the .Net side doesn't release sessions in pool by default?  It sounds like you have access to additional information that I was unable to locate.  (ie. information that contradicts the documentation.)

Thanks again, David

Posted by dbeavon on 13-Apr-2017 08:23

Is there any way to *disconnect* the state-free client sessions from the broker side (as discovered via asbman -clientdetail)?  I haven't found a way to disconnect specific sessions and free up the memory.  Such a thing would be useful if there were individual/misbehaving client sessions.  Obviously we can stop the entire state-free broker but that is extremely disruptive to *all* client applications that are using the broker (rather than just disrupting a single misbehaving client session).

I haven't found anything like this yet.  It seems to be a pretty significant oversight.  IMHO, any kind of client/server technology should allow administration on the server side to manage (disconnect) the misbehaving clients.

(In theory the state-free clients are "session-free" and, taking that terminology literally, you wouldn't think these clients should consume long-lasting connection-related resources on the server.  But we've found that, in the case of classic appserver, there are "broker" connection resources which are persistent - and for very long periods of time.)

I may take a peek at OE PAS (apache tomcat) and see if that technology has features to manage its connected clients, whether session-free or not.  

Posted by Brian K. Maher on 13-Apr-2017 23:30

David,
 
Didn’t the ReleaseConnections() method we talked about do that for you?
 
Brian

Posted by dbeavon on 14-Apr-2017 07:37

Yes, it did work, based on my preliminary testing.  But even then, it only manages connection resources for the *one* client application (and it is only possible to perform this operation on the .Net  *client* side within the process itself.)

But if you have a state-free broker running on a server with numerous client application instances connected to it (a hundred) and multiple client connections from each application (session-pool sessions) then the amount of resources consumed by all this will grow quite large.  I was asking if there were some way to administratively free up resources from the server-side of things (eg. disconnect *all* clients tied to a single user name or all that are from a single remote I.P. address).

The only recourse an administrator has is to stop and restart the entire broker, but that will be disruptive to *all* one hundred of the remote applications (some of which are more important than others).    The reason I brought up OE PAS technology is because I suspect tomcat has a way to administer its remote sessions at a finer level of control than to just perform a full restart.

Another quick thought:  I suspect that one possible solution for us may be to set up *several* state-free brokers that cater to different populations of remote clients (ie. one broker per application type).  Then a "low-priority" broker could be restarted without impacting high-priority applications, because those would be connected to a different broker.

Posted by Brian K. Maher on 14-Apr-2017 09:26

Hi David,
 
Can you explain why the broker having a high number of connections is a problem?  What about it is negatively impacting the business?
 
Brian

Posted by dbeavon on 14-Apr-2017 10:38

It is primarily all about memory and cpu activity (garbage collection?) of the Java broker process for the state-free appserver.   Our state-free broker uses over 1 GB of resident memory on HP-UX and 3 GB of virtual memory.  This consists of over 1000 client connections from about 24 different remote I.P. addresses, and probably around 200-300 separate remote processes (.Net ).

Also, there is a max client connections number (sum of all client apps * sessions used by those apps) which we have run into on a couple occasions (most recently at the 1000 mark). When no more client connections can be made, we are forced to double this max client connections property in the ubroker.config and have to restart the broker, thereby dumping all client connections in the middle of the work day.

Posted by Brian K. Maher on 14-Apr-2017 11:02

Hi David,
 
Thanks for the info.  Honestly, I think the real answer if to fix the client side code so it handles the connection pooling more efficiently.  I mentioned to you the other day when we talked that you may want to consider capping the max connections in the client app and using the WaitIfBusy property so that if a thread requests a connection to run something and all are busy that the thread will block.  You may want to give that some thought.
 
As far as I am aware, there is no way to disconnect the connections from the broker side and doing so would wreak havoc on the connection pools in each client application and causing you more problem because the pool would give our a connection that it doesn’t know has been broken and the app would try to make a call on it and end up throwing an exception.
 
Brian

Posted by dbeavon on 17-Apr-2017 07:14

I understand that disconnecting from the server side is not preferable, but disconnecting a *subset* of clients is better than being forced to eventually stop and reset the *entire* broker.  These are both server-side administration operations, and both would cause a similar type of problem in the connection pools of the related client applications.  I was simply saying that it would be better if the broker allowed an administrator to manage a smaller subset of client connections that are behaving improperly (ie. because of their greedy connection pools, or infinite connection lifetimes, or whatever).  

Thanks for the feedback on our state-free connectivity issues.

Posted by Brian K. Maher on 17-Apr-2017 07:20

David,
 
Please submit an enhancement request to the Ideas section of Communities.
 
Brian

Posted by dbeavon on 17-Apr-2017 08:31

Sounds good. I will first be investigating the corresponding behavior on the OE PAS side of things so I can speak in a more intelligible way.  I suspect most of these concerns are not new ideas, only new to classic appserver.

This thread is closed