The role of the Broker in a Stateless/State-free AppServer, is to manage the execution of remote procedure calls by client applications on AVM sessions running on Agent processes.
Remote procedure calls are serialized by the client into a series of request messages, which are sent from the client through the Broker to an Agent process where they are deserialized and executed.
Once the procedure finishes executing, the results are serialized by the Agent (aka server) into a series of response messages and are sent back through the Broker to the Client.
The Broker manages the interaction between the client and Agent processes using several threads.
These threads interact with each other to transfer the messages from the client to the Agent, and back again.
A thread is created for each client that connects to the Broker (client threads) and for each Agent that it manages (server threads).
Each of these threads has a message queue where it can receive messages.
The client and server threads use the message queue in a producer/consumer relationship to move messages from one thread to the other.
By default, these message queues are
unbounded. That is, there is no limit to the number of messages that may be enqueued to a given queue (beyond availability of JVM memory). This allows the queues to grow and shrink in size based on the relative execution speed of the producer and consumer threads.
This scheme works well if the producer and consumer threads normally operate at the same rate, or if the total size of the request/response is relatively small (by some definition of “small”) and enough JVM memory is available:
Using
jConsole from the Java that the AdminServer\Broker are using can be used to monitor the memory used in the heap.
However, if the producer thread enqueues messages faster than the consumer thread can dequeue them, the queue can potentially grow large (assuming the request is very large). If the code hasn't changed, then this is related to the source data-load this instance was processing. When a single request generates a large number of messages as the response, this implies a large amount of data being sent as output like a request that results in
nSent increasing by 20000 could consume up to 20000 * 32k = 64000k = 625Mb of memory at once.
While the Java heap space for AppServer can be increased, if for some reason the transaction scope (or a recursive looping happens) eventually it will still Java OOM anyway. It's not a fix. When the appserver exhausts the heap, the root cause is in the code. The consequence is downtime because that single agent can bring down the whole CAS via the Broker.
The amount of data transmitted from the client can be reduced by using the
-mc option which causes the AppServer (producer) to compress the messages transmitted to the client (consumer). For further information refer to Article:
The amount of data transmitted from the producer (AppServer agent) can be throttled using a complimentary solution to guard against Java heap exhaustion, by specifying the maximum size of the queue. This is configured in the ubroker.properties file with the
queuelimit parameter and defaults to a value of 0 which is
unlimited.
When the
queuelimit property is set to a number higher than 0, it causes the producer (aka the AppServer server) thread to block once the maximum number of messages has been added to the queue, thereby allowing the consumer (aka client process) to dequeue the messages. As messages are removed from the queue by the consumer thread, the producer thread again able to enqueue messages until the limit is reached and the process repeats as needed.
Every environment / application is different, the size of table contents is different, the number of consumers is different, the network hardware or topology is different. All these factors play a part in the flow of data from producer to consumer there is no one-size-fits-all value that can be stated on how the
queuelimit value should be set.
- Setting the queuelimit too high may allow the queue to build up and therefore cause the out of memory to occur.
- Performance may be affected when using queuelimit but exact amounts can vary. It's a tuning parameter, where nSent metrics (messages sent by the agent to the broker) where each message can be up to 32k in size, can be used. A "queuelimit" set to [ nSent X 2 ] is not a bad place to start.
- There's also queuelimit logging entry (since 11.6.3)
brkrLogEntryTypes=Ubroker.QueueLimit:2,UBroker.Basic:2 and set queueLimit=1024 for example
Set the jvm property name in jvmargs for the BROKER: psc.oe.qlpct.red (logs at 90% of queueLimit)
Then use jConsole to keep close tabs on the memory used in the heap.
- It is advised to use the queuelimit parameter with OpenEdge 11.3.3 Service Pack, 11.4.0 or later. For further information refer to Article:
