PASOE - Sizing Your Machine (ATM Benchmark)

Sizing your PASOE machine to your application and load is a very important step to moving your web application to production.
This article provides a framework to successfully size your machine for your PASOE application.

The major sections covered by this Article when Sizing Your Machine for a PASOE environment are:

• Performance Tuning Basics
• Application Infrastructure
• Testing Environment
• Monitoring Tools
• Knowledge of PASOE Tuning Parameters
• The Sizing/Performance Process

The information in this Article has since been published in the OpenEdge Documentation:

Progress Application Server for OpenEdge: Machine Sizing Guide, Overview - Performance Tuning Basics
https://docs.progress.com/bundle/pas-for-oe-sizing/page/Performance-Tuning-Basics.html 
Manage Progress Application Server (PAS) for OpenEdge - Tune PAS for OpenEdge instances
https://docs.progress.com/bundle/pas-for-openedge-management/page/Tune-PAS-for-OpenEdge-instances.html

Performance Tuning Basics

There are only four bottlenecks possible in any application environment. They are CPU, Memory, Disk I/O and Network. There can be several reasons why any of these bottlenecks can occur, but finding the bottleneck and relieving it, and doing this repetitively, is the only way to properly size or tune your application.  

The point is to find the bottleneck in these four areas by looking at the symptoms, alleviate the cause, by making only ONE change at a time. Then run your test again to see how your change worked. Then find the next bottleneck.

Examples:

• High Load - number equals number of CPUs
• 0% idle time

• Not enough CPUs
• Too many CPU's, How to optimize database's shared memory access in multi-CPU environments?  
• Too much swapping (memory to disk)
• Application issue

• 0% memory available
• swap space being used

• Memory leaks, Detect PASOE memory leaks with Dynamic Objects Logging  
• Not enough memory to meet application load requirements

Application Infrastructure

1. Your client machine or test harness.
2. Your PASOE instance and application.
3. Your Database.

You can use this data later to scale your application if PASOE and the database are on the same machine. However, trying to size your PASOE application with the database running on the same machine would introduce too many variables.

Testing Environment

What is your test harness?

• Will you be running JMeter to drive your web application?
• Will you be using Load Runner?

It doesn't matter what the test harness of choice is used, but remember a few points:

1. The test machine always needs to have excess CPU, Memory and Network capabilities.
2. Make sure you add some "think time". Real Applications never have all requests running immediately after the response. For example, we ran our tests with a 1 to 3 second random think time interval.
3. The test harness must gather roundtrip (a.k.a. elapsed times) metrics
4. The test harness must gather and reports errors.

Monitoring Tools

Monitoring tools are essential for running your sizing or performance tests. They must be reliable, consistent and run in the background. The data must be archived while being run.

Monitoring tools used in our sizing tests:

• Top - linux system performance monitoring tool
• OE Manager REST API - PASOE monitoring tool
• JMeter - not only the test harness but also to monitor responses for elapsed time and error reporting

Knowledge of PASOE Tuning Parameters

There are many parameters available for tuning PASOE, you only need to know a few for most tuning situations, at least for baseline initial runs.

• Run this to get the current value -> tcman config psc.as.executor.maxthreads
• Run this to change the value -> tcman config psc.as.executor.maxthreads=<new value>

• maxAgents=10

• maxConnectionsPerAgent=16

• maxABLSessionsPerAgent=200

• numInitialAgents=1

1. The Tomcat Maximum Simultaneous Connections should be okay at 300.
2. Set the maxAgents=1 (we want to run all the tests with 1 Multi-Session-Agent (_mproapsv) running multiple Sessions).
3. Set the maxConnectionsPerAgent=50 (this should be more than needed for the baseline).

1. Start your database on a machine big enough to handle the requests.
2. Start the PASOE instance with the values above and connecting to the database at startup.
3. Start scripts to gather system information (top) and PASOE information (OEManager REST API).  Example scripts are provided in the Appendix to this Article.
4. Start your test harness running a small concurrent user count (example: 10).
5. Verify best performance.
6. Save all information.
7. Increase concurrent user load, and maxConnectionsPerAgent, incrementally.
   1. When you get to 300 concurrent clients increase the Tomcat maxthread values.
   2. Keep maxthreads and maxConnectionsPerAgent values the same and keep 1 maxAgent.
8. Increase until roundtrip time starts to fall.
9. Analyze your bottleneck.
10. Correct bottleneck (if CPU 100% or Memory 100% increase machine size).
11. Repeat until you are satisfied.

The following Article provides an example of hardware sizing for PASOE to handle up to 500 concurrent client clients load using the ATM Benchmark and JMeter:

• PASOE - Example of PASOE Machine sizes for ATM Benchmark on Amazon AWS  

Progress Articles:   

• How to optimize database's shared memory access in multi-CPU environments?  
• Where to find more about PASOE HealthScanner?  
• Tuning PASOE OpenEdge Database Connections  
• How to enable Swagger for PASOE instance   
• How to monitor PASOE instance via Tomcat Manager?  
• How to monitor Progress AppServer Tomcat with JConsole  
• What is jMeter?  
• Is there a configuration in PASOE that queues requests without a timeout error when all ABL sessions are busy?  
• 12.1 to 12.2 configuration migration  

APPENDIX

Monitoring with UNIX/Linux TOP command

top - 16:25:13 up 159 days, 14:33, 21 users,  load average: 1.01, 1.02, 1.00
Tasks: 374 total,   1 running, 373 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  1.6%sy,  0.0%ni, 96.0%id,  0.0%wa,  0.0%hi,  1.6%si,  0.0%st
Mem:   8045544k total,  7529984k used,   515560k free,   321100k buffers
Swap:  6495228k total,  3919288k used,  2575940k free,  4740468k cached
 
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 34031 bob       20   0 59356 1912  484 D  8.6  0.0   2872:19 _mproapsv
 45296 roy       20   0 19532 1608 1044 R  0.7  0.0   0:00.04 top
  1762 root      20   0  209m 7320 1212 S  0.3  0.1 109:30.56 vmtoolsd
  2554 qpidd     20   0  481m 1168  788 S  0.3  0.0  11:19.23 qpidd
 49314 roy       20   0 98.2m 2676 1516 S  0.3  0.0   0:02.95 sshd
 62073 root      20   0 4783m 732m  19m S  0.3  9.3  48:45.89 java
 90709 bob       20   0 4486m 293m  18m S  0.3  3.7   0:35.71 java
117103 bette     20   0  206m 3920  932 S  0.3  0.0  80:07.48 vmtoolsd
123863 root      20   0 4134m 478m  21m S  0.3  6.1   3:17.90 java

 

CPU(s) %us %sys %id %wait	User percentage of all CPUs System percentaage of all CPUs Idle percentage of all CPUs Wait percentage of all CPUs	Most time should be _mproapsv Too much here? Swapping? 0% means 100% load Time waiting? Swapping?
Memory	Total and used
Swap	Memory swapped to disk	Always want 0 memory in swap
COMMAND _mproapsv java	Should be using the most CPU Should be in the top few (#2)	Remember this is 100% times # of CPUs