
Christian Bryan (Brytive LTD) asked a question.
High System CPU and Interrupts on Azure VM running Openedge DB's
Looking for a bit of help, we have been having some issue with our Production server on AZURE which runs:
- Progress DB's
- PASOE
- Code and Spools
- Batch Queues
Its running with a very high CPU and high number of system interrupts and I can’t get to the bottom of why no one process seems to be taking up the CPU, the memory is okay.
We have tried upgrading the server to:
Standard E20as v4
vCPUs 20
RAM 160 GiB
But he only seems to eat up all the CPU during the day, once everyone logs off its back to normal.
We have about 1000 users logging now at peak.
It runs every layer of infrastructure:
The database, the application UI logic (via PASOE), and the application batches
Have you thought about transitioning to client-server and using multi-layer architecture? There have been recent enhancements by Progress to modernize in this way, including "multi-threaded servers" and "server-side join".
The problems you describe are hard to avoid in this configuration. For better management, UI and batches and RDBMS should each be running on their own servers. When that happens CPU problems are incredibly EASY to isolate.
I don't think you are alone in building your OE environment this way. Eg. we have built ours the exact same way. ;)
Can you share what specific processes are consuming the most CPU? That is the easiest way to guess about the cause.
_mproapsv
_progres
_sqlsrv2
_mtprosrv
etc
Hi David
The issue is that no single openedge process is consuming the CPU
There are high system interrupts which seem to only occur under user load once all the users log off for the day the system interrupts disappear.
We can't see any issues with drivers or excessive file access etc. So we are thinking that the DB is maybe creating the system interrupts?
See example snapshot:
Is that coreServiceShell for antivirus?
That sort of crap needs to be heavily configured. It needs to be kept away from database directories, log files, r-code files, and so on.
Consider using standard PaaS services like Azure SQL or Azure AppService. Would these platforms be consuming over 20 percent of the CPU doing security checks on data and code? Of course not. Azure customers would be very frustrated if they learned that 20% of the overhead costs of using Azure SQL were being spent to pay for Trend Micro Antivirus.
Obviously I'm not saying the antivirus software is not needed at all. But it needs to cause a LOT fewer problems than the ones it solves.
I would bet money that if you turned off the antivirus for a day, your system would run a lot different than it does now. Do you have a pre-prod environment to explore these theories?
Another thing on windows, is that you may be having trouble with default OS-level security configuration.
I remember having to turn off a feature of windows called DEP:
Data Execution Prevention . More info:
https://community.progress.com/s/article/P119884
The Linux folks will always say that openedge runs better on Linux than windows. Normally they either don't know the reason for that, or they don't want to admit that it is because the OS runs without so much security-related overhead, and sometimes without any antivirus at all. If you want OE to run as well on Windows and it does on Linux, then you have to start stripping back some of the overhead that you get by default on Windows.
I think Openedge 12 is DEP compliant now so i don't think it's that but certainly you raise some good points. We have MS dialing in tomorrow to take a look so hopefully they spot something.
High CPU with many system interrupts usually indicates I/O or network overhead; with 1000 PASOE/OpenEdge users, check disk latency and network drivers for bottlenecks.
Check your I/O limits. I wouldn't be surprised to find you're actually hitting the I/O threshold for your database disk and that's causing the issues. We've been through that pain on AWS recently. A combination of moving the disks to higher throughput and also increasing the -B significantly has put us on the right track.
BTW, this is where proactive monitoring of the database using tools like ProTop will save you a lot of time and effort.
We are also seeing a high throughput on our NIC, there is only one and its handling up to 300Mbps