PERFORMANCE OF CONCURRENT READ-XML CALLS ON PASOE SLOWER

Posted by Blake Stanford on 12-Apr-2019 14:27

I recently came across the below article.

Article URL: https://knowledgebase.progress.com/articles/Article/performance-of-concurrent-read-xml-calls-on-pasoe-slower

Our system has a significant amount of code that uses the READ-XML and in many cases we are constrained by our customers to use XML, so the workaround suggested by Progress on this issue is, in my opion, not acceptable.  Either we modify a significant portion of our code (unreasonable), regress to limiting threads per multi-session agent (unacceptable, why be multi-threaded) or stay on classic appserver (impossible with v12).  

Was an enhancement request was put in and/or does Progress plan to address this issue in ways other than the suggested workarounds (i.e. fix it correctly)?  

All Replies

Posted by dbeavon on 12-Apr-2019 17:29

That is an interesting KB.  Thanks for pointing it out.  Its odd that parsing XML would have a concurrency problem.  I guess you need to be careful when picking your third-party libraries (... says the person who pulls dependencies from very questionable nuget packages ;)

I like the part that describes what's wrong (ie. the "XML/ICU transcoder").  It sounds like they stole that from the star-trek technobabble generator: http://www.technobabble.biz

How *many* concurrent XML requests are you making before you are impacted?  We do quite a bit of work with XML and I had not noticed this (or at least, not as much as some of our other bottlenecks).  Does it cause a problem when reading XML from under five ABL sessions per process?  

The reason I'm asking is because it seems to me that limiting the number of threads per msagent is reasonable .  (Or at least I'd say the default configuration was EXTREMELY unrealistic... I believe they were originally allowing !200! concurrent ABL sessions per msagent process.)  If it isn't the third-party XML library, then you might still end up hitting a different type of in-process bottleneck.  For example we were having trouble with concurrent use of the CLR bridge, and they can't blame any third-party for that (... there are no bugs in Microsoft's .Net framework!  ;-)

Another workaround that comes to mind is that you could use your ABL code in PASOE to launch the "_progres -b" for the sake of the XML alone.  You'd pass it the xml via the temp directory or something like that  Then you would have the entire process all to yourself while reading the XML.  It sounds like the XML part of your workload is a big enough thing that it may warrant an out-of-process solution.   Ie. the overhead of launching independent processes may be small relative to how long it takes to process the XML. (I don't know if there are legal/licensing consequences for starting a _progres session from a PASOE server.  That was a question I had posted a few weeks , but I think it might be better to ask a Progress salesrep .)

Posted by Blake Stanford on 12-Apr-2019 18:38

The benefits of using PASOE are the resource savings and performance through threading in the multi-session agent instead of having multiple single threaded agents.  We have a large number of transactions being processed from sources like EDI, Imports and interfaces with other systems.  If we reduce the threads to say 5 per agent, it is still highly probable that we could have all 5 threads running READ-XML.  If we can run 20, 30, 40, 50 threads per agent without performance problems, unless using READ-XML, then we shouldn't have to reduce it just because we use READ-XML.  It seems to me that this is a bug, as it is opposes the very benefits the multi-session agent is supposed to provide.  I'm sorry Progress chose poorly with their 3rd party software, but we shouldn't settle for a work around because of it, especially if we lose the benefits of moving to PASOE.

Posted by dbeavon on 12-Apr-2019 22:37

I definitely agree with you.  Did you discover the article after experiencing the performance problem first-hand?   What symptoms do you see?  A bunch of sessions that are stuck at the same point in the call stack for long periods of time?  How many sessions?  I'd like to be on the lookout for this before we get bitten ourselves.

>> ... highly probable that we could have all 5 threads running READ-XML ..

Yes, but still not nearly as bad as having 100 or 200 threads lining up for a chance to use that READ-XML method.  We have already limited the number of ABL sessions (per _mproapsv) to 25 and maybe that is why we haven't noticed this particular bottleneck (yet).  We have more _mproapsv processes running, but it seems like a fairly reasonable trade-off ... and any redundant memory usage is a fairly minor issue for us, relatively speaking.

>> It seems to me that this is a bug ...

I agree.  The problem with performance-related bugs is how to prioritize them.   I think they want us to request an "enhancement to the product" for the purpose of voting.  I would definitely vote for this, but I wouldn't want the work to come in front of some other bug fixes that I'm waiting to see in OE 11.7.5.  (Things that are causing programs to fail altogether).

Workarounds are par for the course.  XML functionality has always been limited in ABL.  Eg. we still can't do basic things with XML such as an xpath search.  We are very accustomed to hopping outside of our ABL code for tons of other critical functionality (sending SMTP emails, making LDAP queries, interacting with messaging brokers, and so on).  This is not altogether a new problem.

Do you run PASOE on linux or windows?  Here is another possible workaround in the KB if you are running PASOE on windows.

knowledgebase.progress.com/.../000041926

That KB article has a sample of how to use the CLR bridge and process your XML with the System.Xml namespace.  Technically this is hopping outside of ABL as well, but not all the way out of the _mproapsv process.  I've found that the CLR bridge takes a minimal amount of time to be initialized within a PASOE ABL session, and, afterwards, it is much faster to use these namespaces than any comparable ABL code.  If you are on linux this won't be an option (then again, perhaps linux will get a .Net core bridge one day).

Posted by onnodehaan on 14-Apr-2019 15:35

For me personally I can't understand why PSC won't fix these sorts of problems. Of not recognize the hurt it is doing to some ISV's.

The answer states "It was determined that this is expected behavior for the third party software involved, and would require significant research to potentially find a replacement product which works better in a multi-threaded model.".

That is very true of course. It is the expected behaviour for the third party software, but NOT for PSC end-users, customers, ISV's. OpenEdge is increasingly becoming a back-end languague in my opinion, processing data from website's and apps. XML is still very widely used and that will remain so for the forseeable future. So I would thing that if PSC wants OpenEdge to stay relevant it should not settle for "it was determined..."-like answers, but should take full responsibility in taking away bottlenecks like this.

And if that's something that has a lower prio for now, fine... but in that case, they should at the the very minimum recognize the pain that these kind of problems cause us and make sure that it gets the attention it needs somehow.

Posted by frank.meulblok on 15-Apr-2019 08:01

Let's just put the "This is expected behavior of a third-party component" reasoning on shaky ground as well.

OpenEdge uses the Xerces parser. That has publically documented limitations, but also offers ways arounds those. See:

xerces.apache.org/.../faq-parse-3.html

That is a bit long-wided in saying the parser is not designed to be reentrant, so that's why you end up queueing if you share an instance between threads. With the solution being to have each thread use it's own instance of the parser.

Question remains how easy it is for Progress to implement that fix.

Posted by dbeavon on 15-Apr-2019 13:53

@frank.meulblok - That is a much better explanation than the tech-babble on the Progress KB.

Here is the key - if you *share* a parser across multiple threads, then they will have to take turns.

But when each thread has its *own* instance of the parser then the "the instances can be used concurrently, without external synchronization".

I would guess that PASOE, being multi-threaded, is not using that third-party library correctly.  (It is used in a way that is valid when the AVM is only hosting ABL code in a single-threaded way.)  It sounds like the Xerces library is perfectly capable of being used in "isolation" on multiple threads.  But from an OpenEdge point of view, that was never a requirement, nor a consideration, until PASOE came along.  Remember that the other runtime host processes (_proapsv, _progres, etc) were all single-threaded, and PASOE is the first example one that is multi-threaded.

There are other technical issues with third-party software that arise based on the fact that PASOE hosts multiple threads.  Another example that comes to mind is the fact that "STATIC" members in ABL are different from one thread (ABL session) to the next.  But when you use the .Net CLR bridge to access the "static" members in .Net, those will all reference the *same* underlying members within the *same* appdomain.  You have to be careful about understanding the difference in what "static" means in the context of each of these runtimes.  Fortunately most static members of .Net are thread-safe ... but that doesn't mean there won't be some concurrency control that requires threads to be temporarily blocked.  (The blocking will may appear like .Net is just running slow, until there is some closer examination.)

The question about the isolation of ABL sessions brings a question to mind.  Does each ABL session represent a separate "AVM"?  Ie does PASOE host multiple "AVM's" or a single "AVM" with multiple sessions?  Does the term AVM even get used in the context of PASOE?  The docs use it in the context of the single-threaded hosts but not for PASOE: documentation.progress.com/.../index.html

Posted by gus bjorklund on 15-Apr-2019 20:14

a single avm with multiple sessions or an avm per session amount to the same thing.

the agent process has one copy of the avm code (actually one copy shared amongst all the agent processes), some agent specific private data, a copy of the session-specific data for each (configurable number) of the sessions, several utility threads, and some different (configurable) number of worker threads that take an incoming request, bind to relevant session's data, perform the request, and send the reply back. each request for a given session may be handled by a different worker thread. worker threads sleep when there is nothing for them to do.

each set of session data has lots of stuff that is not shared with other sessions, such as database connections, lbi and dbi and other kinds of files and so on.

Posted by dbeavon on 15-Apr-2019 21:31

By that definition, are you saying that multiple distinct _progres processes share a single AVM?   Your second paragraph didn't use this TLA enough.  It wasn't clear where you draw a boundary between one AVM and another.

I suppose I wasn't very clear myself, for that matter.  When I think of a JVM, it is an executing process that contains shared resources.  See the following link...

www.javaworld.com/.../what-is-the-jvm-introducing-the-java-virtual-machine.html

...   "When developers talk about the JVM, we usually mean the process running on a machine ..."

In my mind a JVM (or AVM), it is defined by an execution context with resources that are being shared.  For example an AVM would  create only *one* instance of a shared singleton; and it will only run a static constructor *once*.  If there are static constructors being run on *every* ABL session, then it seems to me that each of them should be considered *separate* AVM.

>> the agent process has one copy of the avm code

I'm assuming you are talking about the $DLC installation, rather than custom r-code.  I'm quite certain that each ABL session within PASOE has its own copy of ABL r-code.  Consider this scenario for example, you use the "-q" startup parameter and you deploy new r-code, and trim a subset of sessions (inactive ones).  That would allow the new sessions to pick up different r-code files - since changes were made to r-code in the underlying PROPATH.

Posted by Blake Stanford on 16-Apr-2019 14:02

We are just starting to move forward with CASOE to PASOE converstion.  We are in the process of Application and load testing, trying to find the right balance between threads, agents, connections, etc...  

I'll be happy to share when we have more definitive results on the performance impact of the READ-XML issue.

We run 99.8% of our appservers on Linux so .Net is not an option.

Posted by gus bjorklund on 16-Apr-2019 14:04

> On Apr 15, 2019, at 5:32 PM, dbeavon wrote:

>

> are you saying that multiple distinct _progres processes share a single AVM?

sorry i was not clear. i will try again. without using the word "avm" (it has no actual definition).

i mean that multiple _progres processes share the same memory-resident executable, hence they share the same /code/ (and static data (text strings, constants, etc.)). each process has its own process-private data and also shared data (like database shared memory, shared procedure libraries, etc.)

inside the agent process, there are some number of threads that each have their own execution context. some of these threads are for running 4GL code and others are for other (internal) purposes. all of the threads in the process can see all of the data in the process but each has its own execution context.

some of the process-private data in the agent process is the data associated with 4GL sessions (datbase connections, transactions, r-code, persistent procedures, 4GL call stack, 4GL created objects, 4GL variables, etc.). each 4GL session has a set of these things, all tied tegether. other process-private data is for the 4GL interpreter's use.

each worker thread has an instance of the 4GL interpreter code and some related data. there is one copy of the 4GL interpreter code in the process (obtained from the executable, mapped to shared code segments by the operating system) and multiple copies of the (process-private) interpreter's data (one for each worker thread). when a worker thread gets a request, it finds the (process-private) session data associated with the target of the request and binds to it. it then collects the parameters that were sent and runs the 4GL r-code that was requested (i.e. called remotely by the client). when the request has been processed to completion, final results are sent back to the client and the worker thread unbinds from the session. it can then work on the next request which may be for the same or another session.

one more point: code that is executed by the worker threads (or other threads) must be written specially so that it can work independently of all other threads. the 4GL interpreter has been extensively modified to be able to do this. to accomplish such a feat, one has to change many internal data structures and global variables so that there is a separate copy for each thread or arrange for a single copy to be used exclusively by one thread at a time (by locking and unlocking the data structure somehow).

that is at the heart of the XML question that started this thread. the agent process apparently uses a single Xerces XML instance for all the XML work done on behalf of all the 4GL sessions. therefore when an XML operation is taking place, some internal lock prevents other threads from interfering with the handling of the in-process XML operation(s). this is not a Xerces bug. it is a 4GL agent limitation. it could be remedied without much effort, but that is easy for me to say because i dont have to do it.

hope that is clearer.

Posted by Blake Stanford on 16-May-2019 20:03

Thanks for all of the explanation, however, I would really like to know if Progress is going to fix the issue or leave us with the work around described in the KB article

Article URL: knowledgebase.progress.com/.../performance-of-concurrent-read-xml-calls-on-pasoe-slower

Posted by David Cleary on 17-May-2019 14:37

We are looking at building the Xerces parser with an alternative converter that doesn't require a lock when being called. If this is promising, we will need to figure out how we will deliver it without affecting applications that may be parsing an encoding that isn't supported by the new converter.

This thread is closed