Classic WebSpeed Agents are intermittently stuck in the "CONNECTING" State when the CGIIP messenger crashes

Classic WebSpeed Agents are intermittently stuck in the "CONNECTING" State, when the CGIIP messenger crashes while freeing memory before the WSM can retry the connection after the first connection fails due to BUSY agents.

May 20, 2021
Knowledge

Title

URL Name

Classic-WebSpeed-messenger-is-failing-to-retry-when-webspeed-agents-are-busy-errors-502-2-are-returned-on-the-web-browser-side

Article Number

000201080

Environment

Product: OpenEdge Version: 11.7.00 to 11.7.09 OS: All supported platforms Other: WebSpeed, Apache, IIS, cgiip

Question/Problem Description

Classic WebSpeed Agents are intermittently stuck in the CONNECTING State when the CGIIP messenger crashes
WSM crashes when the Webspeed Agent is busy and the messenger retries sending the request to the server.
WebSpeed messenger fails to retry when Agents are BUSY, errors 502.2 are returned on the web browser side

“No available servers” is reported in the Broker log files confirmed by extended logging in the msgr where a 0 size Port Response Buffer is past back.
The Messenger is not attempting to connect to the Agent when the port number is successfully returned on the second Port call
The Messenger does not make the connection after receiving a second port number from the RETRY and the Agent hangs in the CONNECTING state until it times out where it becomes AVAILABLE again

Steps to Reproduce

In a production environment this issue is not reproducible at will. In a test environment, to reproduce the error 502.2: 1. Create %WRKDIR%\noWait.htm ------------ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> <HTML> <HEAD> <META NAME="wsoptions" CONTENT="no-output"> <TITLE>Test</TITLE> </HEAD> <BODY BGCOLOR="33FF6E"> <CENTER> SpeedScript test <SCRIPT LANGUAGE="SpeedScript"> {&OUT} "no waiting! ". </SCRIPT> </BODY> </HTML> ------------ 2. Create %WRKDIR%\wait30sec.htm: ------------ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> <HTML> <HEAD> <META NAME="wsoptions" CONTENT="no-output"> <TITLE>Test</TITLE> </HEAD> <BODY BGCOLOR="#FF9F33"> <CENTER> SpeedScript test <SCRIPT LANGUAGE="SpeedScript"> {&OUT} "waiting 30 seconds! ". PAUSE 30 NO-MESSAGE. </SCRIPT> </BODY> </HTML> ------------ 3. Compile the noWait.htm & wait30sec.htm files: http://localhost/scripts/cgiip.exe/WService=wsbroker1/workshop Tools: -> File Tools In 'Files' select these 2 .htm files and click on the Puzzle icon to compile 4. Configure the following properties in ubroker.properties: [UBroker.WS.wsbroker1] initialSrvrInstance=1 maxSrvrInstance=1 [WSA.WSA1] logEntryTypes=WSADefault,MsgDebug loggingLevel=4 Enable enhanced logging in the messenger's ubroker.properties: [WebSpeed.Messengers] logEntryTypes=BrkrMsgr:4,UBNet:4,AgntMsgr:2,BrkrConDisc:2, MsgrTrace:2 loggingLevel=3 5. Restart the webspeed instance: $ wtbman -i wsbroker1 -stop $ wtbman -i wsbroker1 -start 6. create and run a .bat script to run the code examples: # Firefox Browser: start "C:\Program Files (x86)\Mozilla Firefox\firefox.exe" http://localhost/scripts/cgiip.exe/WService=wsbroker1/wait30sec.r timeout /t 2 /nobreak >nul start "C:\Program Files (x86)\Mozilla Firefox\firefox.exe" http://localhost/scripts/cgiip.exe/WService=wsbroker1/noWait.r # Internet Explorer Browser: call "C:\Program Files\Internet Explorer\iexplore.exe" http://localhost/scripts/cgiip.exe/WService=wsbroker1/wait30sec.r timeout /t 2 /nobreak >nul call "C:\Program Files\Internet Explorer\iexplore.exe" http://localhost/scripts/cgiip.exe/WService=wsbroker1/noWait.r 7. http://localhost/scripts/cgiip.exe/WService=wsbroker1/noWait.r will return a 502.2 error in the web browser

Clarifying Information

The classic webspeed connection flow is described in Article: What does technically mean that a Webspeed agent is in connecting state

The 'connecting' state denotes the time between when the Agent is reserved by the Broker to the time the Messenger actually connects to it.
Messenger logging shows that the webspeed messenger does not try to connect to a webspeed agent when a webspeed agent is BUSY.

While WebSpeed Agents are stuck in CONNECTING State the WebSpeed application needs to be restarted and continues processing without issue until the next time
This issue is intermittent and not reproducible
Reducing connectingTimeout (10 secs) to make the Agent become available quicker appears to alleviate the situation for longer periods
Increasing the number of agents (maxSrvrInstance) does improves the situation but not when application requests are high

Error Message

The following error 502.2 is displayed in the web browser: ------------ HTTP Error 502.2 - Bad Gateway The specified CGI application misbehaved by not returning a complete set of HTTP headers. The headers it did return are "". Most likely causes: • The CGI process was shut down or terminated unexpectedly before it finished processing the request. • The CGI process has a flaw and does not return a complete set of HTTP headers. Things you can try: • Check the event logs on the system to see whether the CGI process is shutting down unexpectedly. • Troubleshoot the CGI application to determine why it is not sending a complete set of HTTP headers. Detailed Error Information: Module CgiModule Requested URL http://localhost:80/scripts/cgiip.exe/WService=wsbroker1/noWait.r Notification ExecuteRequestHandler Physical Path C:\inetpub\Scripts\cgiip.exe\WService=wsbrokerl\noWait.r Handler CGI-exe Loqon Method Anonymous Error Code 0xc0000374 Loqon user Anonymous More Information: This error occurs when the CGI process handling the request exits before it finishes sending the response to IIS. View more information >> On Linux the following messages are seen in the system httpd log: ------------ [client <IP ADDRESS>:59707] End of script output before headers: wspd_cgi.sh [client <IP ADDRESS>:59707] free(): double free detected in tcache 2: /var/www/cgi-bin/wspd_cgi.sh [client <IP ADDRESS>:59707] /var/www/cgi-bin/wspd_cgi.sh: line 73: 1494625 Aborted (core dumped) $DLC/bin/cgiip -i wsbroker1 -f ./ubroker.properties: /var/www/cgi-bin/wspd_liv.sh

Defect Number

OCTA-34684

Enhancement Number

Cause

The cgiip[.exe] messenger crashes when retrying to connect to an available webspeed agent when the first connection fails because the webspeed agent is busy.

This issue is due to the WebSpeed messenger crashing when the server agent is BUSY and messenger retries sending a request to the server.

The messenger made the first request for available agent port and failed as the agent was in a BUSY state at the time serving other client request, which is expected.
In the second RETRY attempt, the messenger crashes when freeing memory of an variable.
A socket exception occurs while reading the message from the cgiip client.
The Agent remains in CONNECTING state “waiting for a connection that will never happen” and eventually ends the request after the configured connectingTimeout expires
And after the timeout (default 60 seconds) the Agent state changes to AVAILABLE

When this situation occurs on a busy system, all Agents become stuck in a CONNECTING state.
This causes more requests to queue on the Broker, which eventually time out “the first attempt” where a new request may just get an agent assigned on the first request.
If they get an Agent assigned on the retry, they will never attach to it where that Agent is then stuck in a CONNECTING state and so the cycle continues.
Until all the requests arriving at the Broker slows down and eventually fail.

Resolution

Fixed version(s): OpenEdge 11.7.10

Workaround

Notes

Progress Articles:

How to configure IIS on Windows for WebSpeed?
How to enable/disable the WSMAdmin tool for WebSpeed Messenger?
How to enable enhanced logging for the WebSpeed Messenger in OpenEge 10 & 11 ?

Keyword Phrase

Last Modified Date

10/25/2023 1:34 PM

Classic WebSpeed Agents are intermittently stuck in the "CONNECTING" State when the CGIIP messenger crashes

Files(0)

Was this article helpful?

Was this article helpful?

Classic WebSpeed Agents are intermittently stuck in the "CONNECTING" State when the CGIIP messenger crashes

Information

Related Articles