Client process (batch or interactive) crashing after being blocked on record lock when dbnotification is enabled

Client process (batch or interactive) crashing after being blocked on record lock when dbnotification is enabled
Crash or hang after interactive or batch client waits for a record lock to be granted.

Known symptoms:
1. After upgrading to OpenEdge 12.2.2 a large number of crashes on FIND statements when a record lock is requested and another user attempts to lock the same record.

The C Stack from _progres reads:

bmLocateBuffer2
cxLocateValidBlock
cxSearchTree
cxPositionToNext
cxNextPrevious
cxRowidNext
dbCursorFindAndGet

There are other manifestations of the same issue where the following message is seen on the client or written to the client log file:

Failed to get database notification information from server
Disconnect from server, server received invalid message code (2661)

There are other manifestations of the same issue where the following message is seen in the database .lg file:

Message recvd while waiting: uc_wait=1, msg=136

2. Client hung while searching for record.
protrace from hung client process:

utnap
cxGetRootAndArea
cxLocateValidBlock
cxSearchTree
cxPositionToNext
cxNextPrevious
cxRowidNext
dbCursorFindAndGet
dsmCursorFindAndGet
qrIxGet
qrNext
dbqry
profnd
fdfnd
bfFindRow
rnbfnxtDoit
rnbfnxtBody
rnbfnxt
rnexec_entry
rninterpret
rnrq
main

3. PAS Instance seems to be running but client connections hang and agents show RUNNING but never move beyond.
All PAS agents getting spawned but clients aren't able to process work.
 

1. It is only possible to reproduce this while connected to multiple databases.
2. Stack trace from hung client process shows no DB connected, while there are multiple database connected on client startup.
protrace shows:

** Databases (logical/type/physical) **

** End of Protrace **

 

This issue only occurs when the -dbnotifytime / -usernotifytime database startup parameter is active. The notification occurs during the lock which causes a context switching error, when clients connect to multiple databases, resulting in the crash when the lock is achieved by the client and subsequently the context is already switched.
This defect also could cause all PASOE agents to get stuck. And new client connections cause more and more PAS agents to be spawned until all configured agents are spawned. Eventually, new client sessions will then cause the (1890) message to be reported.

This was encountered in OpenEdge 12, because in prior versions neither of these parameters were active by default which they are since OE 12.

 

Fixed version(s): OpenEdge 12.4, OpenEdge 12.2.4, OpenEdge 11.7.9

When starting the databases, explicitly disable database & client notification when clients are in an environment that connects to multiple databases.
    -dbnotifytime 0 
    -usernotifytime 0