Orphan Remote Clients cannot be disconnected resulting in bi growth

When a Remote Server encounters an error disconnecting users that have terminated their session, the Remote Server exits
The Remote Server exists leaving user entries in memory which are not cleaned up, resulting in bi growth and no means to disconnect the remote client/server connection.

A Remote Server fails to undo a transaction of one of its clients, then successfully disconnects from the database.
An orphan remote client session with an long open transaction cannot be disconnected when the remote server restarts locking the bi cluster chain for as long as the database remains running.

02:59:01.983] P-6620924 F SRV   335: (3728)  SYSTEM ERROR: lklocky: forgot to wait, recid 678125826
02:59:01.986] P-6620924 I SRV   335: (2252)  Begin transaction backout. 
02:59:01.987] P-6620924 F SRV 335: (815) SYSTEM ERROR: Error in undo of record delete 
02:59:02.005] P-6620924 I SRV 335: (739) Logout usernum <20>, userid <>, on tty batch
02:59:02.005] P-6620924 I SRV 335: (2520) Stopped.

• The remote server (335) has freed the slot in connection table for usernum <20> and its own slot 335 but the slot of usernum <20> is not released from shared memory. 
• The Remote Server should not terminate (2520) when there is still a remote client of a server that has not been successfully disconnected by the server
• usernum <20>, remains in the user control table, despite the 739 Logout message. 
• The logout message (739) is not correct: PROMON still shows usernum <20> days later. 

This remote server (SRV 335) starts again (as expected), which knows nothing about the orphan usernum <20>
When remote server (SRV 335) is restarted, it does allow attempts to disconnect usernum <20> but it cannot be disconnected

02:59:04.972] P-20187122 | SRV   335: (5646)  Started on port 22532 using TCP IPV4 address 0.0.0.0, pid 20187122. 
16:04:49.328] P-18876396 | PROMON533: (-----) User <20> disconnect initiated
16:04:49.341] P-20119592 | BROKER  0: (-----) Notifying srvr 335, conn 335, to terminate remote user <20>
16:04:49.341] P-20119592 | BROKER  0: (-----) Sending signal 5 to user <20>

• The remote session <20> with an active transaction remains in connection table until db shutdown. 
• The orphan remote session with no server response results in BI file growth due to the open transaction

When a problem causes a remote server to terminate, the server will normally also disconnect all clients it's serving. If for example the remote server runs into an issue backing out the transaction for that client.

The server does not handle this situation properly and instead finishes disconnecting itself from the database. This leaves a remote client user still connected in memory, but there is no longer a server control in memory for its server.

The watchdog process was never designed to handled this situation, relying on the remote server which is still attached to shared-memory to properly handle it's connected remote clients before disconnecting itself.

The watchdog process never considered this case previously. The watchdog is designed to check for "dead" connections in two ways:

1. It scans the list of remote Servers looking for a dead server that's still connected.
2. It scans the list of users looking for a dead self-service user that's still connected.

Fixed version(s): 11.7.6.0, OpenEdge 12.2

Shut the database down at the next earliest convenience.

The longer the database is left running, the longer bi recovery time will take. For OpenEdge 11.6.1 or later databases when the database is next started, first truncate the bi file with the -crStatus and -crTXDisplay parameters before re-starting multi-user. For further information refer to Article:

•  How to estimate how long BI Recovery will take to complete

Progress Articles:

 Remote Server fails to undo a transaction twice 3728 815 leaving an orphaned remote client in shared memory  
 USR latch remains after user logs out preventing shutdown