When a user connects to a Progress database in multi-user mode, the user control table in shared memory is updated to add the user. The watchdog process periodically checks to make sure that all users listed in the shared memory user control table have corresponding OS processes running. When the WatchDog detects that a user process is listed in the Progress user control table but does not have an active OS process, the shared memory user/remote server will be disconnected from the Progress database and the following message examples will be written to the log file:
BROKER 0: Disconnecting dead user 41 (2527)
BROKER 0: BROKER detects death of server 6464 (1153)
WDOG 12: Disconnecting client 55 of dead server 1 (2526)
However, if the watchdog process detects that shared memory or buffer latches have been left in an inconsistent state by this "dead process", by design the
Broker will then shut the database down to protect the integrity of the database. This abnormal termination of the process that still holds latches (locks) in shared memory, will appear prior to the 2249 message:
SYSTEM ERROR: User 41 died during microtransaction. (2256)
SYSTEM ERROR: Releasing regular latch. latchId:<latch-num> (5028)
User 41 died holding <num> shared memory locks. (2522)
User 41 died with <num> buffers locked. (2523)
User 41 died with <num> buffers locked. (5027)
The database manager will initialise an emergency shutdown with the following message written to the log file, followed by all users being logged out and the server shutting down:
BROKER 0: Begin ABNORMAL shutdown code 2 (2249)
The code number in the (2249) message above is for future use and currently has no meaning. This
error in itself does not indicate database corruption, merely that measures have been taken by the broker to prevent potential corruption. When re-starting the database, crash recovery takes place and the database will be ready for normal use if no errors are encountered during bi crash recovery. Crash recovery may take some time, especially if during the Physical and Logical undo phases there are uncommitted transactions to roll back. Refer to Article
In addition to the above errors, some of the following messages may be written to the .lg file by the WDOG, if enabled or the Primary login BROKER if not, as the abnormal shutdown proceeds and logs out all the users:
SYSTEM ERROR: redundant lwake user <n> latch <x>
SYSTEM ERROR: bkrlsbuf: cannot release buffer lock, use count 0 is invalid. (1051)
These errors in themselves do not indicate database corruption.
Events that could cause a process to terminate abnormally include:
For those sites where this occurs frequently,
one way to prevent an abnormal shutdown is to run all users as "remote" clients (i.e. not shared memory, but TCP/IP) even if they are logging in directly from the machine where the database is. In this way, the user process does not access shared memory directly. They connect to a database Remote Server process which accesses shared memory instead. The Server process obtains shared memory and buffer latches for the user process and when the user process abnormally terminates, the Server process usually remains running to clean up remaining latches held for that user's session.
When users dying holding shared memory or buffer latch (lock) occurs repeatedly, it is important to determine the root cause of the user process dying and to resolve that problem.
