What does the Database Watchdog do

What does the Database Watchdog do?
What is the database watchdog's function?
How does the Database Watchdog know when a shared memory user has died?
Why should the database WDOG be started?

When a database is started multi-user, the Database Watchdog runs all the time:

• The Watchdog is a background process for shared memory. Its purpose is to protect the database from corruption.
• The Watchdog process wakes up approximately every 10 seconds and performs its function.
• If the WDOG is not started as a separate process, the BROKER handles all the Watchdog functions.
• After the database has started the WDOG can be started from the command line with the Watchdog startup script: PROWDOG, or set to startup in conmgr.properties:  watchdogprocess=true
• Benefits to starting the WatchDog as a separate process:
  • It frees up the database broker from watchdog functions, to do the multitude of other things the Broker does
  • It can check for the disappearance of the database broker process itself - there is nothing else that can do this function.

What does the Database Watchdog do?

Watchdog process works in two (2) areas to protect the database and to resolve conflicts:
A.  Shared Memory

In Shared Memory the Watchdog looks for missing shared-memory processes. The watchdog is designed to check for "dead" connections in two ways:

1. It scans the list of Remote Servers looking for a dead server that's still connected where the pid is gone.
2. It scans the list of users looking for a dead self-service user that's still connected where the pid is gone.
3. In OpenEdge 11.7.6, 11.2 watchdog process checks will be enhanced to also consider an additional case where a remote user may not have an associated remote server in the server list with matching pid/server number, and the process for the user's server PID is gone to clean up after orphan remote clients.

When a system process has disappeared from the system, the Watchdog senses the missing process by:

1. Checking the user table in shared memory and compares it to system processes.
2. It marks the user table and goes back to sleep.
3. If the user table shows the process missing 3 times (or 60 to 90 seconds) watchdog initiates removing the user slot from the _usrctrl table in shared memory:

• The Watchdog backs out all transactions for the missing process by assuming that process's userid
• The Watchdog releases all locks held by the missing process from shared memory.
• The Watchdog finally removes the userid from the _usrctrl table in shared memory

The watchdog process never terminates any shared-memory process. It's responsibility is to cleanup after a process that was directly connected to shared-memory has terminated. This is further discussed in Article:

•  How the Database Watchdog Works?  

If the process was killed by a signal that cannot be trapped (kill -9/ endtask) and that process still had a lock on shared memory, the watchdog will bring down the database broker to preserve database integrity. Refer to the following Article and related article links:

•  What causes DBDOWN errors 3719 5029 or 3718 5028  

B. The database .lk file

Simply for historical interest, in older mid-range computers like VMS, the dbname.lk file did not exist all locking is handled by the VMS Operating System.

While all processes use the dbname.lk file to determine if the database is currently in use, the watchdog uses the 'dbname.lk' file to protect the database from having multiple processes writing to it when they shouldn't.

•  The database .lk file.  

Watchdog checks:

If any of the following checks fail, the watchdog will protect the database by initiating an abnormal shutdown:

1. Checks to see if the Database Broker is still alive, only the Watchdog can do this. Refer to Article  Investigating why the database crashes with error 4194  
2. Verifies the dbname.lk file exists on disk. Refer to Article  What causes ".lk is missing, shutting down 4195   
3. Verifies the dbname.lk file content is valid.  Watchdog uses the 3 items recorded in the dbname.lk file that are created when the database is accessed:

• Single user or Multi-user database start up mode 
• Process ID of process that started the database - PID of the Broker process or user process if in single user mode.
• Hostname where the process is running - Even if the .lk file is on a filesystem, the database could be started from a different machine that has the same filesystem mounted (when the database is already up and running). Knowing the host name verifies if the process id is on the same system or someplace else. For example, refer to the following Article  Why does an OpenEdge Database DBDOWN when the hostname is changed? 

Progress Articles: 

How the Database Watchdog Works?  
How does the TCP KeepAlive mechanism work?  
What does error 794 mean?