When the database is started multi-user, it performs periodic tests to ensure the <database>.lk file is intact. The Broker Watch Dog process (or WDOG itself if started) performs this test, which is one of the many functions it performs. When it is unable to check the status of the <database>.lk file, the database will shutdown to preserve database integrity otherwise another process could open the database while it is already running. For further information refer to Articles:
Investigative measures:Finding the reason for the watchdog process actions, is what this Article hopes to address.
1. OS resource contention or hardware failure can fail the WDOG check to read the .lk file. The database usually succeeds crash recovery and restarts without issue.
This file check on the dbname.lk file can not be performed when resource contention, hardware failures or something else prevents the file from being read at the time. Any application software (not just Progress surprisingly most of the time without severe consequence), can and will behave erratically under these conditions.
For Example:
- Starting new programs the system will normally devote higher CPU time to the new tasks. In some instances a program which absorbs 100% CPU utilization for a period of time, during the period the Broker makes it's file system checks) the Broker may be unable to complete its programmed tasks and shutdown the database.
- Starting non-Progress utilities or even another database with high shared memory may cause memory contention at the time the Broker checks the status of the .lk file.
- Non-Progress utilities temporarily block access to database files, including the database lk file which fails the WDOG check to read the .lk file.
- Similarily disk i/o contention by one or more processes on the same disk that the database lk file is housed can cause the read to fail.
These instances are usually accompanied by errors:
<file-name> is missing, shutting down... (4195)
Error opening file <file-name>, errno = <errno>. (4029)
<dbname>.lk is missing, shutting down... (4195)
Intermittent hardware problems like a failing disk controller can cause causing database file locations to be temporarily inaccessible. The accompanying errors listed above may also be reported on other database files used by the application at the same time (.db or .d1 usually)
The following messages, often but not exclusively precede 4195 which may occurs on one or more of the databases used:
(9450) bkioRead:Insufficient disk space during Read
(9450) bkioWrite:Insufficient disk space during write,
(14676) bkioWrite:Unknown O/S error during write
(291) Insufficient disk space or Write access denied
(43) ** Cannot find or open file <file-name>, errno = <number>
(1450) ** stget: out of storage
[and perhaps]: (13198) The after-image management daemon cannot find a destination in its current list with enough space to continue archiving extents.
The critical difference is that:
4195 lk is missing, shutting down... is fatal, where others retry or are less fatal (the client or remote server terminates)
We also need to understand that the
'insufficient disk space" messages are not to be taken literally:
- This essentially is better worded as ''insufficient disk space .. or resources or access denied at the time'
- There are multiple reasons for this generic error being thrown including 3rd Party access to the runtime database environment, however memory resources is the most probable cause when the rolling symptoms above are presented.
There has been a long standing issue with Windows Memory Management circa Windows 2003, exacerbated in later versions where higher memory are available. Specifically when the PoolUsageMaximum (%) of the maximum PagedPoolSize is reached, at which stage the Memory Manager trims pool memory which may not be released fast enough. There are times when subsequent database startup then fails or succeeds but then fails pretty quickly because memory still hasn't been flushed and requires a machine reboot to resolve. For Windows Servers hosting databases, virtual memory may need increasing and OS specialists will need to monitor and further tune the server to support database shared-memory requirements.
System resource allocation and perhaps more importantly efficient application resource usage, is paramount to stable production systems.
- Review the OS log files for further details which may have been logged at the time.
- Review Operating System resource monitors to determine when process(es) are using more system resources than expected
- Investigate what else has been introduced to the environment, which are (now) using more system resources than before
- Add more system resources to cope with increased demand when warranted in the first instance or review how to spread the resource requirement for the resources available.
2. On UNIX systems, scripts that delete the dbname.lk file without checking for completion of database shutdown.
This is usually, but not always accompanied by failure to restart the database with error 1260
Shared memory in use by another process. (1260)
Never assume to delete database .lk files without investigating each instance of why they're remaining in the first instance. At the very least modify scripts to verify that the database has indeed successfully shutdown before considering deleting the database.lk file.
Another example of this is in Windows and Unix production environments, is when after a single-user connection fails due to network issues the lk file is deleted in order to start the database multi-user. Further details of an incident that was fully investigated are described in the following Article:
3. A similar message
4194 results when the database Broker itself is terminated with an OS kill or taskill, where the database engine forces an abnormal database shutdown as a result.
(4194) Broker disappeared, updating dbname.lk file
For further troubleshooting advice, refer to Article
Database crashes with error 4194 
