Investigating why Flushing Buffer Pool: buffers remaining during database shutdown

Investigating why database shutdown is delayed by: Flushing Buffer Pool:  buffers remaining

Process left hanging in memory when PROSHUT command is issued
The database takes a long time to shut down while the processes are left hanging in memory

During the shutdown process, a complete flush of the dirty buffers takes place.

This issue has been known to occur at customer sites running with a really big -B value and no APW's, because it is just taking too long to flush the buffer pool to disc and resolve contention of backout processes. Only Enterprise Licenses support page writers (APW)

The message is informational, not an error. Eventually the system should shut down the database. Further investigations to find out why is it taking so long to flush memory to disk will alleviate this condition.

This Article only applies to OpenEdge versions prior to 10.2B, where shutdown improvements and the -shutdownTimeout parameter were introduced to limit the time period the PROSHUT process will wait for the database activity to end.  Refer to Articles:

 Message 13228 in the database log when shutting down prevents shutdown   
 Is there a parameter that will allow a DBA to control how long it takes to PROSHUT a database down normally?

1.  Before shutting the database down, disconnect all users. For further information, refer to Article   How to disconnect all self and remote users from a database?  

2. If the database needs to be more aggressively shut down, issue a forced shutdown:
Any active connections will be terminated and the shutdown process will not have to wait for these to exit themselves.
If the reason for shutting the database down is in order to back it up, insert a command to truncate the .bi file before starting the backup.
For further information, refer to Article  "What happens during PROSHUT -by and PROSHUT -F?"   

If this scenario is happening often, it is worth considering the following the next time this issue occurs:

1.) Before killing running processes or the Broker itself, execute the following:

a) Run a PROMON session before proshut to gather all information on user activity.
b) Check the system for any rogue processes that haven't shut down and produce a stacktrace against them. There could be a process locking the shutdown.
c) Produce a stacktrace against the shutdown _mprshut process itself with kill -16 <process-id> to generate a protrace file
d) Send Progress Technical Support the stacktraces that result from the above actions to analyse which processes are locking the shutdown.

2. Once PROSHUT has been killed:

Does a "PROUTIL -C truncate bi" perform a crash recovery without any error messages?
Errors would indicate file corruption which would need to be investigated at a system level.

3. Is this action being performed from the same_user_account / environment every time?
Permissions issues or the DLC environment variable not being set correctly could be influencing this behaviour.

4. Is shared-memory sufficiently tuned?
While very large shared memory (specifically -B Buffers) without the assistance of performant page writers will delay a normal shutdown while flushing buffers to disk,
Under-tuned (small) shared memory will delay normal shutdown when transaction activity needs to be backed out, having to re-read blocks into memory from disk.
As a general rule, the ‘Buffer Hits’ in PROMON should equate to > 85% or paging, under ‘typical' database load.

5.  Other considerations 
What is the disk wait activity like?
How long is the Buffer Pool flushing allowed to complete?
What other activity is running on the system?
Are only Progress processes hanging or are there other processes hanging as well?

If these cannot be explained, then it may be worth getting the system checked out by the Operating System vendor.

6. In the database log file, check to see if any of the following messages are recorded:
This indicates a contention situation for buffers in the -B buffer pool: (there probably aren't enough to go around). Increasing the -B parameter has been known to get us around the problem without the database becoming corrupted. If the force option is needed, (proshut dbname -F) then this could indicate a bigger problem and we should do a DATABASE SCAN followed by database repair actions if there are data corruption in the report.

++ 1124 errors

If issued by “BROKER”, then this indicates an OpenEdge defect because Brokers do not read blocks, it flushes the buffer pool to disk during shutdown.

++ SYSTEM ERROR: Unable to kill parent process, errno= . (1680)

Verify minport maxport settings for remote servers. Progress strongly discourages using ports < 2000, as these are reserved for UNIX processes. Changing these to port ranges above 5000 circumvents system errors concerning being unable to kill parent processes.

All of the above could be direct or indirect contributing factors to the delaying of the completion of the database shutdown.

References to Other Documentation:
Progress Article(s):
 Message 13228 in the database log when shutting down prevents shutdown