1. PASOE Agent log fileLook at the PASOE Agent log file in order to track the PID that terminated unexpectedly. If a new PASOE Agent PID is started while the PID that crashed got no further messages in the log file, that is an indication that the process terminated abruptly without disconnecting from the Databases or shutting down.
In order to get more details about the issue in the PASOE Agent log file, it is recommended to configure the following log entry types:
-MSASSignals : log entry type in order to have the PASOE Agent log file recording all signals the process received. That can help to explain if what happened to a process was related to a signal received.
-
4GLTrace : log entry type is a diagnostic feature that records the execution of the following 4GL statements in a log file: RUN, User-defined Functions, PUBLISH and SUBSCRIBE. That can reveal more detail about what the process was running shortly after terminating unexpectedly. If disk space is a concern, the recommendation would be to set the logging level to 2 and use a script to manage the log files as explained in the article in the Notes section below.
2. Operating System log fileWhen processes on your server require more memory than is available, there is a risk of downtime. To prevent such an ‘out of memory’ event from happening, the Linux OOM Killer process tries to kill processes in order to free up memory. This OOM Killer process is a last resort measure to prevent the server from going down in its entirety.
Look at the OS System log file (usually under /var/log/messages) and try to find errors like Out of Memory as in the example below for the PASOE Agent PID that crashed or stopped writing messages to the PASOE Agent log file:
[Wed Aug 31 16:36:42 2016] Out of memory: Kill process 43805 (keystone-all) score 249 or sacrifice child
[Wed Aug 31 16:36:42 2016] Killed process 43805 (keystone-all) total-vm:4446352kB, anon-rss:4053140kB, file-rss:68kB
[Wed Aug 31 16:56:25 2016] keystone-all invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[Wed Aug 31 16:56:25 2016] keystone-all cpuset=/ mems_allowed=0
[Wed Aug 31 16:56:25 2016] CPU: 2 PID: 88196 Comm: keystone-all Not tainted 3.10.0-327.13.1.el7.x86_64 #1
How to troubleshoot Out of memory (OOM) killer in Red Hat Enterprise Linux?, https://access.redhat.com/solutions/2612861
3. Operating System AuditingConfigure the Operating System to track kill signals as a kill -9 would cause a process to be killed without writing to the log file remaining information about whats happened. See below a link for an article which contains detailed information about OS Auditing configuration.
