How to analyze a process crash or hang when the cause is unknown

How to analyze a process crash or hang when the cause is unknown

How to troubleshoot crashing or hanging _progress, _proapsv, _mprosrv or other Progress tools
How to use ADPlus, a debugger and other tools to troubleshoot hangs or crashes

Prologue: this knowledge base entry pertains to process crashes or hangs, not ABL errors nor application hangs that are due to the process running a loop in the ABL. These latter issues need to be investigated using regular 4GL debugging techniques. See the Debugging and Troubleshooting Guide for more details. Also, this article is not intended to be a replacement to a thorough debugging and troubleshooting methodology. Whenever possible, crash or hang issues should be submitted to Progress Technical Support as a simple reproducible case that allows Technical Support to reproduce the issue in-house.

When a Progress process is found crashing or hanging, the general approach to diagnose the issue is as follows:

1. Try to isolate the specific feature or behavior of the product that is triggering the issue. For this purpose the debugging output options for logging can be useful. See the related articles linked below for details. Then try to find a matching article in the knowledge base.
2. If the issue cannot be isolated, or no matching article is found, contact Progress Technical Support for further investigation.

During the troubleshooting process, it may be that the debugging output available through standard product features is not sufficient to provide an understanding about the cause of the issue. Should this case arise, Progress Technical Support may ask to investigate the current internal state of the running process at the moment it crashes or hangs. This requires additional details from the running environment that can be gathered before the issue is submitted to Technical Support. In order to optimize the time needed for a Technical Support investigation, use the information below to prepare the case for submission.

Gather the following details:

• The name (and path) of the executable file running the process being considered. It need to be an executable provided by Progress (i.e. the information below does not apply to 3rd-party executables).
• The exact version number and service pack / patch level of the Progress executable that crashes or hangs. In case of doubt, collect the exact size (in bytes) and time stamp of the executable file.
• A stack trace of the executing process at the point in time where the issue arises. If the process seems to be hanging, multiple stack traces generated at short intervals can be useful to determine whether the process is waiting or running in a loop. To obtain a stack trace:
  • If the process is crashing, a protrace file is generated, and the protrace file contains a list of memory addresses and function names, then it may contain a valid stack trace and can be submitted to Technical Support.
  • In other cases, consider the two sections below (depending on the operating system).

Consider contacting Progress Technical Support before attempting these steps. In some situations (especially for some executables) these steps will bring no valuable information.

On Unix-related operating systems (Linux included):

Note that the generation of a core file may be disabled on some systems; see the Notes section for details.

If the process is _progress, _proapsv or _mprosrv, and it appears to be hanging:

1. Send the signal SIGUSR1 to the hanging process multiple times (at 1-second intervals). This should cause no other side-effect than forcing the generation of a stack trace. The process should continue running afterwards.
2. If a protrace file is generated and contains a list of memory addresses and function names, then it may contain an appropriate stack trace and can be submitted to Technical Support.
3. Use strace (truss, tusc, etc dependent on OS) to determine if the process is performing any operations. See notes section below for instructions on strace.)

If the process is Java-based (including _sqlsrv2):

1. Try to obtain a Java exception log or stack trace. For this purpose refer to your Java VM documentation or contact Progress Technical Support.

If no protrace file is available, the process is not Java-based, and a debugger (dbx, gdb or other) is available on the system:

1. If the process is still running, determine the process ID (PID)
2. If the process has crashed and a core file has been generated, collect the core file
3. Attach the debugger to the process. For example:
   
   dbx <path to executable> <PID or core file>
   gdb <path to executable> <PID or core file>
    
4. At the debugger prompt enter the command to generate a stack trace (with gdb and dbx the command is called "where") and save the output. This can be submitted to Technical Support.

If no protrace file is available, the process is not Java-based, and no debugger is available:

1. Ensure that a « core » file is generated. If the process is hanging, send the signal SIGABRT to force the generation of a core file.
2. Contact Progress Technical Support for further instructions.

On Microsoft Windows platforms:

If the process is Java-based (including _sqlsrv2.exe):

1. Try to obtain a Java exception log or stack trace. For this purpose refer to your Java VM documentation or contact Progress Technical Support.

If no protrace file is available and the process is not Java-based:
Consider using the tool ADPlus from Microsoft to get additional information about the issue.

Prerequisites:

• If the issue is a crash, then the crash must be reproducible. No additional information can be gathered using this method after the crash has happened.
• ADPlus must be installed. See article What is ADPlus and where can it be found?.
• The instructions below refer to the ADPlus executable file adplus.exe; older versions of ADPlus may use adplus.vbs instead.
• The instructions below assume that the current directory is the directory where adplus.exe or adplus.vbs is located. If this is not true, fully qualify the name "adplus.exe" or "adplus.vbs" when running the executable. As of this writing  adplus.exe is typically found in "C:\Program Files\Debugging Tools for Windows (x64)" on 64-bit Windows.
• As of this writing there is typically a file adplus.doc in the same directory as adplus.exe. This file contains complete documentation on ADPlus usage.

For hanging processes:

1. Obtain the process ID of the process (using Windows' Task Manager)
2. Use ADPlus to generate a memory dump and stack trace of the hanging process:
   
   adplus.exe-hang -p <PID> -o <desired output directory>
    
3. Collect the generated log file and memory dump. As a first step send the log file to Progress Technical Support. Archive the memory dump in case the Technical Support engineer requests it at a later time.
4. Use Process Monitor to see if the process is performing any operations. (See download link in Notes section below.)

For crashing processes, when the crash does not happen immediately on startup:

1. Ensure the process is started
2. If the crash is known to happen in a specific process:
   1. Obtain the process ID of the process (using Task Manager)
   2. Attach Adplus to the process:
       
      adplus.exe -crash -p <PID> -o <desired output directory>
3. If the crash can happen in multiple running processes based on the same executable file:
   1. Determine the name of the running image (executable name)
   2. Attach Adplus to the family of processes:
       
      adplus.exe -crash -pn <process name> -o <desired output directory>
       
   3. Trigger the crash or wait for it to happen
4. After the crash happens, collect the generated log file and memory dump. As a first step send the log file to Progress Technical Support. Archive the memory dump in case the Technical Support engineer requests it at a later time.

This article is intended to help the investigation process but is not guaranteed to be sufficient to completely isolate the cause of the issue. Progress Technical Support may ask for additional troubleshooting steps and details on top of the points outlined here.

Contact Progress Technical Support before using this article if you are using a Temporary Fix or another unofficial patch with the product.

Do no send large files (core files, memory dumps) by e-mail to Progress Technical Support. Instead, ask Technical Support to send information on how to use a FTP server to exchange large files.

Enabling and disabling core files on Unix and Linux:
Limits on the size of a core file on Unix / Linux systems consist of two parts:

1. Hard limits imposed by the operating system or system administrator
2. Soft limits that can be manipulated via the shell.

The specifics of where and how the hard limit is set vary by operating system. Additional restrictions may be set on core dumps by suid programs. For more information, see the operating system documentation or the system administrator.

Use the shell command ulimit -c to get or set the current soft limit for core file generation.

 

References to Other Documentation:

OpenEdge Development: Debugging and Troubleshooting

Progress Articles:

Debugging
Both Unix and Windows

How to turn on 4GL tracing within WebSpeed, AppServer and the 4GL Client?

Unix only

How to use dbx to determine the general nature of what a hung process is doing.
How to attach to a process using  gdb  to generate a stack
How to get a stack trace from a Progress process on Linux?
How to use dbx to obtain a stack trace when no core is dumped
How to attach to a process using DBX
How to determine if DBX has been installed on IBM AIX
How to use strace to see if a process is hung.

Windows Only

Process Monitor download page: https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

Crashes
Unix Only

How to produce a readable text file from a core file that is created as a result of a Progress system error?
Why is a core or protrace file not created when a Progress process abnormally terminates?

Windows Only

 What is ADPlus and where can it be found?