The associated stack trace for parallel investigations with the associated log files should first be gathered before issuing a forceful process shutdown (
kill -9) will get rid of the AdminServer processes. It is very important that the correct process is terminated and the corresponding evidence generated is recorded. Please note simply issuing a kill against the process is not necessarily the best way forward in all cases and should be taken under cautious advisement.
Once a hang occurs, iow: the
proadsv command stops responding, the Steps below should be used to ensure that the appropriate information is gathered. While undertaking these steps, be sure to keep track (by PID) which processes were killed.
Summary of Steps:
1) Find the AdminServer process and the path to the java runtime executable.
2a) If the feature is available, ask the Java runtime to generate a Java core file.
2b) Ask the Java runtime executable to generate a UNIX core file.
2c) If possible, get the stack trace from the core file
3) Terminate the remaining processes
4) Restart the AdminServer.
Detailed Steps:0) Open a UNIX session as the system administrator or as the user who initially started the AdminServer process.
1) Find the AdminServer process, using the "ps" command.
Example:
AIX/Solaris: $ ps -ef | grep java | grep AdminServer
GNU/Linux: $ ps auxw | grep java | grep AdminServer
Take note of the
full path to the java executable for the core file produced in Step 2c below.
Example: "-Dadmsrv.jvm=
/usr/java15_64/sdk/jre/bin/java"
2) Ask the process to tell you as much as possible about its current state saving the associated evidence produced.
2a) Create a javacore
Steps differ depending on the platform, consult the related man pages for further details, this feature is not available on all UNIX implementations. In all cases ensure you have the
correct process-id.
AIX: Send it a SIGFPE signal (signal 8). This asks the java layer (javacore) to create a dump of its current state in a file named "javacore<number>" in the working directory. A listing of javacore's created can be referenced in "/tmp/javacore_locations".
$ kill -8 <PID>
For other systems using the Sun Java virtual machine send the process a SIGQUIT signal (signal 3). This asks the java layer to create a Java stack trace on its console, which is redirected to the AdminServer log file, or a file named admserv.exp in the working directory.
$ kill -3 <PID>
2b) If the process did not terminate while generating a java core, send the signal SIGABRT to the process, to get the information from the c-layer.
$ kill -6 <PID>
This will result in a "core" file in the working directory. Save it for Step 2c.
A number of "protrace.<pid>" files could also be created (from the running Progress sessions started by AdminServer). Save this evidence as well.
2c) Use a debugger to get a stack trace from the "core" file. The complete path to the java executable is needed here (extracted in Step 1 above).
For example if dbx is available, using the information saved in Step 2b:
$ dbx /usr/java15_64/sdk/jre/bin/java core | tee core.out
[dbx]where
.
[dbx]detach
[dbx]quit
If dbx is not available use gdb instead. (
http://www.gnu.org/software/gdb/download)
3) Remaining processes managed by the AdminServer may have not terminated gracefully with the previous steps and will need to be removed. (eg: java, _proapsv)
For remaining processes, first send SIGTERM (kill -15) which are not responding
After a delay, if this is not sufficient use SIGKILL (kill -9 ...).
If on AIX, run "
/usr/sbin/slibclean" which removes any currently unused modules in kernel and library memory.
4) Restart the AdminServer
$ proadsv -start -port <portnumber> -adminport <portnumber>
