For cases where
asbman -stop will not stop an AppServer Broker but hangs instead, an emergency shutdown of said Broker can be instructed with the
asbman -kill instruction. The AppServer Broker will then terminate, and the prior
asbman -stop session that was hanging will terminate cleanly. When
asbman -stop is hanging, from another session, run:
$ asbman -i <broker name> -kill
Understanding the differences between ASBMAN -stop and -kill:
- If the Broker is not involved in an action when either an asbman -kill or asbman -stop is first sent to a Broker, it will terminate the Broker after it has performed normal shutdown operations.
- When the Broker is involved in an action when an asbman -stop is sent to a Broker, it does not cause the Broker to terminate immediately, until outstanding actions complete.
- When the Broker is involved in an action when an asbman -kill is first sent to a Broker, the kill is interpreted the same as an asbman -stop instead of a -kill, which does not cause the Broker to terminate immediately. Only when a second asbman -kill is sent to the Broker, will this then cause the Broker to terminate without any consideration of requests currently being processed. Only when an asbman -stop | -kill is followed by another -kill, will it behave as an emergency shutdown.
Understanding The Broker Emergency Shutdown Feature:This emergency shutdown feature is provided as a "
last resort" measure to shutdown the AppServer Broker when normal shutdown process don't work. It is not intended to be used as a substitute for normal shutdown. As such, the emergency shutdown works as follows:
1. When the asbman -kill command is first issued (without a pre-ceeding asbman -stop), the AppServer logs that fact as an 8422 message in the log file:
Emergency Shutdown requested. (8422)
2. The Broker then attempts to perform a normal shutdown which involves:
- The orderly shutdown of the Listener Thread,
- Disconnecting each Client Connection,
- Initiating Disconnect Processing,
- Stopping each of the Agent processes.
3. When normal shutdown does not succeed, the 'asbman -kill' command should be re-issued. When the Broker receives this command again, the emergency shutdown is performed. That is, the Broker process exits immediately without regard for the current state of Client requests or Agents.
When all the Agents don't normally shutdown from the first request, asbman -kill says: don't wait for the Agents, ** just shutdown the Broker **
Once the Broker is shutdown, there will be orphan Agents. The Agents should notice (next time a free cycle comes up) that the PID for the Broker is gone and they should then shutdown.
If an orphaned Agent is in a state where it never has a free cycle to check to see if the Broker PID is still there, it will remain running even when the Broker is restarted (asbman -start). This could be caused by:
a. Agent is in an infinite loop
b. Agent is blocked waiting for a response, which it may never get. This response could be network related or a call outside using OS-COMMAND.
'Best Practices' procedure for shutdown of an AppServer Broker
- Always try a normal shutdown first with asbman -stop.
- If it does not succeed, use the asbman -kill command which terminates the Broker.
- Wait for several minutes to see if all of the agent processes still persist on the machine (or some do unblock?)
- kill SIGQUIT the remaining agents taking note of their PID's for tracing in the related Broker and Server .log files
- Collect the protrace files to determine:
a. If there was still persistent 4GL at the time
b. What the stack trace says the agent was doing (in a network call, O/S call, etc). If it is network related, UBNet logging or Network specialists can further investigate through Wireshark tracing for example.
