Salesforce

AppServer Agents remain in SENDING state forever when TCP chimney (TOE) is not turned off

« Go Back

Information

 
TitleAppServer Agents remain in SENDING state forever when TCP chimney (TOE) is not turned off
URL NameP179611
Article Number000163960
EnvironmentProduct: Progress
Version: 9.1x
Product: OpenEdge
Version: 10.x, 11.x
OS: Windows 2003 and later
Question/Problem Description
AppServer Agents remain in SENDING state forever when MicroSoft TOE is not turned off
The AppServer Agents need to be manually terminated
Communication link failure when the SPID is manually killed from MS SQL server
Unable to communicate with the databases
Disconnecting database due to server communication failure
 
Steps to Reproduce
Clarifying Information
The AppServer Agents become available when SPID is killed off from MS SQL server and communication link failure reported
This issue is most often seen with Broadcom Network Cards that have TOE enabled.
Error Message
Defect Number
Enhancement Number
Cause
Microsoft TOE (TCP Chimney) is not properly turned off.
Dataserver and AppServer process are waiting on network resources that are blocked by TOE.
Resolution
TCP Chimney Offload is a networking technology that helps transfer the workload from the CPU to a network adapter during network data transfer. TCP Chimney Offload enables the network subsystem to offload the processing of a TCP/IP connection to a network adapter that includes special support for TCP/IP offload processing, which is a hardware implementation of the TCP/IP stack
  • The TOE (TCP Chimney) feature was first introduced in Windows 2003 and is enabled when either the earlier Scalable Networking Pack (kb 912222) or Service Pack 2 for Win2003 or are installed on a Windows 2003 server.  
  • In Windows 8 / Windows Server 2012, TOC is disabled by default
  • Since Windows 10 Creators Update, TOC is deprecated 
 
To determine it TOC has been fully disabled verify the following:

1.   If the file version of %windir%\system32\netsh.exe is 5.2.3790.2670 or higher it is possible that TOE is enabled.  
If the version number is 5.2.3790.1860 (SP1) or lower the TOE features need not be worried about.  

2.   Open the registry editor: RUN > regedit

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 
 
Review the values of the following:  EnableTCPChimney, EnableTCPA, and EnableRSS.   
  • If all three are set to a value of Zero, consider TOE disabled.  
  • If they’re all set to 1, consider disabling them during troubleshooting.
3. Find in any active connections are in the “Offload State: Offload”

Open a command prompt and run the following command on the server:

$   netstat -t
 
In the Offload State Column:
  • InHost - the TCP Chimney Offload feature is disabled and offloading is not occurring. 
  • Offload - TCP Chimney Offload is enabled on the NIC on which that connection is established.

Disabling the TCP Chimney Offload feature

1. One of the three TOE features can be disabled ‘on the fly’.  Disabling the TCPChimney (but not RSS or TCPA) is often sufficient to deal with problems related to TOE.   But not always.  To disable the TCPChimney, open a command prompt and run the following command on the server: 

$   netsh int ip set chimney DISABLED
  • No reboot is needed.  
  • No IISRESET is needed either.
2. A more thorough (and preferable) alternative to using the netsh command is to set the values of EnableTCPChimney, EnableTCPA, and EnableRSS to 0 in the registry:
 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Change the dword values to zero for the following three keys from 1 to 0:
  • "EnableTCPChimney"=dword:00000000
  • "EnableTCPA"=dword:00000000
  • "EnableRSS"=dword:00000000

A reboot is needed for these registry value changes to take effect.

To view the current status:
$   netsh int tcp show global

Example:

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : disabled
NetDMA State                        : disabled
Direct Cache Access (DCA)           : disabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : none
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled
Initial RTO                         : 3000
Receive Segment Coalescing State    : disabled
Non Sack Rtt Resiliency             : disabled
Max SYN Retransmissions             : 2
TCP Fast Open                       : enabled


TCP Chimney Offload (TOE) is not unique to Windows Operating Systems, for example refer to Article:

Workaround
Notes

Reference to Written Documentation

Why Are We Deprecating Network Performance Features (KB4014193)?
https://techcommunity.microsoft.com/t5/Core-Infrastructure-and-Security/Why-Are-We-Deprecating-Network-Performance-Features-KB4014193/ba-p/259053


This article mentions the registry entries to check that will ensure disablement of TCP Chimney.
http://blogs.msdn.com/b/chaun/archive/2008/02/20/disabling-the-tcpchimney-during-iis-6-0-troubleshooting.aspx

 

Keyword Phrase
Last Modified Date11/20/2020 7:01 AM

Powered by