Steps for Troubleshooting Slow Network Issues

The following are a few steps which are useful to follow while troubleshooting slow
network communications over a WAN link.

**To troubleshoot network issues more effectively you must know the Ins-and-Outs of
the TCP/IP stack.  For a great beginning (and in depth information) please refer to
the book “Windows Server 2003 TCP/IP Protocols and Services Technical
Reference”.**

1) Take traces simultaneously between client and server

2) Check for delays in these queries
a) QUERY_PATH_INFO
b) QUERY_FS_INFO
c) FIND_FIRST2
d) NT_CREATE_ANDX
e) Trans2 GET_DFS_REFERRAL, NEGOTIATE, SESSION_SETUP_ANDX, LOGOFF_ANDX

        Note : If you are using Network Monitor, you can see the delays better if
you use the following filter:
SMB:Command == 0xA0 ( NT transact )
    Use this filter in combination with the following display options:
Time: (x) Seconds from previous frame
             You can identify the corresponding requests and answers by looking at
the following SMB frame attribute:
SMB: Multiplex ID (MID)

Average Response time from server should be like .002 ms.  If you see a server-side
delay from the client and a client-side delay from the server, then it’s probably a
network issue of some sort.

3) Try sending files to and from different servers and clients on the same segment.
 Try also sending file from server seeing the issue to a different receiving
client.  This may narrow it down to a specific machine seeing the issue.

4) Update all the NIC drivers on client and the server.
5) Set Network card and switch to a 100mb full instead of autodetect  (it is very
important that the two must match.  If they are mismatched then you could see
multiple problems)

6) If possible, start the machine in safe mode with networking and test.  Sometimes
third party components may cause delays.  In cases such as antivirus possibly
causing the slowness (very typical as it sometimes tries to scan everything leaving
and entering the machine) the only true way to test is to fully uninstall the
product because of the Kernel level filter drivers. (Also confirm for any 3rd party
servers such as SAN, SOLARIS, IBM etc.)

7) Try to access share on itself by giving loopback address to the server / client 
which will eliminate network components and take a trace should be done prior to
updating MRXSMB and Redirector

8) Update MRXSMB.sys file and RDBYSS.sys file both on the server side and the
client side  and also netbt.sys file.  Updating the Netbt.sys file will Improve
Redirector / server connections ( solution for Nagling and TCP delayed
acknowledgment ) Here are some articles for each that will help you in updating:

- Latest MRXSMB.sys file for Windows Servers (All Versions)
a) Windows 2003 Server – KB article 913835
b) Windows 2000 Server -  KB article 907546
c) Windows XP SP2 ( All versions) – KB article 896427

      – Latest RDBYSS.sys file for Windows Servers ( All versions)
a) Windows 2003 Server – KB896427
b) Windows 2000 Server – KB907546
c) Windows XP SP2 ( All versions) – KB 903891

      – Latest NetBt.sys file for windows Servers ( All versions)
a) Windows 2000 Server – KB article 984250

* Please check the KB for more updated fixes.  These are just pointers to help
out.
9) A common issue in networking is to mistake a shell issue with a slow network
issue. These issues can be seen in Mapped drives and browsing.  Any time you are
using the GUI you are using shell32.dll.   This would be considered a Performance
based issue. To test this theory you can go to a command prompt and do a Net View
then a Dir of the network resource. Does it populate all information quickly? If so
it may be a shell issue.  Prior to bringing in performance you can try updating the
shell32.dll.  Here are some articles for each that will help you in updating:

- Latest Shell32.dll file for Windows Servers (All Versions)
a) Windows 2003 Server – KB article 903154
b) Windows 2000 Server -  KB article 898068
c) Windows XP SP2 ( All versions) – KB article 912680

* Please check the KB for more updated fixes.  These are just pointers to help
out.

10) TCP Checksum Offloading : The TCP checksum is added to the packet by the
network interface,  not by  the OS’s TCP/IP stack; when capturing on an interface,
packets being sent by the host on which you ‘re capturing are directly handed to
the capture interface by the OS, which means that they are handed to the capture
interface with out a TCP checksum being added to them; So one way to prevent this
from happening is to disable TCP checksum offloading.

Disable TCP / IP checksum offloading. Please follow the procedure to disable the
TCP / IP checksum Offloading
     
a). Click Start, click Run, type ncpa.cpl , and then click OK.  
b). Right-click your network adapter, and then click Properties.  
c) Click the General tab, and then click Configure.  
d) Click the Advanced tab.  
e) In the Property box, click any one of the following properties.   Note We
recommend that you start with the Offload TCP Segmentation property. The problem is
probably caused by the Offload TCP Segmentation property because full support for
this feature is not implemented in the Vpcnets2.sys driver.  • Offload TCP
Segmentation   • Offload Receive IP Checksum   • Offload Receive TCP Checksum   •
Offload Transmit IP Checksum   • Offload Transmit TCP Checksum       
f). In the Value list, click Off, and then click OK.  
g). Test to see if the performance issue has been resolved. If the problem still
occurs, re-enable the property that you turned off, and then disable a different
task offload property. Test if performance has improved after you disable each
property.  

If your NDIS driver does not have the settings available to disable tcp checksum
offloading, then you can disable it by editing the registry as follows

a) Click Start, click Run, type regedit, and then click OK.
b) Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet
<http://wiki.colinux.org/cgi-bin/CurrentControlSet>\Services\Tcpip\Parameters
c) Click Edit, point to New, and then click DWORD Value.
d) Type DisableTaskOffload <http://wiki.colinux.org/cgi-bin/DisableTaskOffload> as
the entry name, and then press ENTER.
e) Right-click DisableTaskOffload
<http://wiki.colinux.org/cgi-bin/DisableTaskOffload>, and then click Modify.
f) In the Value data box, type a value of 1, and then click OK.
g) Quit Registry Editor.
h) Restart your computer

11) Use TCPDELACKTICKS if you see that server is not sending any packets nor it is
not responding for more than 200ms , at that point you see TCP ACK packet sent to
resume connection and this behaviour is because of TCP Delayed Ack.  We can edit
TCPDelAckTicks registry value for better performance , use the following registry
key
       
a).  Start Registry Editor (Regedt32.exe). 
b).        Locate and click the following key in the registry, where Adapter GUID
is the globally unique identifier (GUID) for the network adapter that connects to
the clients: 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\Ada
pter GUID  
 On the Edit menu, click Add Value, and then add the following registry value: 
Value name: TcpDelAckTicks Data type: REG_DWORD Value data: You can set this value
to a range from 0 to 6. The default setting is 2 (200 milliseconds).  
 Quit Registry Editor. 
 Restart Windows for this change to take effect.  
This registry value specifies the number of 100-millisecond intervals to use for
the delayed ACK timer on a per-interface basis. By default, the delayed ACK timer
value is 200 milliseconds. If you set the TcpDelAckTicks value to 0, delayed
acknowledgments are disabled. This setting causes the computer to immediately send
an ACK packet for every packet it receives.

 
12) You can also try using TCPAckFrequency registry key.  This will cause every
packet to get Acknowledged, you can reference more information on this by looking
at KB article 328890 or by reading the TCP/IP Protocols and Services Technical
Reference. Understand that this will also cause an increase in data on the lines as
you are acknowledging every packet.  After following 328890, if you don’t see any
changes please look at KB article 815230.  It updates TCPIP.sys.
      
13) Enabling ReadAhead feature on Sequential Reads ( This is very important and
probably a best resource ). Please go through 328237 KB article number. And look
for
        
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanWorkStation\Parameters
             Value name: ReadAheadGranularity
                         Data type: REG_DWORD
      
                       
14) Opportunistic Locking :   By default, opportunistic locking is enabled for
server message block (SMB) clients that run one of the Windows operating systems.
Opportunistic locking allows clients to lock files and locally cache information
without the risk of another user changing the file. This increases performance for
many file operations but may decrease performance in other operations because the
server that grants the opportunistic lock must manage the breaking of that lock
when another user requests access to the file. A network trace is going to be the
determining factor in this.  You will see 8 – 16 second delays in which it is
trying to undo the opportunistic lock.  It may try multiple times.  With that being
the case you can go through the following article to disable Opportunistic
locking:
                       
      <http://vkbinternal/VKBWebService/ViewContent.aspx?scid=KB;EN-US;296264>

15) Check for SMB signing between both client and server.

a) To check for SMB signing Please go to the following registry key
b) Locate, and then click the enablesecuritysignature value under the following key
in the registry:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanserver\Parameters
c) To disable SMB signing Please refer to KB article 321733

             
16) The following KB article is one of the useful article I have found that goes
over Network Shares opening slowly, read only or if you get an error message while
trying to open:
<http://vkbinternal/VKBWebService/ViewContent.aspx?scid=KB;EN-US;814112>

17)  Windows 2000 and Windows 2003 server uses Large File Support and not all
earlier version   of OS have this capability to perform faster Write and Read file
operations
So our main goal is to send large Read&X and Write&X , so we can tweak the
performance using SizReqBuf Registry key on Windows NT server . And the main reason
is
Windows NT server works in Raw SMB mode so Windows NT server doesn’t have Large
File Write Support.
To work around this performance issue, make the following registry change on the
Windows NT 4.0-based server:
      a).  Start Registry Editor (Regedt32.exe). 
      b). Locate and click the following key in the registry: 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters  
 On the Edit menu, click Add Value, and then add the following registry value: 
Value Name: SizReqBuf Data Type: DWORD Value Radix: Decimal Value: 65535  
     c)   Quit Registry Editor. 
     e). Restart the server 
After you make this change, the performance of the write process is approximately
the same as a read operation between the two computers.                            
                                              

Note that the SizReqBuf value controls the buffer size for CORE SMB requests.
Setting it to 64 KB has approximately the same effect as Large Write support, which
uses 60-KB buffers.
From the tests completed , please set the ReadAheadGranularity to 16 on the
clients.