tcp

Sep 30 2009   1:06PM GMT

Where do TCP resets come from?

Posted by: Troy Tate
tcp, udp, network management, network performance, network monitoring, application performance, network analysis, performance analysis, protocol analysis, packet capture

I recently came across an excellent article on the topic of TCP resets. TCP is a connection-oriented protocol as opposed to the connectionless nature of UDP. So, if there are TCP resets on your network, this is not a bad thing and is just inherent in the protocol. Without TCP resets, a host could have a lot of partial connections established which are in the wait state awaiting further transmissions. This can exhaust the number of available sockets and cause the host to become unresponsive. This is what happened several years back with the TCP SYN flood and LAND denial of service attacks. Another reset type includes the ACK/RST. This is where a client attempts to connect to a service that is not available on that destination host.

If you manage a network and have taken packet captures to work on a problem and have seen RST packets or if you need to do this at some point in your career, you need to understand the purpose and source of the RST packets. Take a few minutes, read this excellent article that is the best explanation that I have seen on this topic. You will become better informed and better able to understand the nature of the network beast.

Where do resets come from? (No, the stork does not bring them.)

Thanks for reading and let’s continue to be good network citizens.

Sep 14 2009   1:49PM GMT

Microsoft does not patch vulnerability for supported version of Windows

Posted by: Troy Tate
Microsoft, information security, vulnerability, risk management, patches, tcp-ip, tcp, tcp/ip, Windows, windows 2000, support, Microsoft support, threat, risk

Last week was the September issue of Microsoft “patch Tuesday”. The September 2009 Microsoft Security Bulletin lists a number of vulnerabilities. Microsoft held the bulletin webcast on Wednesday, September 9, to discuss the vulnerabilities and customer concerns.

One particular bulletin is creating some concerns for Microsoft Windows 2000 users. MS09-048 is a bulletin for a vulnerability to the TCP/IP stack in all current supported versions of Windows. The bulletin describes the vulnerability:

Vulnerabilities in Windows TCP/IP Could Allow Remote Code Execution (967723)

This security update resolves several privately reported vulnerabilities in Transmission Control Protocol/Internet Protocol (TCP/IP) processing. The vulnerabilities could allow remote code execution if an attacker sent specially crafted TCP/IP packets over the network to a computer with a listening service. Firewall best practices and standard default firewall configurations can help protect networks from attacks that originate outside the enterprise perimeter. Best practices recommend that systems that are connected to the Internet have a minimal number of ports exposed.

Even though the bulletin here describes it as potential remote code execution, the webcast focused more on the denial of service threat due to this vulnerability. Unfortunately, Microsoft has chosen to not issue a patch for Windows 2000, even though Windows 2000 is a supported version of Windows with regards to patches and security fixes. ComputerWorld gives a good amount of detail in the article: Microsoft: Patching Windows 2000 ‘infeasible’ Dark Reading published Microsoft, Cisco Issue Defenses For TCP Denial-Of-Service Attack and The Register published Microsoft, Cisco issue patches for newfangled DoS exploit.

I know that there is a reasonable population of Windows 2000 machines in operation at my organization. So, this choice by Microsoft to not issue a patch for this vulnerability raises some concerns. Fortunately the vulnerable population is not publicly exposed and does not have mobile users. The layered defenses we have in place should help mitigate the risks to our environment. However, the risk is still there and the threat needs to be addressed. What other vulnerability will come out that Microsoft chooses not to address in a supported operating system? Are you facing the same situation in your environment? How large is the risk to your environment? What are you doing to address these threats? Why are you doing what you are doing? Share your thoughts with other ITKE readers.

Thanks for reading & let’s continue to be good network citizens.

Apr 29 2009   12:11PM GMT

Doing Microsoft packet analysis? - Microsoft releases Network Monitor 3.3

Posted by: Troy Tate
packet analysis, packet capture, protocol analysis, tools, analysis, analysis tools, Microsoft, network analysis, network, tcp, udp, network monitor

If you do packet capture or analysis in a Microsoft environment, then you are probably already familiar with Microsoft Network Monitor. If not, please read my real-world use of it for PROTOCOL analysis vs protocol analysis (with a small p). Microsoft has updated Network Monitor to v3.3. The announcement of its release can be found on the Technet blog. Some of the new features listed are:

· Ability to capture WWAN (mobile broadband) and Tunnel traffic on Windows 7.

· Full Hyper-V support on Windows Server 2008

· Right-click-add-to-alias: Right-click a frame in the Frame Summary window with an IPv4, IPv6 or MAC address to add that address as a new alias. This is one of those little things that simplifies your work-flow.

· Right-click-go-to-definition: Have you ever wondered where and how the protocols fields you see in the Frame Details are defined in our in-built parsers? Wonder no more. Introducing right-click-go-to-definition: right-click a field in the Frame Details window and select Go To Data Field Definition or Go To Data Type Definition to see where the field is defined in the NPL parsers.

· Autoscroll: Another one of those little, but priceless things … auto-scroll. See the most recent traffic as it comes in. In a live capture, click the AutoScroll button on the main toolbar to have the Frame Summary window automatically scroll down to display the most recent frames as they come in. Click Autoscroll again to freeze the view in its present location.

Several other new features are described in the Technet blog. If you capture packets on a Microsoft network, then you should get this upgraded version to add to your toolbox.

Thanks for reading and let’s continue to be good network citizens.

Jan 9 2009   4:38PM GMT

PROTOCOL analysis vs protocol analysis (with a small p)

Posted by: Troy Tate
protocol analysis, SMTP, tcp, network monitor, wireshark, Microsoft, Microsoft Exchange, patches, OSI model

Recently we had an issue at a site where outbound messages larger than 1MB were backing up in the outbound message queue. The messages were tagged with a 421 4.4.2 Connection dropped error. This was a puzzling issue since the smart relay host was on the local LAN, and in fact, on the same switch as the Exchange server.  We checked the switch ports and NICs for errors. None were found. We knew messages were successfully coming inbound through this site because the smart relay host was processing hundreds of them per hour (we use regional hubs and this is one of our hub sites).

We first contacted the vendor for the smart relay host appliance and opened a support ticket. No real issues were identified at first review. Since the errors were being reported at the Exchange server, we contacted Microsoft and opened a support ticket. We spent hours testing and changing configuration to another regional smart relay host which seemed to get the messages delivered successfully, but we were still not able to find out what was causing the conversations with the local smart relay host to timeout.

So, we went into deeper debug mode since the application and server event logs did not shed any light on the issue. The Microsoft engineer enabled protocol logging on this particular send connector. The protocol logs did give a little more information on the situation. A snippet is shown below.

2009-01-08T22:36:19.495Z,SendConn,08CB3FF87FA34699,16,exchsvr:20709,relayhost:25,>,RCPT TO:<someone@there.com>,
2009-01-08T22:36:19.495Z,SendConn,08CB3FF87FA34699,17,exchsvr:20709,relayhost:25,<,”250 Requested mail action okay, completed.”,
2009-01-08T22:36:19.589Z,SendConn,08CB3FF87FA34699,18,exchsvr:20709,relayhost:25,>,DATA,
2009-01-08T22:36:19.589Z,SendConn,08CB3FF87FA34699,19,exchsvr:20709,relayhost:25,<,”354 Enter mail, end with “”.”” on a line by itself.”,
2009-01-08T22:36:25.417Z,SendConn,08CB3FF87FA34699,20,exchsvr:20709,relayhost:25,-,,Remote
2009-01-08T22:37:25.431Z,SendConn,08CB3FF87FA346A1,0,,relayhost:25,*,,attempting to connect
2009-01-08T22:37:25.431Z,SendConn,08CB3FF87FA346A1,1,exchsvr:20736,relayhost:25,+,,

The conversation seemed to go fine at the beginning but something was happening at the end. Since this log did not freely give up that information, we used Microsoft’s Network Monitor 3.2 (btw-if you are still using an older version of Network Monitor, you should upgrade to v3.2. It does have some nice features that make it more user friendly - but not as nice as Wireshark) to capture the actual packets between the Exchange server and the smart relay host. We ran Network Monitor directly on the Exchange server.

At this point, we were able to capture the transaction failures. The results were very interesting and a good lesson in packet analysis versus protocol analysis. The packet analysis showed that TCP was working well. Everything at layer 4 and below seemed to be working well. This was a relief. However, it appeared that the actual problem existed at layer 6 & 7. The Exchange server was ending the SMTP (Simple Mail Transport Protocol) conversation with the “.” command (a single dot on a line by itself). The Exchange server was then waiting for the smart relay host to reply with a 250 2.6.0 status message saying the message was successfully queued for delivery. The Exchange server would then reply with a QUIT command and end the SMTP session. Since the smart relay was not responding at all with the expected status message, the SMTP conversation was timing out and messages were building up in the queue.

We found out that there were some patches for the smart relay host so we applied those. Once that was done, the messages seemed to flow normally. The other puzzling thing about this is that we have two other hub sites with the same configuration that are not experiencing this problem. So, sometime today we will be rolling out the patches to those smart relay hosts to prevent this problem from happening at those sites. This issue started out of the blue but seemed coincide with the same time Exchange Server 2007 rollup 5 was applied.

The point of this whole blog posting is that while the TCP protocol was working fine and everything looked good there, the SMTP protcol was not working correctly. It is important for a network engineer to understand networking through all of the OSI layers. You cannot just assume that if things are working well at the lower levels that things at the higher levels will work too. The reverse logic is true also. So, understand the protocols at the lower layers and also the PROTOCOLS at the upper layers if you really want to be an effective troubleshooting expert.

Let’s be good network citizens out there!

Jan 6 2009   4:45PM GMT

Swiss-army knife for public network testing

Posted by: Troy Tate
toolkit, tools, testing, connectivity testing, website, dns, ping, tracert, icmp, tcp, udp, public network, ssh, SSL, cryptography, crypto, crypto testing, hash, typosquatting

Sometimes it is necessary to test connectivity outside of your private company network. There are several resources I use. I will share a couple of those with you in this posting.

One of my favorite and most frequently used sites is Network-Tools. This website allows you to test Traceroute, PIng, Domain Name Server (DNS) lookup, Whois, and DNS record lookups. This is an excellent resource like DNSTools or DNSStuff.

Another site with useful public internet testing tools is Serversniff.net. You can use this site to perform TCP pings rather than the standard ICMP pings. There is also a step-ping test. This provides the ability to have increasing ping packet sizes to see if there is a bottleneck somewhere before the tested host. There are lots of other tools available on this website. I recommend you check it out and see which offer value to you in your support activities.

Unfortunately, these tools only work from the public internet. You will not be able to test hosts on your private network, but hey, shouldn’t you already have some other testing tools in your toolbag for the private network? I’m sure I will describe more tools as the year moves on.

Thanks for reading & let’s practice safe networking out there! Please feel free to leave comments for other readers so they can adequately support their networks.