IT Trenches » network management

Is there an orange ball in information security?

Troy Tate — Fri, 10 Sep 2010 17:05:59 +0000

In Japan many retailers have what look like orange balls or waterballoons near checkouts. Bruce Schneier, a leading information security expert, recently blogged about these orange balls as anti-robbery devices. Could we find any type of orange ball to use for information security, to “mark” packets of attackers and then track them back down? What do you think?

Thanks for reading & let’s continue to be good network citizens!

I resemble that award winning case study – wait, it IS me!

Troy Tate — Wed, 30 Sep 2009 13:36:27 +0000

Have you ever wondered if vendor case studies are actually solutions to real life issues or if they are stories about compensated organizations using a particular vendor solution? Well, I am here to tell you that I know of at least one case study that is about an organization addressing real-life issues that was featured in an award winning vendor case study. The organization is the company I work for and the case study is about the challenges we faced with replacing an under-performing legacy Frame Relay network with a more efficient and flexible global solution that delivers high availability, remote access, and integrated security. For the record, no compensation was given for being the subject of this vendor case study.

The case study won the 2009 Best Deployment Scenario – VPN/IPSec/SSL and was featured in the Info Security Products Guide. The winning case study and announcement can be found at Manufacturing Company Achieves Security and Performance Goals with Virtela’s Remote Access Services from the Cloud.

See all 2009 Best Deployment Scenarios and Case Studies. This would be a good time to look at these and see if any of the solutions may meet some of the information security needs of your organization. Consider putting the solutions in your 2010 budgets.

Feel free to leave comments here or contact me through ITKE if you would like more information. Thanks for reading & let’s continue to be good network citizens.

Where do TCP resets come from?

Troy Tate — Wed, 30 Sep 2009 13:06:36 +0000

I recently came across an excellent article on the topic of TCP resets. TCP is a connection-oriented protocol as opposed to the connectionless nature of UDP. So, if there are TCP resets on your network, this is not a bad thing and is just inherent in the protocol. Without TCP resets, a host could have a lot of partial connections established which are in the wait state awaiting further transmissions. This can exhaust the number of available sockets and cause the host to become unresponsive. This is what happened several years back with the TCP SYN flood and LAND denial of service attacks. Another reset type includes the ACK/RST. This is where a client attempts to connect to a service that is not available on that destination host.

If you manage a network and have taken packet captures to work on a problem and have seen RST packets or if you need to do this at some point in your career, you need to understand the purpose and source of the RST packets. Take a few minutes, read this excellent article that is the best explanation that I have seen on this topic. You will become better informed and better able to understand the nature of the network beast.

Where do resets come from? (No, the stork does not bring them.)

Thanks for reading and let’s continue to be good network citizens.

Performance monitoring dashboard – fping and URL ping

Troy Tate — Fri, 25 Sep 2009 15:15:02 +0000

In part one of this series, I discussed ping and pathping. These tools are good for some interactive realtime testing. However, what do you do when you want to run these types of tools over an extended period and then do statistical analysis? In cases like this I use the fping tool. I recently completed an analysis task requiring comparison of network ping times against web server response times. The tool I used for measuring webserver response (time to first byte) is called URL ping. Users were reporting slow webserver (Sharepoint) performance. Everyone was saying it is a network issue. Since there are so many “moving” parts between the users and the webserver farm, I wanted to prove to them that the network was not the issue but that something inherent in the way the webserver responds to the requests is the real issue.

The URL ping tool is described in the Microsoft IT Showcase: sharepoint Performance Optimization Technical White Paper. It is a C# script that will need to be compiled using the proper include files and configuration for your environment. I configured the script to log in CSV format the date, time and time to first byte from the webserver in question. Using the console output redirect pipe, I logged the information to a text log file. I configured a Windows scheduled task to run the URLping command over the time period of interest. The URLping command ran every 30 seconds logging the results to the specified text file.

During the same period, ever 30 seconds, I ran the fping utility from Kwakkelflap.com. The fping utility is much more flexible than the ping tool that is part of the Windows operating system. Some of the features that make fping so useful includes:

Time between pings can be adjusted as needed from 1ms to 5s.
Beep on every successful or unsuccessful reply allowing you to test your network status in the background.
Ping multiple hosts with one simple command.
Read a hostlist from a file
Output redirection to a file for parsing.
Ping with random data, or data you provide

The results from these two utilities were logged into two separate CSV files. Since the tests were running every 30 seconds, I could take the fping results and the urlping results and combine them into one spreadsheet for analysis. I wanted the units for the urlping and fping response times to be the same. I had to divide the fping results, which were in milliseconds (ms), by 1000 to convert the results to seconds to match the units of the urlping results. I then graphed the results. This information is shown below. The blue data points are the urlping results. The pink data points are the ping times.

URLping vs fping

During this sample period of 54 hours, the maximum urlping response time was about 30 seconds. The maximum fping response time was 0.4857 seconds (or 485.7 ms). The average urlping response time was 2.75 seconds and the average fping response time was 0.06 seconds (or 60 ms). As you can see, the network ping response times are much lower than the webserver response times.

We found it very interesting that there was an elevation in webserver response times from about 15:45 on 9/23/09 until just after 01:30 on 9/24/09. Note that the ICMP ping times were not elevated in a similar manner during this period. Further investigation on this issue would be required.

I ran a correlation statistical analysis to see if the fping (icmp) response times and url ping times were related. The graph below has the ICMP ping time as the X-axis and the url ping time as the Y-axis. As you can see, there is very little correlation (0.251) between the two measurements.

URL pings vs ICMP pings correlation

Based on this information, I was able to convince the team that the webserver response is not related to the network response.

These tools are simple to use and should be in your toolkit as a network administrator. How often are you told that the network is having a problem yet you know that there is something else happening? Stay tuned… more to come!

So for now – share with the other ITKE readers ways that you look at network/application performance. What tools do you use to instrument and diagnose network issues?

Thanks for reading and let’s continue to be good network citizens.

Performance monitoring dashboard – designing and instrumentation

Troy Tate — Fri, 28 Aug 2009 16:57:38 +0000

One of my biggest challenges as a network manager is when users cry “the network is slow”. Some of you may have tools available to you where you can instantly dig in and see what the user might be seeing. There are some vendors out there with application and network monitoring tools. Netscout is one that comes to mind. However, I don’t have tools like that available so I have to work through several layers of data collection methods and tools to get a picture of what might be happening. Maybe you are in the same boat. Getting an answer to “the network is slow” is not a simple or quick activity. How do you deal with this? Following are some ways that I use to try and address the situation.

The number 1 and 2 tools that can find out if there is a potential problem are ping and pathping (Windows XP). To test a network path between a client and host for latency and packet loss use a command something like:

ping -n 100 -l 1000 host.com

The -l is a LOWER case L. This command will send 100 pings of 1000 bytes each to host.com. This will give you latency for larger packets and also if there is any packet loss along the path. To get more details about where along the path packet loss is happening, use the command:

pathping host.com

This also does some pings along the path but will inform you of where along a traceroute the pings are getting lost. Note that ICMP must be enabled along the path for these commands to give you the information you may need to resolve the problems.

If the network path and hosts are all on your private network, you may need to capture some additional performance data. This is where network diagrams and service diagrams come into play. In a typical webserver farm environment, there are potentially a lot of “moving” parts along the path between a client and an application. The application path between a client might go something like this:

client -> LAN switch -> router -> WAN link -> router -> LAN switch -> web server -> database/index server

And then return through the same or similar path. This example has 8-9 various connections/hosts that could impact client application performance. This is no longer just a network data traffic issue. I won’t be getting into the LAN switch or the router performance monitoring at this time. I will leave that for another posting. I will also make another entry discussing more about reviewing the server performance issues. Stay tuned.

So for now – share with the other ITKE readers ways that you look at network/application performance. What tools do you use to instrument and diagnose network issues?

Thanks for reading & let’s continue to be good network citizens.

Nmap v5 released – nearly 600 changes!

Troy Tate — Mon, 20 Jul 2009 18:05:07 +0000

Fyodor has announced the release of Nmap v5. This is the first major release since 1997. There are over 600 changes in the new version.

According to Wikipedia:

Nmap is a security scanner originally written by Gordon Lyon (also known by his pseudonym Fyodor Vaskovich).^[1] Nmap is a “Network Mapper”, used to discover computers and services on a computer network, thus creating a “map” of the network. Just like many simple port scanners, Nmap is capable of discovering passive services on a network despite the fact that such services aren’t advertising themselves with a service discovery protocol. In addition Nmap may be able to determine various details about the remote computers. These include operating system, device type, uptime, software product used to run a service, exact version number of that product, presence of some firewall techniques and, on a local area network, even vendor of the remote network card.

If you have not used Nmap before, you should become add it to your toolbox and become familiar with this extremely useful network administration and testing tool. Some of the additions in v5 include:

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:”";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:”Times New Roman”;
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:”Times New Roman”;
mso-bidi-theme-font:minor-bidi;}

1. Ncat included with Nmap. If you are familiar with netcat, then enough said. If not, ncat is a “reinvention” of the infamous netcat tool. Ncat is defined as the ‘swiss army knife’ for security testing and admin functions and can be considered a ‘piece of clay’ as you’ll find it’s range of uses is only limited by the user’s imagination and technical skills.

See http://nmap.org/ncat/

2. Ndiff scan comparison tool can be used to compare two Nmap XML files – in essence, you can scan a host today and scan it tomorrow and use Ndiff to compare the two to see differences in the results.

See http://nmap.org/ndiff/man.html

3. Performance enhancements have been made possible by the numerous scans Fyodor made of the internet last summer and finding the most commonly-open ports and reduce the number of ports scanned by default. In addition, you can define your own scan rate and bypass Nmaps congestion control algorithms.

4. The Nmap Scripting Engine (NSE) scripts have been improved and 32 new scripts added including scripts for MSRPC/NetBIOS atacks, queries and vulnerability probes, brute force attack scripts against SNMP and POP3 and more. NSE scripts/modules are defined at http://nmap.org/nsedoc/

5. The Nmap Book – this is a MUST HAVE for anyone involved in network troubleshooting or security! This is the best technical book that has come out in many years!

Get this right now at http://nmap.org/book/

I have a copy of the Nmap book that I ordered from Amazon. It is a great reference addition to your technical library and will be of great use. Nmap is a very technical tool but there are graphical interfaces for its use.

Let other ITKE members know how you use Nmap. Leave some tips/tricks here for our readers.

Thanks for reading and let’s continue to be good network citizens.