WAN

Sep 30 2009   1:36PM GMT

I resemble that award winning case study - wait, it IS me!

Posted by: Troy Tate
case study, WAN, frame relay, mpls, vpn, network management, industry award, ipsec, SSL, ssl vpn, information security, remote access, Security, security management

Have you ever wondered if vendor case studies are actually solutions to real life issues or if they are stories about compensated organizations using a particular vendor solution? Well, I am here to tell you that I know of at least one case study that is about an organization addressing real-life issues that was featured in an award winning vendor case study. The organization is the company I work for and the case study is about the challenges we faced with replacing an under-performing legacy Frame Relay network with a more efficient and flexible global solution that delivers high availability, remote access, and integrated security. For the record, no compensation was given for being the subject of this vendor case study.

The case study won the 2009 Best Deployment Scenario - VPN/IPSec/SSL and was featured in the Info Security Products Guide. The winning case study and announcement can be found at Manufacturing Company Achieves Security and Performance Goals with Virtela’s Remote Access Services from the Cloud.

See all 2009 Best Deployment Scenarios and Case Studies. This would be a good time to look at these and see if any of the solutions may meet some of the information security needs of your organization. Consider putting the solutions in your 2010 budgets.

Feel free to leave comments here or contact me through ITKE if you would like more information. Thanks for reading & let’s continue to be good network citizens.

Jul 29 2009   5:51PM GMT

Network Computing magazine is BACK! - WAN Optimization issue

Posted by: Troy Tate
Network Computing, online publication, wan optimization, WAN, optimization, online magazine

The Network Computing magazine was always one of my favorite trade publications. It covered a lot of very technical things from basic to advanced levels. I was very disappointed when they stopped publication. So, it is with a lot of excitement that I am sharing with ITKE members that Network Computing magazine is back! It is available online for your perusal. It is now a quarterly publication. The July 2009 issue focuses primarily on WAN optimization technologies.

Take a few minutes look it over. You may gain a better understanding of WAN optimization and how it might help your organization. Go to Network Computing for the July 2009 issue.

Thanks for reading & let’s continue to be good network citizens.

Mar 10 2009   4:47PM GMT

Saving Money & Stopping spam - change domain names

Posted by: Troy Tate
spam, email, domains, cost savings, cost reduction, WAN, internet, planning, operations

Are you getting lots of SPAM? Is your organization’s internet link being saturated due to tons of inbound spam and maybe outbound non-delivery notices for invalid addresses? About 3 years ago, ours was too. Continued »

Jan 29 2009   9:13PM GMT

Nifty tools for tracking down that “interesting” network traffic

Posted by: Troy Tate
pstools, Sysinternals, Microsoft, Routers, Cisco, troubleshooting, toolkit, Security, WAN, LAN, malware, network analysis, network monitor, network troubleshooting

My previous posting was meant to help you determine the source of potentially dangerous network traffic at your network’s edge. This post is meant to help you identify applications and traffic on your local network that seems to be “interesting”. I define “interesting” as something that you don’t know much about but would find it interesting to learn more about and maybe take some action to shutdown.

As you may already know, I work at an international company with sites around the globe. There are over 2500 computer nodes not including printers, servers, switches, etc. Sometimes it is necessary to identify what traffic is crossing the network links between the sites. There are lots of tools and processes that can be used to gather this information. I will outline a couple here.

Our WAN edge routers are from Cisco. One of the features that can be enabled on a Cisco router is the ip cache flow feature. The show ip cache flow command returns some very useful information. An example is shown below.

show ip cache flow

IP packet size distribution (116972772 total packets):

1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480

.000 .375 .090 .023 .010 .007 .006 .003 .002 .014 .011 .010 .009 .005 .004

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608

.004 .003 .006 .028 .378 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes

64 active, 4032 inactive, 4367569 added

80215342 ager polls, 0 flow alloc failures

Active flows timeout in 30 minutes

Inactive flows timeout in 15 seconds

IP Sub Flow Cache, 21640 bytes

0 active, 1024 inactive, 0 added, 0 added to flow

0 alloc failures, 0 force free

1 chunk, 1 chunk added

last clearing of statistics never

Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)

--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow

TCP-Telnet         724      0.0         7   430      0.0       6.1      15.4

TCP-FTP          13859      0.0         9    93      0.0       6.7       3.4

TCP-WWW        3537205      0.8        14  1021     12.2       3.7       9.7

TCP-SMTP           290      0.0       104   989      0.0       5.5       1.8

TCP-X                3      0.0         2    42      0.0       0.3       1.3

TCP-BGP             18      0.0         1    43      0.0       0.0      13.9

TCP-Frag           112      0.0        37    78      0.0      18.3      15.5

TCP-other       684674      0.1        12   831      2.0       6.4       7.0

UDP-DNS           1973      0.0         1    72      0.0       0.1      15.4

UDP-NTP            248      0.0         1    77      0.0       0.0      15.4

UDP-Frag             3      0.0         1    45      0.0       0.0      15.6

UDP-other        10247      0.0         1   210      0.0       0.8      15.4

ICMP             97640      0.0        19    83      0.4      18.6      15.4

GRE              20509      0.0      2598   150     12.4     165.6      14.5

Total:         4367505      1.0        26   593     27.2       5.2       9.4

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts

Tu0           10.aa.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.cc.12.200    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.dd.12.8      Fa0/0         10.bb.12.150    06 0D0A 0871   467

Tu0           10.ee.12.200    Fa0/0         10.bb.ee.140    06 0A23 01BD     1

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.130    06 048A 07DA     1

Tu0           10.gg.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.hh.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.11     06 048A 04A7     1

Tu0           10.oo.12.210    Fa0/0         10.bb.12.200    11 0035 EA0B     1

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055D     5

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055E    10

As you can see it includes statistics about the packet size distribution, the various protocols and amount of traffic for each protocol and then a summary listing of the traffic through the various interfaces on the router. In this case, the traffic is passing through a couple of encrypted tunnel interfaces. This is where things get interesting when troubleshooting traffic on a link. The first column is the source interface, then the source IP address. The third column is the destination interface followed by the destination IP address. The next 3 columns give some critical information about the traffic between the source and destination hosts. These values are all given in HEX. There is the protocol number (e.g. 01 - ICMP, 06 - TCP, 11 - UDP). See the protocol listing at IANA for more information on these numbers - remember to convert from HEX to decimal.

The next two columns are the source port and destination port pairing. These values are also in HEX. So, converting values like 01BD to 445 indicates that the traffic is Microsoft DS according to the port number listing at IANA. Port 0035 (53 decimal) would be DNS traffic. Port 0050 (80 decimal) would be http traffic. Port 01BB (443 decimal) would be https. So, as you can see, lots of information is right there on the router and no sniffing is required to see what traffic is on your network.

Once you find an “interesting” source and destination pair that concerns you, you might consider finding out what application is generating the traffic between that source / destination pair. This can be done unobtrusively using some of the excellent tools from the Microsoft/Sysinternals toolkit. For example, the following command will list the current tcp & udp connections on a remote computer (10.xx.50.81) - note that you must have administrative access to the remote computer to run this command (netstat is not a Sysinternals tool but is built into the Windows operating system):

psexec \\10.xx.50.81 netstat -ano

The output would look something like this:

PsExec v1.94 - Execute processes remotely

Copyright (C) 2001-2008 Mark Russinovich

Sysinternals - www.sysinternals.com

Active Connections

Proto Local Address Foreign Address State PID

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 852

TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4

TCP 0.0.0.0:5800 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:8085 0.0.0.0:0 LISTENING 1456

TCP 10.xx.50.81:139 0.0.0.0:0 LISTENING 4

TCP 10.xx.50.81:445 10.bb.50.64:1826 ESTABLISHED 4

TCP 10.xx.50.81:1221 10.xx.12.200:135 ESTABLISHED 608

TCP 10.xx.50.81:1222 10.xx.12.200:1026 ESTABLISHED 608

TCP 10.xx.50.81:1822 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1823 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1827 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1828 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1829 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1830 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1831 10.xx.50.241:8080 ESTABLISHED 3756

TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING 2412

UDP 0.0.0.0:445 *:* 4

UDP 0.0.0.0:500 *:* 608

netstat exited on 10.xx.50.81 with error code 0.

So, these results show that the host has various tcp & udp connections that are in an established state. It shows the source & destination ports (again like the show ip cache flow results). The other very useful piece of information that is shown is the PID or process identifier. This number matches a process running on the remote computer. So, to find out what the various running processes are and their PID’s, run the following command:

pslist \\10.xx.50.81

The results returned are like the following:

pslist v1.28 - Sysinternals PsList

Copyright ¬ 2000-2004 Mark Russinovich

Sysinternals

Process information for 10.xx.50.81:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time

Idle 0 0 1 0 0 0:37:20.984 0:00:00.000

System 4 8 67 316 0 0:00:48.343 0:00:00.000

smss 464 11 3 21 164 0:00:00.015 4:43:15.698

csrss 528 13 15 545 2520 0:00:13.484 4:43:14.792

winlogon 552 13 19 524 9488 0:00:04.265 4:43:14.370

services 596 9 16 295 1876 0:00:04.281 4:43:14.183

lsass 608 9 20 428 4160 0:00:02.843 4:43:14.167

svchost 792 8 17 193 3284 0:00:00.796 4:43:13.667

svchost 852 8 10 371 2144 0:00:35.421 4:43:13.370

svchost 916 8 70 2092 16500 0:00:54.359 4:43:13.292

svchost 968 8 6 84 1596 0:00:00.921 4:43:13.245

svchost 992 8 15 292 3044 0:00:00.843 4:43:12.714

spoolsv 1196 8 12 142 3492 0:00:00.296 4:43:12.277

stormliv 1324 8 9 163 4952 0:00:08.343 4:43:04.339

EngineServer 1444 8 3 35 576 0:00:00.078 4:43:03.995

FrameworkService 1456 8 21 356 20632 0:00:37.203 4:43:03.573

VsTskMgr 1504 8 19 243 7128 0:00:29.578 4:43:02.714

MDM 1556 8 4 86 1092 0:00:00.140 4:43:02.495

mfevtps 1580 8 6 126 6848 0:00:02.609 4:43:02.370

ArchivingORBService 1636 8 4 88 3304 0:00:15.031 4:43:01.964

svchost 1696 8 5 118 2608 0:00:00.453 4:43:01.777

CcmExec 1836 8 13 810 14688 0:00:12.796 4:43:01.214

Mcshield 1880 13 26 182 45316 0:02:15.078 4:42:59.464

naPrdMgr 1964 8 6 130 208448 0:01:05.328 4:42:57.902

mfeann 1968 8 8 151 2264 0:00:01.625 4:42:57.855

alg 2412 8 5 102 1256 0:00:00.109 4:42:17.303

wmiprvse 2876 8 4 140 4132 0:00:00.781 4:42:09.979

wmiprvse 2660 8 7 146 1996 0:00:00.828 4:39:42.549

explorer 3676 8 12 442 17392 0:01:01.828 3:59:34.124

hkcmd 4092 8 2 86 896 0:00:00.140 3:59:30.406

igfxpers 816 8 3 93 868 0:00:00.078 3:59:30.343

UdaterUI 3388 8 5 115 1648 0:00:00.859 3:59:27.390

shstat 3252 8 10 98 2160 0:00:00.812 3:59:27.093

ctfmon 3968 8 1 67 984 0:00:00.156 3:59:25.828

Then if we need to remotely stop a running process that we consider suspicious or “interesting” issue the following command:

pskill 3968 \\10.xx.50.81 - note you can use either the PID # or the name of the process - however, you should use the PID if there are multiple instances of the application running

The results of the command, if successful, should look like:

PsKill v1.12 - Terminates processes on local or remote systems

Copyright (C) 1999-2005 Mark Russinovich

Sysinternals - www.sysinternals.com

Process 3968 on 10.xx.50.81 killed….

This process has become very useful when finding some rogue processes (malware) on some remote computers and there is no other way to disable the system or application. You can also issue a psshutdown command in a similar fashion, but the user may attempt to restart the machine again and then you will have to again shutdown the rogue application. There’s lots of ways to handle this situation including shutting down the LAN switch port if you have that access and privilege.

Let me know what processes you go through when managing remote systems where you may have limited physical access. Good luck out there and let’s be good network citizens!

Nov 11 2008   4:07PM GMT

Did you see this? - Microsoft SharePoint Toolkit

Posted by: Troy Tate
administration, Networking, tools, Microsoft Windows, Monitoring, Development, reporting, internet, WAN, LAN, debugging, performance monitoring, SharePoint, design, MOSS, troubleshooting, Performance, howto, network analysis, Metrics, awareness, diagnostics, toolkit, analysis

Many organizations are finding value in the Microsoft SharePoint technologies. Whether you use the free Windows SharePoint Services or the Microsoft Office SharePoint Server, your organization will gain a lot of value from using these services. To enhance your ability to manage these technologies, there is a project on Codeplex called the SharePoint Toolbox. Per the website, the purpose of this project is as follows:

This project includes powerful and useful tools and add-ons for SharePoint that help developers and IT pros implement SharePoint based solutions more quickly and managed them more effectively. Contributions will come from the Microsoft SharePoint Product Group, Microsoft SharePoint Online Services Group, Microsoft Information Technology Group, and Microsoft Consulting Services Group.

I have personally used the CopyTimer utility  to measure throughput from remote sites to a SharePoint server. It worked well and helped gather some excellent data about the site and global network performance.

Enjoy using these tools and give me some feedback on what you find useful and how SharePoint provides value to your organization.

Oct 27 2008   8:52PM GMT

Did you see this? - (Wire)Sharkfest 2008 videos - including Vint Cerf - now available

Posted by: Troy Tate
Networking, forensics, Security, tools, Microsoft Windows, Linux, Monitoring, web, reporting, Google, internet, IT education, WAN, LAN, performance monitoring, troubleshooting, Performance, Network TAPs, howto, network analysis, Metrics, wireshark, packet capture, research, education, toolkit, man-in-the-middle, analysis

Checkout the Sharkfest 2008 videos at LoveMyTool.com. If you use Wireshark or want to learn network troubleshooting, this is one of the best resources you can have in your toolkit. The videos will give you a better understanding of this tool and other tools out there.

There is even a video of Dr. Vinton G. Cerf, vice president and Chief Internet Evangelist for Google. He is responsible for identifying new enabling technologies and applications on the Internet and other platforms for the company. Widely known as a “Father of the Internet,” Vint is the co-designer with Robert Kahn of TCP/IP protocols and basic architecture of the Internet.

Have a great day and thanks for stopping by!

Oct 9 2008   3:56PM GMT

Virtual Enterprise VOIP panel discussion

Posted by: Troy Tate
administration, Networking, Cisco, Monitoring, VoIP, unified communications, IP telephony, DataCenter, IT education, WAN, LAN, PSTN, design, howto, risk, education

As you may have already read, I will not be attending the Enterprise VOIP event at CampIT Conferences in Chicago on 10/14. Well, I thought I would bring my portion of the discussion to you in this virtual panel discussion and maybe you and I both can gain some from this forum.

Some background on our environment: IP phone population - over 400, distributed at 4 sites, largest ~150, smallest 60; all Cisco

Why implement VOIP?

• greenfield site - needed a phone system and VOIP made sense for a new site install to position for future

• acquired company in process of implementing VOIP - came into a situation where an acquisition had purchased VOIP and I became owner of the implementation; had issues with chosen vendor and equipment lists; eventually came out successful but was not without its pain during implementation.

• forward looking strategy - setup the company to have regional communication hubs for IP telephony; we have VOIP in North America, Europe and Asia now; this could permit us to leverage our WAN for toll bypass provided we build other local site infrastructure to support this technology.

Our biggest challenges:

• users: they find the phones easy to use and very good features; however, there are some features like managing meet-me conference calling that they feel are too onerous so don’t take the time to use this cost-saving feature

• administrators: setting up phones is an infrequent event so it is not a real simple task to setup a new phone; moves are made easier than traditional systems; troubleshooting skills are different since voice now is carried over the data network until it reaches a PSTN gateway

Best features:

• dial another site using extensions rather than 10 digit or more dialing

• “on phone” directory - can lookup another IP phone user’s extension directly on the phone rather than finding them on a piece of paper or website somewhere

• easier conference calling than old system

• mobile-phone like features: listing missed calls; call history log

• moves are made easier; adds are a challenge since done infrequently

Desires for additional features/services:

• video

• more ringtones (must have been someone young and a heavy cell phone user)

• integration with e-mail/web

What are the risks?

• it’s challenging to implement in an “old school” infrastructure environment (flat network, no-vlans, hubs still in use, etc.) It takes lots of forethought and understanding VLAN’s, WAN links, need to update staff skills.

• The network MUST be reliable or voice will suffer. Traditional phone companies have had 100+ years to make a bulletproof network.

• Costs. It’s not cheap to implement this technology. You have to weigh the ability of the organization to support non-industry leading implementations versus choosing the best technology you can afford.

• Maintenance. Upgrading the software in the servers, gateways and phones is much riskier than upgrading a traditional PBX environment.

What are the rewards?

• It works!

• It positions the organization to take advantage of other services provided that it is not simply an IT-led project but meets business requirements.

Feel free to add comments on your own experiences, concerns. This is a great forum and keep up the good work of information sharing!

Oct 3 2008   7:59PM GMT

Did you see this? - Open Source Tools University

Posted by: Troy Tate
administration, Networking, Firewalls, forensics, Security, tools, Monitoring, reporting, internet, IT education, WAN, LAN, debugging, Data security, SSL, performance monitoring, blogging, design, anti-virus, troubleshooting, Performance, howto, network analysis, Sandbox, Metrics, wireshark, packet capture, research, blog, podcast, diagnostics, toolkit, analysis

If you are like me, you like those little goodie tools like nmap and wireshark that do something that is actually pretty complex but do it well and have a great following. I just came across this website that I am going to have to take some time to go through and find all of the nuggets it offers. Hope you get some use out of it too and let us know what you discover and how it made your job easier.

LoveMyTool

There are presentations on this site like the Wireshark IO Graph for Response Time Analysis (by Ray Tompkins).This should be a great online learning experience. You will find contributors like Sake Blok, a Wireshark Core Developer and Denny K Miu of StartupforLess.org - A Survival Guide for Bootstrapping Entrepreneurs

Sep 30 2008   1:34PM GMT

Did you see this? - Laura Chappell’s Troubleshooting & Security Summit

Posted by: Troy Tate
Networking, forensics, Security, tools, Monitoring, reporting, DataManagement, WAN, LAN, Data security, malware, SSL, performance monitoring, troubleshooting, honeypot, Performance, Network TAPs, howto, network analysis, Metrics, wireshark, risk, packet capture, research, awareness, education, toolkit

Maybe you already know Laura Chappell (The Viral Bitgirl), if not then this is your chance to meet her and gain loads of knowledge in 2 days.

On November 4-5, 2008 - Las Colinas, TX (near Dallas-Ft Worth airport) Laura will be holding a Troubleshooting and Security Summit.

In two full days you will walk away with more security, optimization and troubleshooting knowledge than you’d get after spending months in the field figuring this out.

Learn the best practices and most efficient tools to use to analyze wired and wireless network performance to optimize and secure network communications from Laura Chappell, Founder of Wireshark University and Protocol Analysis Institute. See the Summit 08 special pricing and group discount information below. Register today at www.chappellsummit.com.

Key points include:
* TCP Enhancements in Vista/Server 2008
* Faster File Transfers with SMBv1 vs. SMBv2
* Traffic Analysis between Virtualized Hosts
* Proven Techniques to Baseline the Network
* Latency Chokepoints
* Automatic Traffic Capture and Analysis
* Network Security and Forensics Procedures
* Key Points to Deploying Decoys
* Suspicious Traffic Signatures
* Handling Traffic Evidence

Bring Your Own Laptop (BYOL) Format
This hands-on lab-based course offers a series of demonstrations and individual hands-on labs to rapidly improve and expand your skill set. You will leave with your laptop loaded with tools, trace files and configured to improve network performance and security immediately after class.

GUEST SPEAKERS
*Gerald Combs, Creator of Wireshark - Must-Know Steps to Analyzing Virtualized Communications and the Future of Wireshark

* Tom Quilty, Cybercrime Investigator for BD Consulting and Investigation - Preparing for and Handling a Data Breach or Theft

Register Today - Seating is Limited
Register online at www.chappellsummit.com. Registration $1,295 - Early Bird $995 (ends midnight PDT Tuesday 9/30/08)

Group Discounts: Bring in two or more people from your company and receive $100 off each additional registration. Contact Brenda Czech at +1 408-378-7841 for more details.

Wireshark University Savings: Attendees receive the Wireshark University WSU03 Troubleshooting Network Communications self-paced course free with the student kits. Registered attendees also receive a 50%-off coupon on Wireshark University Self-Paced Courses.

Register today.
www.chappellsummit.com

If you go, please share some of the tips and tricks you gained with the ITKE population. Help spread the word!

Sep 19 2008   1:16PM GMT

Crunching numbers - is this any way to manage a network?

Posted by: Troy Tate
administration, tools, Monitoring, reporting, DataManagement, WAN, performance monitoring, Performance, howto, network analysis, Metrics, facility, toolkit, facility management

I just got done catching up crunching wide area network usage statistics for the last 6 months. Wow… what a job! I should be doing it at the end of each month but I got behind due to other major activities like moving a data center and implementing a new e-mail system for >2000 users. Those kind of major activities seem to take over the day so routine items sometimes get left behind.

Getting back to the WAN statistics. I download usage stats daily. The stats are in 10 minute increments. So, I get really good detailed information about utilization at the sites. Well, 10 minute stats over a 24 hour period is about 144 data points per day per site (actually, multiply X2 since there are stats for inbound AND outbound usage). Since this is such a large volume of data, I distill it down to the busy business hours of 7AM to 7PM local site time, Monday through Friday. For a regular month, this may give me around 1600 data points each for inbound and outbound. I also have to do some work in converting the dates/times from the vendor reports to Excel-friendly format.

I take these data points and run them through Excel performing some frequency plots and trend analysis. This gives me an idea of utilization at the site during the past month and possible trends for the future month. As you can see, this is a labor-intensive activity. I don’t know of another way of  getting this information given the current toolset I have available. Do any of you have a similar challenge? How do you address it? I do think the task is worth the effort since a global WAN is a significant monthly expense.

As always, thanks for checking out my blog. Let’s be good network citizens together & practice safe networking!