As you may already know, I work at an international company with sites around the globe. There are over 2500 computer nodes not including printers, servers, switches, etc. Sometimes it is necessary to identify what traffic is crossing the network links between the sites. There are lots of tools and processes that can be used to gather this information. I will outline a couple here.

Our WAN edge routers are from Cisco. One of the features that can be enabled on a Cisco router is the ip cache flow feature. The show ip cache flow command returns some very useful information. An example is shown below.

show ip cache flow

IP packet size distribution (116972772 total packets):

1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480

.000 .375 .090 .023 .010 .007 .006 .003 .002 .014 .011 .010 .009 .005 .004

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608

.004 .003 .006 .028 .378 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes

64 active, 4032 inactive, 4367569 added

80215342 ager polls, 0 flow alloc failures

Active flows timeout in 30 minutes

Inactive flows timeout in 15 seconds

IP Sub Flow Cache, 21640 bytes

0 active, 1024 inactive, 0 added, 0 added to flow

0 alloc failures, 0 force free

1 chunk, 1 chunk added

last clearing of statistics never

Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)

--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow

TCP-Telnet         724      0.0         7   430      0.0       6.1      15.4

TCP-FTP          13859      0.0         9    93      0.0       6.7       3.4

TCP-WWW        3537205      0.8        14  1021     12.2       3.7       9.7

TCP-SMTP           290      0.0       104   989      0.0       5.5       1.8

TCP-X                3      0.0         2    42      0.0       0.3       1.3

TCP-BGP             18      0.0         1    43      0.0       0.0      13.9

TCP-Frag           112      0.0        37    78      0.0      18.3      15.5

TCP-other       684674      0.1        12   831      2.0       6.4       7.0

UDP-DNS           1973      0.0         1    72      0.0       0.1      15.4

UDP-NTP            248      0.0         1    77      0.0       0.0      15.4

UDP-Frag             3      0.0         1    45      0.0       0.0      15.6

UDP-other        10247      0.0         1   210      0.0       0.8      15.4

ICMP             97640      0.0        19    83      0.4      18.6      15.4

GRE              20509      0.0      2598   150     12.4     165.6      14.5

Total:         4367505      1.0        26   593     27.2       5.2       9.4

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts

Tu0           10.aa.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.cc.12.200    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.dd.12.8      Fa0/0         10.bb.12.150    06 0D0A 0871   467

Tu0           10.ee.12.200    Fa0/0         10.bb.ee.140    06 0A23 01BD     1

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.130    06 048A 07DA     1

Tu0           10.gg.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.hh.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.11     06 048A 04A7     1

Tu0           10.oo.12.210    Fa0/0         10.bb.12.200    11 0035 EA0B     1

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055D     5

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055E    10

As you can see it includes statistics about the packet size distribution, the various protocols and amount of traffic for each protocol and then a summary listing of the traffic through the various interfaces on the router. In this case, the traffic is passing through a couple of encrypted tunnel interfaces. This is where things get interesting when troubleshooting traffic on a link. The first column is the source interface, then the source IP address. The third column is the destination interface followed by the destination IP address. The next 3 columns give some critical information about the traffic between the source and destination hosts. These values are all given in HEX. There is the protocol number (e.g. 01 – ICMP, 06 – TCP, 11 – UDP). See the protocol listing at IANA for more information on these numbers – remember to convert from HEX to decimal.

The next two columns are the source port and destination port pairing. These values are also in HEX. So, converting values like 01BD to 445 indicates that the traffic is Microsoft DS according to the port number listing at IANA. Port 0035 (53 decimal) would be DNS traffic. Port 0050 (80 decimal) would be http traffic. Port 01BB (443 decimal) would be https. So, as you can see, lots of information is right there on the router and no sniffing is required to see what traffic is on your network.

Once you find an “interesting” source and destination pair that concerns you, you might consider finding out what application is generating the traffic between that source / destination pair. This can be done unobtrusively using some of the excellent tools from the Microsoft/Sysinternals toolkit. For example, the following command will list the current tcp & udp connections on a remote computer (10.xx.50.81) – note that you must have administrative access to the remote computer to run this command (netstat is not a Sysinternals tool but is built into the Windows operating system):

psexec \\10.xx.50.81 netstat -ano

The output would look something like this:

PsExec v1.94 – Execute processes remotely

Copyright (C) 2001-2008 Mark Russinovich

Sysinternals – www.sysinternals.com

Active Connections

Proto Local Address Foreign Address State PID

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 852

TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4

TCP 0.0.0.0:5800 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:8085 0.0.0.0:0 LISTENING 1456

TCP 10.xx.50.81:139 0.0.0.0:0 LISTENING 4

TCP 10.xx.50.81:445 10.bb.50.64:1826 ESTABLISHED 4

TCP 10.xx.50.81:1221 10.xx.12.200:135 ESTABLISHED 608

TCP 10.xx.50.81:1222 10.xx.12.200:1026 ESTABLISHED 608

TCP 10.xx.50.81:1822 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1823 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1827 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1828 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1829 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1830 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1831 10.xx.50.241:8080 ESTABLISHED 3756

TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING 2412

UDP 0.0.0.0:445 *:* 4

UDP 0.0.0.0:500 *:* 608

netstat exited on 10.xx.50.81 with error code 0.

So, these results show that the host has various tcp & udp connections that are in an established state. It shows the source & destination ports (again like the show ip cache flow results). The other very useful piece of information that is shown is the PID or process identifier. This number matches a process running on the remote computer. So, to find out what the various running processes are and their PID’s, run the following command:

pslist \\10.xx.50.81

The results returned are like the following:

pslist v1.28 – Sysinternals PsList

Copyright ¬ 2000-2004 Mark Russinovich

Sysinternals

Process information for 10.xx.50.81:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time

Idle 0 0 1 0 0 0:37:20.984 0:00:00.000

System 4 8 67 316 0 0:00:48.343 0:00:00.000

smss 464 11 3 21 164 0:00:00.015 4:43:15.698

csrss 528 13 15 545 2520 0:00:13.484 4:43:14.792

winlogon 552 13 19 524 9488 0:00:04.265 4:43:14.370

services 596 9 16 295 1876 0:00:04.281 4:43:14.183

lsass 608 9 20 428 4160 0:00:02.843 4:43:14.167

svchost 792 8 17 193 3284 0:00:00.796 4:43:13.667

svchost 852 8 10 371 2144 0:00:35.421 4:43:13.370

svchost 916 8 70 2092 16500 0:00:54.359 4:43:13.292

svchost 968 8 6 84 1596 0:00:00.921 4:43:13.245

svchost 992 8 15 292 3044 0:00:00.843 4:43:12.714

spoolsv 1196 8 12 142 3492 0:00:00.296 4:43:12.277

stormliv 1324 8 9 163 4952 0:00:08.343 4:43:04.339

EngineServer 1444 8 3 35 576 0:00:00.078 4:43:03.995

FrameworkService 1456 8 21 356 20632 0:00:37.203 4:43:03.573

VsTskMgr 1504 8 19 243 7128 0:00:29.578 4:43:02.714

MDM 1556 8 4 86 1092 0:00:00.140 4:43:02.495

mfevtps 1580 8 6 126 6848 0:00:02.609 4:43:02.370

ArchivingORBService 1636 8 4 88 3304 0:00:15.031 4:43:01.964

svchost 1696 8 5 118 2608 0:00:00.453 4:43:01.777

CcmExec 1836 8 13 810 14688 0:00:12.796 4:43:01.214

Mcshield 1880 13 26 182 45316 0:02:15.078 4:42:59.464

naPrdMgr 1964 8 6 130 208448 0:01:05.328 4:42:57.902

mfeann 1968 8 8 151 2264 0:00:01.625 4:42:57.855

alg 2412 8 5 102 1256 0:00:00.109 4:42:17.303

wmiprvse 2876 8 4 140 4132 0:00:00.781 4:42:09.979

wmiprvse 2660 8 7 146 1996 0:00:00.828 4:39:42.549

explorer 3676 8 12 442 17392 0:01:01.828 3:59:34.124

hkcmd 4092 8 2 86 896 0:00:00.140 3:59:30.406

igfxpers 816 8 3 93 868 0:00:00.078 3:59:30.343

UdaterUI 3388 8 5 115 1648 0:00:00.859 3:59:27.390

shstat 3252 8 10 98 2160 0:00:00.812 3:59:27.093

ctfmon 3968 8 1 67 984 0:00:00.156 3:59:25.828

Then if we need to remotely stop a running process that we consider suspicious or “interesting” issue the following command:

pskill 3968 \\10.xx.50.81 – note you can use either the PID # or the name of the process – however, you should use the PID if there are multiple instances of the application running

The results of the command, if successful, should look like:

PsKill v1.12 – Terminates processes on local or remote systems

Copyright (C) 1999-2005 Mark Russinovich

Sysinternals – www.sysinternals.com

Process 3968 on 10.xx.50.81 killed….

This process has become very useful when finding some rogue processes (malware) on some remote computers and there is no other way to disable the system or application. You can also issue a psshutdown command in a similar fashion, but the user may attempt to restart the machine again and then you will have to again shutdown the rogue application. There’s lots of ways to handle this situation including shutting down the LAN switch port if you have that access and privilege.

Let me know what processes you go through when managing remote systems where you may have limited physical access. Good luck out there and let’s be good network citizens!

This project includes powerful and useful tools and add-ons for SharePoint that help developers and IT pros implement SharePoint based solutions more quickly and managed them more effectively. Contributions will come from the Microsoft SharePoint Product Group, Microsoft SharePoint Online Services Group, Microsoft Information Technology Group, and Microsoft Consulting Services Group.

I have personally used the CopyTimer utility  to measure throughput from remote sites to a SharePoint server. It worked well and helped gather some excellent data about the site and global network performance.

Enjoy using these tools and give me some feedback on what you find useful and how SharePoint provides value to your organization.

There is even a video of Dr. Vinton G. Cerf, vice president and Chief Internet Evangelist for Google. He is responsible for identifying new enabling technologies and applications on the Internet and other platforms for the company. Widely known as a “Father of the Internet,” Vint is the co-designer with Robert Kahn of TCP/IP protocols and basic architecture of the Internet.

Have a great day and thanks for stopping by!

Some background on our environment: IP phone population – over 400, distributed at 4 sites, largest ~150, smallest 60; all Cisco

Why implement VOIP?

• greenfield site – needed a phone system and VOIP made sense for a new site install to position for future

• acquired company in process of implementing VOIP – came into a situation where an acquisition had purchased VOIP and I became owner of the implementation; had issues with chosen vendor and equipment lists; eventually came out successful but was not without its pain during implementation.

• forward looking strategy – setup the company to have regional communication hubs for IP telephony; we have VOIP in North America, Europe and Asia now; this could permit us to leverage our WAN for toll bypass provided we build other local site infrastructure to support this technology.

Our biggest challenges:

• users: they find the phones easy to use and very good features; however, there are some features like managing meet-me conference calling that they feel are too onerous so don’t take the time to use this cost-saving feature

• administrators: setting up phones is an infrequent event so it is not a real simple task to setup a new phone; moves are made easier than traditional systems; troubleshooting skills are different since voice now is carried over the data network until it reaches a PSTN gateway

Best features:

• dial another site using extensions rather than 10 digit or more dialing

• “on phone” directory – can lookup another IP phone user’s extension directly on the phone rather than finding them on a piece of paper or website somewhere

• easier conference calling than old system

• mobile-phone like features: listing missed calls; call history log

• moves are made easier; adds are a challenge since done infrequently

Desires for additional features/services:

• video

• more ringtones (must have been someone young and a heavy cell phone user)

• integration with e-mail/web

What are the risks?

• it’s challenging to implement in an “old school” infrastructure environment (flat network, no-vlans, hubs still in use, etc.) It takes lots of forethought and understanding VLAN’s, WAN links, need to update staff skills.

• The network MUST be reliable or voice will suffer. Traditional phone companies have had 100+ years to make a bulletproof network.

• Costs. It’s not cheap to implement this technology. You have to weigh the ability of the organization to support non-industry leading implementations versus choosing the best technology you can afford.

• Maintenance. Upgrading the software in the servers, gateways and phones is much riskier than upgrading a traditional PBX environment.

What are the rewards?

• It works!

• It positions the organization to take advantage of other services provided that it is not simply an IT-led project but meets business requirements.

Feel free to add comments on your own experiences, concerns. This is a great forum and keep up the good work of information sharing!

Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity. It combines the features of two legacy Sysinternals utilities, Filemon and Regmon, and adds an extensive list of enhancements including rich and non-destructive filtering, comprehensive event properties such session IDs and user names, reliable process information, full thread stacks with integrated symbol support for each operation, simultaneous logging to a file, and much more. Its uniquely powerful features will make Process Monitor a core utility in your system troubleshooting and malware hunting toolkit.

Process Monitor runs on Windows 2000 SP4 with Update Rollup 1, Windows XP SP2, Windows Server 2003 SP1, and Windows Vista as well as x64 versions of Windows XP, Windows Server 2003 SP1 and Windows Vista.

I had previously talked about the Sysinternals Live website. This update to one of the excellent tools is well worth your time in investigating. Take a look at the updated tool here. The entire Sysinternals toolset can be found here.

If you have not used these tools yet, then you are definitely missing a critical item for being successful in your IT position. Check them out… it may save your reputation some time!

LoveMyTool

There are presentations on this site like the Wireshark IO Graph for Response Time Analysis (by Ray Tompkins).This should be a great online learning experience. You will find contributors like Sake Blok, a Wireshark Core Developer and Denny K Miu of StartupforLess.org – A Survival Guide for Bootstrapping Entrepreneurs

On November 4-5, 2008 – Las Colinas, TX (near Dallas-Ft Worth airport) Laura will be holding a Troubleshooting and Security Summit.

In two full days you will walk away with more security, optimization and troubleshooting knowledge than you’d get after spending months in the field figuring this out.

Learn the best practices and most efficient tools to use to analyze wired and wireless network performance to optimize and secure network communications from Laura Chappell, Founder of Wireshark University and Protocol Analysis Institute. See the Summit 08 special pricing and group discount information below. Register today at www.chappellsummit.com.

Key points include:
* TCP Enhancements in Vista/Server 2008
* Faster File Transfers with SMBv1 vs. SMBv2
* Traffic Analysis between Virtualized Hosts
* Proven Techniques to Baseline the Network
* Latency Chokepoints
* Automatic Traffic Capture and Analysis
* Network Security and Forensics Procedures
* Key Points to Deploying Decoys
* Suspicious Traffic Signatures
* Handling Traffic Evidence

Bring Your Own Laptop (BYOL) Format
This hands-on lab-based course offers a series of demonstrations and individual hands-on labs to rapidly improve and expand your skill set. You will leave with your laptop loaded with tools, trace files and configured to improve network performance and security immediately after class.

GUEST SPEAKERS
*Gerald Combs, Creator of Wireshark – Must-Know Steps to Analyzing Virtualized Communications and the Future of Wireshark

* Tom Quilty, Cybercrime Investigator for BD Consulting and Investigation – Preparing for and Handling a Data Breach or Theft

Register Today – Seating is Limited
Register online at www.chappellsummit.com. Registration $1,295 – Early Bird $995 (ends midnight PDT Tuesday 9/30/08)

Group Discounts: Bring in two or more people from your company and receive $100 off each additional registration. Contact Brenda Czech at +1 408-378-7841 for more details.

Wireshark University Savings: Attendees receive the Wireshark University WSU03 Troubleshooting Network Communications self-paced course free with the student kits. Registered attendees also receive a 50%-off coupon on Wireshark University Self-Paced Courses.

Register today.
www.chappellsummit.com

If you go, please share some of the tips and tricks you gained with the ITKE population. Help spread the word!

Check out the TCP/IP tutorial. It’s a quick easy read that you can share with those you are trying to explain how a network works.

Checkout this website. I guarantee that if networks are in your blood, you will find something of interest here.

Thanks for your time. Let’s be good network citizens together & practice safe networking!

This event had been in the planning and implementation stages for months. The building had to be outfitted for occupancy and services had to be ready for use on day one.

Fortunately we had implemented a Cisco CallManager solution at the old facility and it was easy enough to move into the new facility. However, some challenges existed with that implementation. First of all, our original implementation was not a full CallManager implementation. It was a Survivable Remote System Telephony (SRST) implementation. The actual CallManager cluster for this site is located in southern California. This site is in northern California, several hundred miles away.

Another issue was that the site was experiencing growth due to the merged office. Our current gateway solution was not large enough to handle the additional handsets. We implemented a larger gateway with capacity to handle the current user population plus some additional growth.

I say “We” because we had a partner helping us with the implementation. This is not something we have in our staff skillset. Our partner helped us with the original implementation at the site. They provided outstanding support and were available to help with other network issues as they arose during the move and day one of business. I was able to breathe a sigh of relief with this partner onsite.

Our servers moved over without a glitch. Our structured cabling plant looks really nice (for the moment, I’m sure) . Our users enjoy the new facility. So, it’s a win-win!

My only real issue happened unfortunately on day one of business in the new facility. For some reason, both the primary and secondary network links went down. This was definitely not a good thing especially since the site used the remote CallManager to manage calls. During this period, the WAN link was unreliable and calls were dropped and phones reset. Another item that you should note is that there was another tenant moving into the building next door. My company and this other shared a telecom closet where all communication circuits entered the facilities. I have major concerns about cross-connects as well as the danger of miscommunication with the carriers about circuits being terminated or orders placed.

My WAN provider went right to work though and had the carriers investigating why both our primary (T1) link and backup (DSL) link went down. The circuits are provided by different carriers so they could work unrestricted on their issues while we waited for resolution. The T1 issue was due to some problem at the carrier central office (hmmmm…. likely story since I had a new neighbor moving in at the same time.) The DSL issue turned out to be some kind of problem with inside wiring.

Our circuits became stable later on day one and have been stable ever since. I’m really glad the event is over! Now we are planning for a new site implementation coming online in January and the facility has not even been built yet. The new facility is not even in the USA so this will be even more of a challenge.

Thanks for your time. Let’s be good network citizens together & practice safe networking!