• In 2007 there were 500 million connected devices or 1/10th of a connected device per person worldwide. In 2010, there are now 35 billion (5 connected devices per person). In 2013, Forester Research projects that there will be 1 trillion (140 per person) connected devices.
• In 2007 there were about 3000 total mobile applications. In 2010, there are 265,000 mobile applications. Current growth trends estimate in 2013 there will be 1.5 million mobile applications.
• In 2007 there were approximately 624,000 security threats (the document doesn’t specify what this really means). In 2010, there will be 2.6 million security threats. The Symantec and Cisco projection for 2013 predicts 5.7 million security threats.

It is amazing how much things in the IT world have changed in the past three years and taking that projection out another three years seems staggering. How is an organization supposed to handle the growing environment and the growing threats? Cisco offers some suggestions in this report:

1. Close gaps in situational awareness. Be aware of the totality of the network.
2. Focus first on solving “old” issues – and doing it well. Begin making improvements in the area of software updates and patches.
3. Educate your workforce on security – and include them in the process. Remember in information sec-u-r-it-y, You Are IT (U-R-IT). Kinda cheesy I know but it is a basic truth. We are all responsible for IT security.
4. Understand that one security border is no longer enough. Business has now become borderless and mobile.
5. View security as a differentiator for your business. “How an enterprise approaches security and responds to trends such as social networking and mobility can have a direct impact on ability to hire and retain talent.”

What do you think is going to happen in the next 3 years with regards to devices, applications, and security threats? Is the Cisco on target, or off base? Let me and other ITKE readers know your thoughts. Thanks for reading and let’s continue to be good network citizens.

Network-Tools owner sues Microsoft, Cisco, Comcast and TRUSTe over IP Address Blacklisting
Suit alleges eavdropping, privacy policy fraud, breach of contract and defamation

Interesting stuff, huh? So why would this suit be raised? According to the page tracking the lawsuit:

The lawsuit claims that Comcast, Microsoft, and Cisco collected information about Smith’s IP addresses and either put them on a “blacklist” or gave them a poor “Reputation Score.” Comcast even blocked his communication link with a mail server he operates outside the Comcast network. The suit claims that in order to collect this information in the first place Comcast, Cisco and Microsoft violated eavesdropping laws. The suit goes on to claim that Comcast, Microsoft, and Cisco failed to adhere to their privacy policies. When Smith tried to use the privacy policies of Comcast, Microsoft, and Cisco to correct the spammer accusations the companies balked. Comcast even told him it didn’t matter what the privacy policy said, he wasn’t getting the information. He filed complaints with the TRUSTe organization that verifies the privacy policies of Microsoft and Comcast but that did no good.

Previous lawsuits against these “blacklists” have been brought by commercial e-mailers against organizations such as Spamhaus. In this case the accused is not a commercial e-mail, not a spammer, and has no mailing lists of any sort. The accused has even made presentations at the Federal Trade Commission against spammers and testified at the first “Spam Summit” more than 10 years ago.

This case seems to cover a lot of things: privacy; net neutrality; service blockage.  For anyone who has gotten on a blocklist or had to get an organization removed from a blocklist, you can understand some of the frustration. Mr. Smith has gone beyond frustration and is taking some specific actions. This could be a very interesting case to follow based on the defendant organizations. Hopefully information will continue to be provided as the case moves forward. A lot of the case documentation presented to the court can be found on the website. It makes for some interesting reading.

Consider also some of the relief being sought under the lawsuit (not all items requested are listed below):

• Prohibit Microsoft, Comcast and Cisco from eavesdropping on Internet communications of the citizens of New Jersey,
• Prohibit Comcast displaying or distributing false or misleading portions of the Privacy Policy, Customer Privacy Notice, Acceptable Use Policy for High-Speed Internet Services, Network Management Policy, Network Management FAQ, Spam Policy and other related information to the citizens of New Jersey,
• Prohibit Microsoft from displaying or distributing false or misleading portions of the Privacy Statement and other related information to the citizens of New Jersey,
• Prohibit Cisco from displaying or distributing false or misleading portions of the Privacy Statement and other related information to the citizens of New Jersey,
• Prohibit TRUSTe from conducting a false or misleading dispute resolutions services to the citizens of New Jersey,
• Prohibit TRUSTe from endorsing any privacy policies displayed to citizens of New Jersey,
• Prohibit TRUSTe from claiming they certify entire companies when they only certify specific web sites,
• Require Microsoft, Comcast and Cisco to provide Plaintiff with all information collected about Plaintiff’s Internet communications or any associated data or any PII and allow Plaintiff to correct any erroneous information, and
• Prohibit Microsoft, Comcast and Cisco from distributing any defamatory information about Plaintiff to any third party,
• Compensatory damages to compensate Plaintiff for being unable to communication via e-mail without disruptions,
• Compensatory damages to compensate Plaintiff for being unable to communication via e-mail without eavesdropping,
• Compensatory damages to compensate Plaintiff for being unable to correct “profiles” maintained by Defendants about Plaintiff,
• Compensatory damages to compensate Plaintiff for time lost in running his business,
• Statutory damages pursuant to the  Electronic Communications Privacy Act,

What do you think of the merits of the case? What have you experienced with regards to these organizations and their services? Please leave comments below.

Thanks for reading and let’s continue to be good network citizens! Sometimes it may require getting nasty though it seems and filing a lawsuit

While you are her on ITKE, why not take some time, read through a few of my blog postings, maybe there is something there that would be of value to you or someone else you know. Send your fellow IT peers to ITKE. Make this the best free online support community and a one-stop shop for getting the support you need for those IT issues we each face every day.

Some of my blogs that will hopefully be of interest to you include:

What did I just do with my contacts list? – Social Engineering/Networking & contact list scraping

Network speed & capacity are NOT the same

Financial crisis due to poor risk understanding & management – IT security next?

Nifty tools for tracking down that “interesting” network traffic

PROTOCOL analysis vs protocol analysis (with a small p)

Good luck with the contest! Stay tuned for more and thanks for reading. Let’s continue to be good network citizens together.

As you may already know, I work at an international company with sites around the globe. There are over 2500 computer nodes not including printers, servers, switches, etc. Sometimes it is necessary to identify what traffic is crossing the network links between the sites. There are lots of tools and processes that can be used to gather this information. I will outline a couple here.

Our WAN edge routers are from Cisco. One of the features that can be enabled on a Cisco router is the ip cache flow feature. The show ip cache flow command returns some very useful information. An example is shown below.

show ip cache flow

IP packet size distribution (116972772 total packets):

1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480

.000 .375 .090 .023 .010 .007 .006 .003 .002 .014 .011 .010 .009 .005 .004

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608

.004 .003 .006 .028 .378 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes

64 active, 4032 inactive, 4367569 added

80215342 ager polls, 0 flow alloc failures

Active flows timeout in 30 minutes

Inactive flows timeout in 15 seconds

IP Sub Flow Cache, 21640 bytes

0 active, 1024 inactive, 0 added, 0 added to flow

0 alloc failures, 0 force free

1 chunk, 1 chunk added

last clearing of statistics never

Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)

--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow

TCP-Telnet         724      0.0         7   430      0.0       6.1      15.4

TCP-FTP          13859      0.0         9    93      0.0       6.7       3.4

TCP-WWW        3537205      0.8        14  1021     12.2       3.7       9.7

TCP-SMTP           290      0.0       104   989      0.0       5.5       1.8

TCP-X                3      0.0         2    42      0.0       0.3       1.3

TCP-BGP             18      0.0         1    43      0.0       0.0      13.9

TCP-Frag           112      0.0        37    78      0.0      18.3      15.5

TCP-other       684674      0.1        12   831      2.0       6.4       7.0

UDP-DNS           1973      0.0         1    72      0.0       0.1      15.4

UDP-NTP            248      0.0         1    77      0.0       0.0      15.4

UDP-Frag             3      0.0         1    45      0.0       0.0      15.6

UDP-other        10247      0.0         1   210      0.0       0.8      15.4

ICMP             97640      0.0        19    83      0.4      18.6      15.4

GRE              20509      0.0      2598   150     12.4     165.6      14.5

Total:         4367505      1.0        26   593     27.2       5.2       9.4

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts

Tu0           10.aa.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.cc.12.200    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.dd.12.8      Fa0/0         10.bb.12.150    06 0D0A 0871   467

Tu0           10.ee.12.200    Fa0/0         10.bb.ee.140    06 0A23 01BD     1

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.130    06 048A 07DA     1

Tu0           10.gg.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.hh.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.11     06 048A 04A7     1

Tu0           10.oo.12.210    Fa0/0         10.bb.12.200    11 0035 EA0B     1

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055D     5

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055E    10

As you can see it includes statistics about the packet size distribution, the various protocols and amount of traffic for each protocol and then a summary listing of the traffic through the various interfaces on the router. In this case, the traffic is passing through a couple of encrypted tunnel interfaces. This is where things get interesting when troubleshooting traffic on a link. The first column is the source interface, then the source IP address. The third column is the destination interface followed by the destination IP address. The next 3 columns give some critical information about the traffic between the source and destination hosts. These values are all given in HEX. There is the protocol number (e.g. 01 – ICMP, 06 – TCP, 11 – UDP). See the protocol listing at IANA for more information on these numbers – remember to convert from HEX to decimal.

The next two columns are the source port and destination port pairing. These values are also in HEX. So, converting values like 01BD to 445 indicates that the traffic is Microsoft DS according to the port number listing at IANA. Port 0035 (53 decimal) would be DNS traffic. Port 0050 (80 decimal) would be http traffic. Port 01BB (443 decimal) would be https. So, as you can see, lots of information is right there on the router and no sniffing is required to see what traffic is on your network.

Once you find an “interesting” source and destination pair that concerns you, you might consider finding out what application is generating the traffic between that source / destination pair. This can be done unobtrusively using some of the excellent tools from the Microsoft/Sysinternals toolkit. For example, the following command will list the current tcp & udp connections on a remote computer (10.xx.50.81) – note that you must have administrative access to the remote computer to run this command (netstat is not a Sysinternals tool but is built into the Windows operating system):

psexec \\10.xx.50.81 netstat -ano

The output would look something like this:

PsExec v1.94 – Execute processes remotely

Copyright (C) 2001-2008 Mark Russinovich

Sysinternals – www.sysinternals.com

Active Connections

Proto Local Address Foreign Address State PID

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 852

TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4

TCP 0.0.0.0:5800 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:8085 0.0.0.0:0 LISTENING 1456

TCP 10.xx.50.81:139 0.0.0.0:0 LISTENING 4

TCP 10.xx.50.81:445 10.bb.50.64:1826 ESTABLISHED 4

TCP 10.xx.50.81:1221 10.xx.12.200:135 ESTABLISHED 608

TCP 10.xx.50.81:1222 10.xx.12.200:1026 ESTABLISHED 608

TCP 10.xx.50.81:1822 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1823 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1827 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1828 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1829 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1830 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1831 10.xx.50.241:8080 ESTABLISHED 3756

TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING 2412

UDP 0.0.0.0:445 *:* 4

UDP 0.0.0.0:500 *:* 608

netstat exited on 10.xx.50.81 with error code 0.

So, these results show that the host has various tcp & udp connections that are in an established state. It shows the source & destination ports (again like the show ip cache flow results). The other very useful piece of information that is shown is the PID or process identifier. This number matches a process running on the remote computer. So, to find out what the various running processes are and their PID’s, run the following command:

pslist \\10.xx.50.81

The results returned are like the following:

pslist v1.28 – Sysinternals PsList

Copyright ¬ 2000-2004 Mark Russinovich

Sysinternals

Process information for 10.xx.50.81:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time

Idle 0 0 1 0 0 0:37:20.984 0:00:00.000

System 4 8 67 316 0 0:00:48.343 0:00:00.000

smss 464 11 3 21 164 0:00:00.015 4:43:15.698

csrss 528 13 15 545 2520 0:00:13.484 4:43:14.792

winlogon 552 13 19 524 9488 0:00:04.265 4:43:14.370

services 596 9 16 295 1876 0:00:04.281 4:43:14.183

lsass 608 9 20 428 4160 0:00:02.843 4:43:14.167

svchost 792 8 17 193 3284 0:00:00.796 4:43:13.667

svchost 852 8 10 371 2144 0:00:35.421 4:43:13.370

svchost 916 8 70 2092 16500 0:00:54.359 4:43:13.292

svchost 968 8 6 84 1596 0:00:00.921 4:43:13.245

svchost 992 8 15 292 3044 0:00:00.843 4:43:12.714

spoolsv 1196 8 12 142 3492 0:00:00.296 4:43:12.277

stormliv 1324 8 9 163 4952 0:00:08.343 4:43:04.339

EngineServer 1444 8 3 35 576 0:00:00.078 4:43:03.995

FrameworkService 1456 8 21 356 20632 0:00:37.203 4:43:03.573

VsTskMgr 1504 8 19 243 7128 0:00:29.578 4:43:02.714

MDM 1556 8 4 86 1092 0:00:00.140 4:43:02.495

mfevtps 1580 8 6 126 6848 0:00:02.609 4:43:02.370

ArchivingORBService 1636 8 4 88 3304 0:00:15.031 4:43:01.964

svchost 1696 8 5 118 2608 0:00:00.453 4:43:01.777

CcmExec 1836 8 13 810 14688 0:00:12.796 4:43:01.214

Mcshield 1880 13 26 182 45316 0:02:15.078 4:42:59.464

naPrdMgr 1964 8 6 130 208448 0:01:05.328 4:42:57.902

mfeann 1968 8 8 151 2264 0:00:01.625 4:42:57.855

alg 2412 8 5 102 1256 0:00:00.109 4:42:17.303

wmiprvse 2876 8 4 140 4132 0:00:00.781 4:42:09.979

wmiprvse 2660 8 7 146 1996 0:00:00.828 4:39:42.549

explorer 3676 8 12 442 17392 0:01:01.828 3:59:34.124

hkcmd 4092 8 2 86 896 0:00:00.140 3:59:30.406

igfxpers 816 8 3 93 868 0:00:00.078 3:59:30.343

UdaterUI 3388 8 5 115 1648 0:00:00.859 3:59:27.390

shstat 3252 8 10 98 2160 0:00:00.812 3:59:27.093

ctfmon 3968 8 1 67 984 0:00:00.156 3:59:25.828

Then if we need to remotely stop a running process that we consider suspicious or “interesting” issue the following command:

pskill 3968 \\10.xx.50.81 – note you can use either the PID # or the name of the process – however, you should use the PID if there are multiple instances of the application running

The results of the command, if successful, should look like:

PsKill v1.12 – Terminates processes on local or remote systems

Copyright (C) 1999-2005 Mark Russinovich

Sysinternals – www.sysinternals.com

Process 3968 on 10.xx.50.81 killed….

This process has become very useful when finding some rogue processes (malware) on some remote computers and there is no other way to disable the system or application. You can also issue a psshutdown command in a similar fashion, but the user may attempt to restart the machine again and then you will have to again shutdown the rogue application. There’s lots of ways to handle this situation including shutting down the LAN switch port if you have that access and privilege.

Let me know what processes you go through when managing remote systems where you may have limited physical access. Good luck out there and let’s be good network citizens!

VOIP – IPT – QOS – COS on and on – Oh My!

CampIT Enterprise VOIP conference

VOIP virtual panel discussion

Recently we had a phone system outage at the largest of these sites. This was a site with a clustered Cisco CallManager solution. This outage lasted 4+ hours. We were definitely surprised that both members of the cluster failed at the same time and how long it took to recover. Since that time we obviously are working with our support vendor to find a better method of providing uptime to the phone system at this site. I am also looking at making sure my other sites are prepared in the event of a similar outage.

The solution for providing a backup to the CallManager cluster is called Survivable Remote System Telephony (SRST). Think of this as CallManager light. A limited number of the phones still have connectivity and can make/receive calls. I say “limited” because the SRST function is dependent on the PSTN gateway hardware. A larger gateway can support more users. The current gateway we had was a Cisco 2821 series router. This would support 96 users. A Cisco 3825 will support 175 users.

One thing I understand though is that you cannot necessarily specify which phones will get serviced by SRST. The phones are serviced on a first-come-first-served basis. This could be an issue if there are phones that should be serviced and an outage is occurring. Unneeded phones would need to be disconnected from the network to provide capacity to support the critical phones.

Hopefully this will be the last of 4+ hour outage for the phone systems at this site and none will happen at my others. The Cisco solution has been very good for my organization and so far has been very reliable with the exception reported here.

Thanks for continuing to read my blog and hope you have a great day on the technology frontier wherever that may be for you!

Some background on our environment: IP phone population – over 400, distributed at 4 sites, largest ~150, smallest 60; all Cisco

Why implement VOIP?

• greenfield site – needed a phone system and VOIP made sense for a new site install to position for future

• acquired company in process of implementing VOIP – came into a situation where an acquisition had purchased VOIP and I became owner of the implementation; had issues with chosen vendor and equipment lists; eventually came out successful but was not without its pain during implementation.

• forward looking strategy – setup the company to have regional communication hubs for IP telephony; we have VOIP in North America, Europe and Asia now; this could permit us to leverage our WAN for toll bypass provided we build other local site infrastructure to support this technology.

Our biggest challenges:

• users: they find the phones easy to use and very good features; however, there are some features like managing meet-me conference calling that they feel are too onerous so don’t take the time to use this cost-saving feature

• administrators: setting up phones is an infrequent event so it is not a real simple task to setup a new phone; moves are made easier than traditional systems; troubleshooting skills are different since voice now is carried over the data network until it reaches a PSTN gateway

Best features:

• dial another site using extensions rather than 10 digit or more dialing

• “on phone” directory – can lookup another IP phone user’s extension directly on the phone rather than finding them on a piece of paper or website somewhere

• easier conference calling than old system

• mobile-phone like features: listing missed calls; call history log

• moves are made easier; adds are a challenge since done infrequently

Desires for additional features/services:

• video

• more ringtones (must have been someone young and a heavy cell phone user)

• integration with e-mail/web

What are the risks?

• it’s challenging to implement in an “old school” infrastructure environment (flat network, no-vlans, hubs still in use, etc.) It takes lots of forethought and understanding VLAN’s, WAN links, need to update staff skills.

• The network MUST be reliable or voice will suffer. Traditional phone companies have had 100+ years to make a bulletproof network.

• Costs. It’s not cheap to implement this technology. You have to weigh the ability of the organization to support non-industry leading implementations versus choosing the best technology you can afford.

• Maintenance. Upgrading the software in the servers, gateways and phones is much riskier than upgrading a traditional PBX environment.

What are the rewards?

• It works!

• It positions the organization to take advantage of other services provided that it is not simply an IT-led project but meets business requirements.

Feel free to add comments on your own experiences, concerns. This is a great forum and keep up the good work of information sharing!