Will Microsoft ever get search right?

I am looking for some documentation on Powershell to better understand how to use it. Per Wikipedia: Windows PowerShell is an extensible command-line shell and associated scripting language from Microsoft. So, I went to the Microsoft.com home page at www.microsoft.com. I typed powershell into the Search field at the very top of the page. I clicked the magnifying glass… waited a few seconds… and NOTHING was returned! So, I clicked on the Live Search option and 39,500 results were returned. So, now when I go to the main Microsoft page and then enter powershell into the same search term field as before and press Enter, the Live search results get returned – filtered for Microsoft.com only. It seems like my Live search excursion “woke up” the main Microsoft website search into knowing some powershell content does exist at Microsoft.com

I have often been frustrated in the past when searching Microsoft support using the exact error or event code from a Microsoft system or application log and nothing gets returned. It just seems like Microsoft is still missing the boat when it comes to search.

So, I guess I will continue to Google for Microsoft support information until I can see that Microsoft is better able to search their own website from their homepage.

Thanks for reading & let’s be good network citizens out there!

ARP as a network auditing tool

ARP – or Address Resolution Protocol is a necessary element for network traffic. Per Wikipedia: “In computer networking, the Address Resolution Protocol (ARP) is the method for finding a host’s link layer (hardware) address when only its Internet Layer (IP) or some other Network Layer address is known. ARP is defined in RFC 826.^[1] It is Internet Standard STD 37.” It is not an IP only protocol.

What this means, is that ARP is not a protocol that is easily blocked or disabled on a network. This is as designed but this also means that attackers can use this protocol for malicious activities. It is important that you understand the ARP protocol and the ways it is used and the dangers associated with it.

Laura Chappell, the BitGirl, has created a new tutorial on using ARP to scan networks which may be firewalled or ICMP pings are blocked. ARP will permit you – and attackers – to find hosts on the network. Take some time and watch this short video and gain some valuable insights into ARP.

Watch Chappell University – Ethical Hacking with NetScanTools Pro – ARP Scanning

Thanks for your time and let’s be good network citizens!

Nifty tools for tracking down that “interesting” network traffic

My previous posting was meant to help you determine the source of potentially dangerous network traffic at your network’s edge. This post is meant to help you identify applications and traffic on your local network that seems to be “interesting”. I define “interesting” as something that you don’t know much about but would find it interesting to learn more about and maybe take some action to shutdown.

As you may already know, I work at an international company with sites around the globe. There are over 2500 computer nodes not including printers, servers, switches, etc. Sometimes it is necessary to identify what traffic is crossing the network links between the sites. There are lots of tools and processes that can be used to gather this information. I will outline a couple here.

Our WAN edge routers are from Cisco. One of the features that can be enabled on a Cisco router is the ip cache flow feature. The show ip cache flow command returns some very useful information. An example is shown below.

show ip cache flow

IP packet size distribution (116972772 total packets):

1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480

.000 .375 .090 .023 .010 .007 .006 .003 .002 .014 .011 .010 .009 .005 .004

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608

.004 .003 .006 .028 .378 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes

64 active, 4032 inactive, 4367569 added

80215342 ager polls, 0 flow alloc failures

Active flows timeout in 30 minutes

Inactive flows timeout in 15 seconds

IP Sub Flow Cache, 21640 bytes

0 active, 1024 inactive, 0 added, 0 added to flow

0 alloc failures, 0 force free

1 chunk, 1 chunk added

last clearing of statistics never

Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)

--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow

TCP-Telnet         724      0.0         7   430      0.0       6.1      15.4

TCP-FTP          13859      0.0         9    93      0.0       6.7       3.4

TCP-WWW        3537205      0.8        14  1021     12.2       3.7       9.7

TCP-SMTP           290      0.0       104   989      0.0       5.5       1.8

TCP-X                3      0.0         2    42      0.0       0.3       1.3

TCP-BGP             18      0.0         1    43      0.0       0.0      13.9

TCP-Frag           112      0.0        37    78      0.0      18.3      15.5

TCP-other       684674      0.1        12   831      2.0       6.4       7.0

UDP-DNS           1973      0.0         1    72      0.0       0.1      15.4

UDP-NTP            248      0.0         1    77      0.0       0.0      15.4

UDP-Frag             3      0.0         1    45      0.0       0.0      15.6

UDP-other        10247      0.0         1   210      0.0       0.8      15.4

ICMP             97640      0.0        19    83      0.4      18.6      15.4

GRE              20509      0.0      2598   150     12.4     165.6      14.5

Total:         4367505      1.0        26   593     27.2       5.2       9.4

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts

Tu0           10.aa.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.cc.12.200    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.dd.12.8      Fa0/0         10.bb.12.150    06 0D0A 0871   467

Tu0           10.ee.12.200    Fa0/0         10.bb.ee.140    06 0A23 01BD     1

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.130    06 048A 07DA     1

Tu0           10.gg.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.hh.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20

Tu0           10.ff.12.150    Fa0/0         10.bb.ee.11     06 048A 04A7     1

Tu0           10.oo.12.210    Fa0/0         10.bb.12.200    11 0035 EA0B     1

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055D     5

Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055E    10

As you can see it includes statistics about the packet size distribution, the various protocols and amount of traffic for each protocol and then a summary listing of the traffic through the various interfaces on the router. In this case, the traffic is passing through a couple of encrypted tunnel interfaces. This is where things get interesting when troubleshooting traffic on a link. The first column is the source interface, then the source IP address. The third column is the destination interface followed by the destination IP address. The next 3 columns give some critical information about the traffic between the source and destination hosts. These values are all given in HEX. There is the protocol number (e.g. 01 – ICMP, 06 – TCP, 11 – UDP). See the protocol listing at IANA for more information on these numbers – remember to convert from HEX to decimal.

The next two columns are the source port and destination port pairing. These values are also in HEX. So, converting values like 01BD to 445 indicates that the traffic is Microsoft DS according to the port number listing at IANA. Port 0035 (53 decimal) would be DNS traffic. Port 0050 (80 decimal) would be http traffic. Port 01BB (443 decimal) would be https. So, as you can see, lots of information is right there on the router and no sniffing is required to see what traffic is on your network.

Once you find an “interesting” source and destination pair that concerns you, you might consider finding out what application is generating the traffic between that source / destination pair. This can be done unobtrusively using some of the excellent tools from the Microsoft/Sysinternals toolkit. For example, the following command will list the current tcp & udp connections on a remote computer (10.xx.50.81) – note that you must have administrative access to the remote computer to run this command (netstat is not a Sysinternals tool but is built into the Windows operating system):

psexec \\10.xx.50.81 netstat -ano

The output would look something like this:

PsExec v1.94 – Execute processes remotely

Copyright (C) 2001-2008 Mark Russinovich

Sysinternals – www.sysinternals.com

Active Connections

Proto Local Address Foreign Address State PID

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 852

TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4

TCP 0.0.0.0:5800 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:8085 0.0.0.0:0 LISTENING 1456

TCP 10.xx.50.81:139 0.0.0.0:0 LISTENING 4

TCP 10.xx.50.81:445 10.bb.50.64:1826 ESTABLISHED 4

TCP 10.xx.50.81:1221 10.xx.12.200:135 ESTABLISHED 608

TCP 10.xx.50.81:1222 10.xx.12.200:1026 ESTABLISHED 608

TCP 10.xx.50.81:1822 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1823 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1827 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1828 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1829 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1830 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1831 10.xx.50.241:8080 ESTABLISHED 3756

TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING 2412

UDP 0.0.0.0:445 *:* 4

UDP 0.0.0.0:500 *:* 608

netstat exited on 10.xx.50.81 with error code 0.

So, these results show that the host has various tcp & udp connections that are in an established state. It shows the source & destination ports (again like the show ip cache flow results). The other very useful piece of information that is shown is the PID or process identifier. This number matches a process running on the remote computer. So, to find out what the various running processes are and their PID’s, run the following command:

pslist \\10.xx.50.81

The results returned are like the following:

pslist v1.28 – Sysinternals PsList

Copyright ¬ 2000-2004 Mark Russinovich

Sysinternals

Process information for 10.xx.50.81:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time

Idle 0 0 1 0 0 0:37:20.984 0:00:00.000

System 4 8 67 316 0 0:00:48.343 0:00:00.000

smss 464 11 3 21 164 0:00:00.015 4:43:15.698

csrss 528 13 15 545 2520 0:00:13.484 4:43:14.792

winlogon 552 13 19 524 9488 0:00:04.265 4:43:14.370

services 596 9 16 295 1876 0:00:04.281 4:43:14.183

lsass 608 9 20 428 4160 0:00:02.843 4:43:14.167

svchost 792 8 17 193 3284 0:00:00.796 4:43:13.667

svchost 852 8 10 371 2144 0:00:35.421 4:43:13.370

svchost 916 8 70 2092 16500 0:00:54.359 4:43:13.292

svchost 968 8 6 84 1596 0:00:00.921 4:43:13.245

svchost 992 8 15 292 3044 0:00:00.843 4:43:12.714

spoolsv 1196 8 12 142 3492 0:00:00.296 4:43:12.277

stormliv 1324 8 9 163 4952 0:00:08.343 4:43:04.339

EngineServer 1444 8 3 35 576 0:00:00.078 4:43:03.995

FrameworkService 1456 8 21 356 20632 0:00:37.203 4:43:03.573

VsTskMgr 1504 8 19 243 7128 0:00:29.578 4:43:02.714

MDM 1556 8 4 86 1092 0:00:00.140 4:43:02.495

mfevtps 1580 8 6 126 6848 0:00:02.609 4:43:02.370

ArchivingORBService 1636 8 4 88 3304 0:00:15.031 4:43:01.964

svchost 1696 8 5 118 2608 0:00:00.453 4:43:01.777

CcmExec 1836 8 13 810 14688 0:00:12.796 4:43:01.214

Mcshield 1880 13 26 182 45316 0:02:15.078 4:42:59.464

naPrdMgr 1964 8 6 130 208448 0:01:05.328 4:42:57.902

mfeann 1968 8 8 151 2264 0:00:01.625 4:42:57.855

alg 2412 8 5 102 1256 0:00:00.109 4:42:17.303

wmiprvse 2876 8 4 140 4132 0:00:00.781 4:42:09.979

wmiprvse 2660 8 7 146 1996 0:00:00.828 4:39:42.549

explorer 3676 8 12 442 17392 0:01:01.828 3:59:34.124

hkcmd 4092 8 2 86 896 0:00:00.140 3:59:30.406

igfxpers 816 8 3 93 868 0:00:00.078 3:59:30.343

UdaterUI 3388 8 5 115 1648 0:00:00.859 3:59:27.390

shstat 3252 8 10 98 2160 0:00:00.812 3:59:27.093

ctfmon 3968 8 1 67 984 0:00:00.156 3:59:25.828

Then if we need to remotely stop a running process that we consider suspicious or “interesting” issue the following command:

pskill 3968 \\10.xx.50.81 – note you can use either the PID # or the name of the process – however, you should use the PID if there are multiple instances of the application running

The results of the command, if successful, should look like:

PsKill v1.12 – Terminates processes on local or remote systems

Copyright (C) 1999-2005 Mark Russinovich

Sysinternals – www.sysinternals.com

Process 3968 on 10.xx.50.81 killed….

This process has become very useful when finding some rogue processes (malware) on some remote computers and there is no other way to disable the system or application. You can also issue a psshutdown command in a similar fashion, but the user may attempt to restart the machine again and then you will have to again shutdown the rogue application. There’s lots of ways to handle this situation including shutting down the LAN switch port if you have that access and privilege.

Let me know what processes you go through when managing remote systems where you may have limited physical access. Good luck out there and let’s be good network citizens!

Do you manage a firewall and want to find the source of malicious network activity? – Check out the Internet Malicious Activity Map

For those of you who manage your own network, you have to consider the strength of the firewall at your network perimiter, the knowledge and skills of those who manage it. You also have to provide technology that can help protect your mobile users. Part of building that secure environment is understanding the environment out there in the wild world web.This is just one of the resources available out there. Please leave feedback if you are aware of others that might be useful to readers.

I recently came across an interesting graph that shows where some of the malicious traffic originates from on the internet. It is called the Internet malicious activity map (PNG) The graph is from Team Cymru. The graph displays in “heatmap” style in a Hilbert Curve (check this out if you are a fan of fractals). This is an interesting way to graph a lot of data in a small space. As is true in heatmaps, the colors indicate the concentration of malicious activity. The lighter the color, the higher the malicious activity. Take a look at the 85.x.x.x/8, 87.x.x.x/8, and 88.x.x.x/8 sections of the graph. Looks like these networks are major sources of malicious activity on the internet. I would recommend reviewing this graph and determining if the address ranges showing high malicious activities are part of your organization’s network. If so, then be very concerned. If not, then does your network receive any traffic originating on these subnets? Maybe you should consider blocking traffic from these source subnets. See the Team Cymru Malevolence Monitoring website for more security oriented information.

Thanks for reading and let’s be good network citizens!

Microsoft guidelines for Turning off Windows AutoRun do NOT work properly!

One of the information security lists I subscribe to is the US-CERT Technical Cyber Security Alerts. US-CERT is the United States Computer Emergency Readiness Team. If you have information security responsibilities, I highly recommend that you visit their website and register for their mailing lists and subscribe to the RSS feeds to get the latest information on information security issues from a trusted US Government source.

In case you have not seen or heard the latest US-CERT Technical Cyber Security Alert reads as shown below. I don’t know about you but the information in this bulletin really concerns me. I know personally how autorun.inf can affect a computer. I recently received a digital picture frame (DPF) as a gift. It is a very nice one in that it can handle several different types of media and is even an MP3 player. When I connected it to my computer the first time, Windows went through the “new device found” routine. Windows found the device as a standard removable storage device. That was no big deal. However, the DPF has 128MB of internal storage and that storage held an autorun.inf file that referenced a trojan executable! Fortunately my anti-virus detected it and deleted the file before it could do damage. How many consumers do not have antivirus? How would the trojan affected their systems? That is a substantial risk in today’s technology environment!

I would highly recommend taking the steps outlined below to ensure that autorun.inf does not take down a critical system within your organization.

Thanks for reading & let’s continue to be good network citizens.

National Cyber Alert System

Technical Cyber Security Alert TA09-020A

Microsoft Windows Does Not Disable AutoRun Properly

Original release date: January 20, 2009

Last revised: –

Source: US-CERT

Systems Affected

* Microsoft Windows

Overview

Disabling AutoRun on Microsoft Windows systems can help prevent the spread of malicious code. However, Microsoft’s guidelines for disabling AutoRun are not fully effective, which could be considered a  vulnerability.

I. Description

Microsoft Windows includes an AutoRun feature, which can automatically run code when removable devices are connected to the computer. AutoRun (and the closely related AutoPlay) can unexpectedly cause arbitrary code execution in the following situations:

* A removable device is connected to a computer. This includes, but is not limited to, inserting a CD or DVD, connecting a USB or Firewire device, or mapping a network drive. This connection can result in code execution without any additional user interaction.

* A user clicks the drive icon for a removable device in Windows Explorer. Rather than exploring the drive’s contents, this action can cause code execution.

* The user selects an option from the AutoPlay dialog that is displayed when a removable device is connected. Malicious software, such as W32.Downadup, is using AutoRun to spread. Disabling AutoRun, as specified in the CERT/CC Vulnerability Analysis blog, is an effective way of helping to prevent the spread of malicious code.

The Autorun and NoDriveTypeAutorun registry values are both ineffective for fully disabling AutoRun capabilities on Microsoft Windows systems. Setting the Autorun registry value to 0 will not prevent newly connected devices from automatically running code specified in the Autorun.inf file. It will, however, disable Media Change Notification (MCN) messages, which may prevent Windows from detecting when a CD or DVD is changed. According to Microsoft, setting the NoDriveTypeAutorun registry value to 0xFF “disables

Autoplay on all types of drives.” Even with this value set, Windows may execute arbitrary code when the user clicks the icon for the device in Windows Explorer.

II. Impact

By placing an Autorun.inf file on a device, an attacker may be able to automatically execute arbitrary code when the device is connected to a Windows system. Code execution may also take place when the user attempts to browse to the software location with Windows Explorer.

III. Solution

Disable AutoRun in Microsoft Windows

To effectively disable AutoRun in Microsoft Windows, import the following registry value:

REGEDIT4

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\IniFileMapping\Autorun.inf]

@=”@SYS:DoesNotExist”

To import this value, perform the following steps:

* Copy the text

* Paste the text into Windows Notepad

* Save the file as autorun.reg

* Navigate to the file location

* Double-click the file to import it into the Windows registry

Microsoft Windows can also cache the AutoRun information from mounted devices in the MountPoints2 registry key. We recommend restarting Windows after making the registry change so that any cached mount points are reinitialized in a way that ignores the Autorun.inf file. Alternatively, the following registry key may be deleted:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2

Once these changes have been made, all of the AutoRun code execution scenarios described above will be mitigated because Windows will no longer parse Autorun.inf files to determine which actions to take. Further details are available in the CERT/CC Vulnerability Analysis blog. Thanks to Nick Brown and Emin Atac for providing the workaround.

IV. References

* The Dangers of Windows AutoRun -

<http://www.cert.org/blogs/vuls/2008/04/the_dangers_of_windows_autorun.html>

* US-CERT Vulnerability Note VU#889747 -

<http://www.kb.cert.org/vuls/id/889747>

* Nick Brown’s blog: Memory stick worms -

<http://nick.brown.free.fr/blog/2007/10/memory-stick-worms>

* TR08-004 Disabling Autorun -

<http://www.publicsafety.gc.ca/prg/em/ccirc/2008/tr08-004-eng.aspx>

* How to Enable or Disable Automatically Running CD-ROMs -

<http://support.microsoft.com/kb/155217>

* NoDriveTypeAutoRun -

<http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/91525.mspx>

* Autorun.inf Entries -

<http://msdn.microsoft.com/en-us/library/bb776823(VS.85).aspx>

* W32.Downadup -

<http://www.symantec.com/security_response/writeup.jsp?docid=2008-112203-2408-99>

* MS08-067 Worm, Downadup/Conflicker -

<http://www.f-secure.com/weblog/archives/00001576.html>

* Social Engineering Autoplay and Windows 7 -

<http://www.f-secure.com/weblog/archives/00001586.html>

____________________________________________________________________

The most recent version of this document can be found at:

<http://www.us-cert.gov/cas/techalerts/TA09-020A.html>

____________________________________________________________________

Feedback can be directed to US-CERT Technical Staff. Please send email to <cert@cert.org> with “TA09-020A Feedback VU#889747″ in the subject.

____________________________________________________________________

For instructions on subscribing to or unsubscribing from this mailing list, visit <http://www.us-cert.gov/cas/signup.html>.

____________________________________________________________________

Produced 2009 by US-CERT, a government organization.

Terms of use:

<http://www.us-cert.gov/legal.html>

____________________________________________________________________

Revision History

January 20, 2009: Initial release

================================================

Security news – Videos from Hack In The Box 2008 Malaysia available for download

The videos from HITBSecConf2008 – Malaysia are now available for download!

Day 1

=====

http://thepiratebay.org/torrent/4654588/HITBSecConf2008_-_Malaysia_Videos___Day_1

Keynote Address 1: The Art of Click-Jacking – Jeremiah Grossman Keynote Address 2: Cyberwar is Bullshit – Marcus Ranum

Presentations:

- Delivering Identity Management 2.0 by Leveraging OPSS

- Bluepilling the Xen Hypervisor

- Pass the Hash Toolkit for Windows

- Internet Explorer 8 – Trustworthy Engineering and Browsing

- Full Process Reconsitution from Memory

- Hacking Internet Kiosks

- Analysis and Visualization of Common Packers

- A Fox in the Hen House – UPnP IGD

- MoocherHunting

- Browser Exploits: A New Model for Browser Security

- Time for a Free Hardware Foundation?

- Mac OS Xploitation

- Hacking a Bird in The Sky 2.0

- How the Leopard Hides His Spots – OS X Anti-Forensics Techniques

Day 2

=====

http://thepiratebay.org/torrent/4654974/HITBSecConf2008_-_Malaysia_Videos___Day_2

Keynote Address 3: Dissolving an Industry as a Hobby – THE PIRATE BAY

Presentations:

- Pushing the Camel Through the Eye of a Needle

- An Effective Methodology to Enable Security Evaluation at RTL Level

- Remote Code Execution Through Intel CPU Bugs

- Next Generation Reverse Shell

- Build Your Own Password Cracker with a Disassembler and VM Magic

- Decompilers and Beyond

- Cracking into Embedded Devices and Beyond!

- Client-side Security

- Top 10 Web 2.0 Attacks

===

On a related note, the registration for HITBSecConf2009 – Dubai (20th – 23rd April) is now open!

http://conference.hitb.org/hitbsecconf2009dubai/

The Call for Papers (CFP) for HITBSecConf2009 – Malaysia (October 5th -

8th) will open in March 2009.

Skype users – does anyone use this on ITKE? – or is it overhyped?

Recently I posted a question about using Skype in a corporate environment. Based on the lack of any feedback, it really makes me wonder if Skype is an overhyped solution for corporate environments. I also posted the same questions to another professional mailing list I subscribe to and received only one (very good) response from that peer group.

So, to be redundant, I want to ask you blog readers the same questions. If there are no responses, then I will take this to mean that either your organizations do not use Skype or that you would rather not share poor experiences. I would appreciate feedback either positive or negative about using Skype in corporate environments.

As is true with most organizations today, we are under pressure to reduce costs where we can. Some users are coming and asking why we are not using Skype for international calling. I’m not sure if I fully understand the risks so I am posting this question here on ITKE.

Has anyone implemented Skype for their organization and is supporting it on company networks and equipment? If not, why not? If you have implemented Skype services, some additional information would be useful.

What precautions were required before implementing this service/application?
What has network usage been like since implementation?
What configuration changes at the firewall (both edge & client) were needed to support the application?

Please feel free to share any other advice you may have about this type of service/application.

PROTOCOL analysis vs protocol analysis (with a small p)

Recently we had an issue at a site where outbound messages larger than 1MB were backing up in the outbound message queue. The messages were tagged with a 421 4.4.2 Connection dropped error. This was a puzzling issue since the smart relay host was on the local LAN, and in fact, on the same switch as the Exchange server.  We checked the switch ports and NICs for errors. None were found. We knew messages were successfully coming inbound through this site because the smart relay host was processing hundreds of them per hour (we use regional hubs and this is one of our hub sites).

We first contacted the vendor for the smart relay host appliance and opened a support ticket. No real issues were identified at first review. Since the errors were being reported at the Exchange server, we contacted Microsoft and opened a support ticket. We spent hours testing and changing configuration to another regional smart relay host which seemed to get the messages delivered successfully, but we were still not able to find out what was causing the conversations with the local smart relay host to timeout.

So, we went into deeper debug mode since the application and server event logs did not shed any light on the issue. The Microsoft engineer enabled protocol logging on this particular send connector. The protocol logs did give a little more information on the situation. A snippet is shown below.

2009-01-08T22:36:19.495Z,SendConn,08CB3FF87FA34699,16,exchsvr:20709,relayhost:25,>,RCPT TO:<someone@there.com>,
2009-01-08T22:36:19.495Z,SendConn,08CB3FF87FA34699,17,exchsvr:20709,relayhost:25,<,”250 Requested mail action okay, completed.”,
2009-01-08T22:36:19.589Z,SendConn,08CB3FF87FA34699,18,exchsvr:20709,relayhost:25,>,DATA,
2009-01-08T22:36:19.589Z,SendConn,08CB3FF87FA34699,19,exchsvr:20709,relayhost:25,<,”354 Enter mail, end with “”.”" on a line by itself.”,
2009-01-08T22:36:25.417Z,SendConn,08CB3FF87FA34699,20,exchsvr:20709,relayhost:25,-,,Remote
2009-01-08T22:37:25.431Z,SendConn,08CB3FF87FA346A1,0,,relayhost:25,*,,attempting to connect
2009-01-08T22:37:25.431Z,SendConn,08CB3FF87FA346A1,1,exchsvr:20736,relayhost:25,+,,

The conversation seemed to go fine at the beginning but something was happening at the end. Since this log did not freely give up that information, we used Microsoft’s Network Monitor 3.2 (btw-if you are still using an older version of Network Monitor, you should upgrade to v3.2. It does have some nice features that make it more user friendly – but not as nice as Wireshark) to capture the actual packets between the Exchange server and the smart relay host. We ran Network Monitor directly on the Exchange server.

At this point, we were able to capture the transaction failures. The results were very interesting and a good lesson in packet analysis versus protocol analysis. The packet analysis showed that TCP was working well. Everything at layer 4 and below seemed to be working well. This was a relief. However, it appeared that the actual problem existed at layer 6 & 7. The Exchange server was ending the SMTP (Simple Mail Transport Protocol) conversation with the “.” command (a single dot on a line by itself). The Exchange server was then waiting for the smart relay host to reply with a 250 2.6.0 status message saying the message was successfully queued for delivery. The Exchange server would then reply with a QUIT command and end the SMTP session. Since the smart relay was not responding at all with the expected status message, the SMTP conversation was timing out and messages were building up in the queue.

We found out that there were some patches for the smart relay host so we applied those. Once that was done, the messages seemed to flow normally. The other puzzling thing about this is that we have two other hub sites with the same configuration that are not experiencing this problem. So, sometime today we will be rolling out the patches to those smart relay hosts to prevent this problem from happening at those sites. This issue started out of the blue but seemed coincide with the same time Exchange Server 2007 rollup 5 was applied.

The point of this whole blog posting is that while the TCP protocol was working fine and everything looked good there, the SMTP protcol was not working correctly. It is important for a network engineer to understand networking through all of the OSI layers. You cannot just assume that if things are working well at the lower levels that things at the higher levels will work too. The reverse logic is true also. So, understand the protocols at the lower layers and also the PROTOCOLS at the upper layers if you really want to be an effective troubleshooting expert.

Let’s be good network citizens out there!

Swiss-army knife for public network testing

Sometimes it is necessary to test connectivity outside of your private company network. There are several resources I use. I will share a couple of those with you in this posting.

One of my favorite and most frequently used sites is Network-Tools. This website allows you to test Traceroute, PIng, Domain Name Server (DNS) lookup, Whois, and DNS record lookups. This is an excellent resource like DNSTools or DNSStuff.

Another site with useful public internet testing tools is Serversniff.net. You can use this site to perform TCP pings rather than the standard ICMP pings. There is also a step-ping test. This provides the ability to have increasing ping packet sizes to see if there is a bottleneck somewhere before the tested host. There are lots of other tools available on this website. I recommend you check it out and see which offer value to you in your support activities.

Unfortunately, these tools only work from the public internet. You will not be able to test hosts on your private network, but hey, shouldn’t you already have some other testing tools in your toolbag for the private network? I’m sure I will describe more tools as the year moves on.

Thanks for reading & let’s practice safe networking out there! Please feel free to leave comments for other readers so they can adequately support their networks.

Is this program good or bad for my computer?

Have you ever wondered if an application or running process is good or bad? Google searches do a good job of helping you determine if a process is legitimate or not. I just came across another resource in the fight against malicious software. It is a search engine for files, CLSID’s, and application names. The site is SystemLookup. The search results show whether the item is Malware, spyware, adware, or other potentially unwanted items, Legitimate items, Open to debate, or Currently unknown status. The various categories available for search include:

browser helper objects (BHO), toolbars, search hooks, explorer bars
Internet Explorer Buttons
Layered Service Providers
ActiveX Installs
Extra protocols
AppInit_DLLs & Winlogon Notify
ShellServiceObjectDelayLoad
Shared Task Scheduler
Services

The website is community-based so please contribute and improve this resource for security information.