Mar 10 2009   4:47PM GMT

Saving Money & Stopping spam - change domain names

Posted by: Troy Tate
spam, email, domains, cost savings, cost reduction, WAN, internet, planning, operations

Are you getting lots of SPAM? Is your organization’s internet link being saturated due to tons of inbound spam and maybe outbound non-delivery notices for invalid addresses? About 3 years ago, ours was too. Continued »

Jan 9 2009   4:38PM GMT

PROTOCOL analysis vs protocol analysis (with a small p)

Posted by: Troy Tate
protocol analysis, SMTP, tcp, network monitor, wireshark, Microsoft, Microsoft Exchange, patches, OSI model

Recently we had an issue at a site where outbound messages larger than 1MB were backing up in the outbound message queue. The messages were tagged with a 421 4.4.2 Connection dropped error. This was a puzzling issue since the smart relay host was on the local LAN, and in fact, on the same switch as the Exchange server.  We checked the switch ports and NICs for errors. None were found. We knew messages were successfully coming inbound through this site because the smart relay host was processing hundreds of them per hour (we use regional hubs and this is one of our hub sites).

We first contacted the vendor for the smart relay host appliance and opened a support ticket. No real issues were identified at first review. Since the errors were being reported at the Exchange server, we contacted Microsoft and opened a support ticket. We spent hours testing and changing configuration to another regional smart relay host which seemed to get the messages delivered successfully, but we were still not able to find out what was causing the conversations with the local smart relay host to timeout.

So, we went into deeper debug mode since the application and server event logs did not shed any light on the issue. The Microsoft engineer enabled protocol logging on this particular send connector. The protocol logs did give a little more information on the situation. A snippet is shown below.

2009-01-08T22:36:19.495Z,SendConn,08CB3FF87FA34699,16,exchsvr:20709,relayhost:25,>,RCPT TO:<someone@there.com>,
2009-01-08T22:36:19.495Z,SendConn,08CB3FF87FA34699,17,exchsvr:20709,relayhost:25,<,”250 Requested mail action okay, completed.”,
2009-01-08T22:36:19.589Z,SendConn,08CB3FF87FA34699,18,exchsvr:20709,relayhost:25,>,DATA,
2009-01-08T22:36:19.589Z,SendConn,08CB3FF87FA34699,19,exchsvr:20709,relayhost:25,<,”354 Enter mail, end with “”.”” on a line by itself.”,
2009-01-08T22:36:25.417Z,SendConn,08CB3FF87FA34699,20,exchsvr:20709,relayhost:25,-,,Remote
2009-01-08T22:37:25.431Z,SendConn,08CB3FF87FA346A1,0,,relayhost:25,*,,attempting to connect
2009-01-08T22:37:25.431Z,SendConn,08CB3FF87FA346A1,1,exchsvr:20736,relayhost:25,+,,

The conversation seemed to go fine at the beginning but something was happening at the end. Since this log did not freely give up that information, we used Microsoft’s Network Monitor 3.2 (btw-if you are still using an older version of Network Monitor, you should upgrade to v3.2. It does have some nice features that make it more user friendly - but not as nice as Wireshark) to capture the actual packets between the Exchange server and the smart relay host. We ran Network Monitor directly on the Exchange server.

At this point, we were able to capture the transaction failures. The results were very interesting and a good lesson in packet analysis versus protocol analysis. The packet analysis showed that TCP was working well. Everything at layer 4 and below seemed to be working well. This was a relief. However, it appeared that the actual problem existed at layer 6 & 7. The Exchange server was ending the SMTP (Simple Mail Transport Protocol) conversation with the “.” command (a single dot on a line by itself). The Exchange server was then waiting for the smart relay host to reply with a 250 2.6.0 status message saying the message was successfully queued for delivery. The Exchange server would then reply with a QUIT command and end the SMTP session. Since the smart relay was not responding at all with the expected status message, the SMTP conversation was timing out and messages were building up in the queue.

We found out that there were some patches for the smart relay host so we applied those. Once that was done, the messages seemed to flow normally. The other puzzling thing about this is that we have two other hub sites with the same configuration that are not experiencing this problem. So, sometime today we will be rolling out the patches to those smart relay hosts to prevent this problem from happening at those sites. This issue started out of the blue but seemed coincide with the same time Exchange Server 2007 rollup 5 was applied.

The point of this whole blog posting is that while the TCP protocol was working fine and everything looked good there, the SMTP protcol was not working correctly. It is important for a network engineer to understand networking through all of the OSI layers. You cannot just assume that if things are working well at the lower levels that things at the higher levels will work too. The reverse logic is true also. So, understand the protocols at the lower layers and also the PROTOCOLS at the upper layers if you really want to be an effective troubleshooting expert.

Let’s be good network citizens out there!

Dec 3 2008   3:50PM GMT

Holiday greeting cards, holiday shopping and computer security awareness

Posted by: Troy Tate
administration, Firewalls, Security, Microsoft Windows, Browsers, IT education, spam, antivirus, homeland security, Data security, malware, SSL, phishing, Firefox, Microsoft, anti-virus, online identity, risk, awareness, vulnerability, education, data loss

I just sent this email reminder to all users in my organization. I would recommend you do something similar if you are not already ensuring users are aware of these issues. Feel free to use my content and add your own.

 It is that time of year again when folks send electronic holiday greeting cards to one another. Some of the greetings may also be games that bear holiday messages. It is also a time when malicious software spreads using these same types of messages and software. You should also be cautious when doing any holiday shopping online or at stores. It is important that you and those you communicate with understand these risks. Your finances and identity are always at risk in today’s technology environment, but you may be less attentive during the holiday season. The following 10 tips are meant to remind you of some important security precautions.

 

1.    Do NOT use your company email address for personal holiday greetings or shopping activities. Merchants may sell your email address to other non-reputable sources and this puts your company identity at risk.

 

2.    If you receive personal holiday greetings or “cute” games at your company email address, ask the sender to not send those to you at work. Use a personal email account for those communications.

 

3.    If you do receive holiday greetings or games at your personal email address, check with the sender before opening to be sure they sent the message. Spammers and malicious software writers can easily deceive you through social engineering. They will do everything possible to get you to open their message and potentially damage your computer and/or harvest your email address as a valid address.

 

4.    Don’t trust everything you see online. Finding something on the internet does not guarantee that it is true. Anyone can publish information online, so before accepting a statement as fact or taking action, verify that the source is reliable.

 

5.    If it looks too good to be true, it probably is. You have probably seen many emails promising fantastic rewards or monetary gifts. However, regardless of what the email claims, there are not any wealthy strangers desperate to send you money. Beware of grand promises—they are most likely spam, hoaxes, or phishing schemes. Also be wary of pop-up windows and advertisements for free downloadable software—they may be disguising spyware. Close the pop-up windows by clicking the X in the top right corner. Do not click the YES, NO, or CANCEL buttons in the window. It may cause unwanted computer issues if you do. Do not trust what you see in these pop-up windows. Contact IT support if you have any questions or issues.

 

6.    Avoid phishing schemes. Banks and other institutions will not actively solicit personal information by email. When you click a link in an email asking for this type of information, your choice may risk your finances and personal identity. The link may take you to a website hosted by someone with malicious intentions. If you enter your personal information on the website, you have just had your identity taken by a social engineering attack and may have incurred a financial loss.

 

7.    If you are unsure whether an email request is legitimate, try to verify it by contacting the company directly. Do not use contact information provided on a web site connected to the request; instead, check previous statements for contact information. Information about known phishing attacks is also available online from groups such as the Anti-Phishing Working Group (http://www.antiphishing.org/phishing_archive.html).

 

8.    If you believe your financial accounts may be compromised, contact your financial institution immediately and close any accounts that may have been compromised. Watch for any unexplainable charges to your account. Consider reporting the attack to the police, and file a report with the Federal Trade Commission (http://www.ftc.gov/).

 

9.    Do not participate in forwarding chain letters or perpetuating hoaxes or urban legends. Hoaxes attempt to trick or defraud users. A hoax could be malicious, instructing users to delete a file necessary to the operating system by claiming it is a virus. It could also be a scam that convinces users to send money or personal information. Phishing attacks could fall into this category. Urban legends are designed to be redistributed and usually warn users of a threat or claim to be notifying them of important or urgent information. Another common form are the emails that promise users monetary rewards for forwarding the message or suggest that they are signing something that will be submitted to a particular group. Urban legends usually have no negative effect aside from wasted network bandwidth, server resources and time. If you want to check the validity of an email, there are some web sites that provide information about hoaxes and urban legends: Urban Legends and Folklore - http://urbanlegends.about.com/;  Urban Legends Reference Pages - http://www.snopes.com/; Hoaxbusters - http://hoaxbusters.ciac.org/; TruthOrFiction.com - http://www.truthorfiction.com/; Symantec Security Response Hoaxes - http://www.symantec.com/avcenter/hoax.html; McAfee Security Virus Hoaxes - http://vil.mcafee.com/hoax.asp

 

10. Protect yourself while shopping online. Use and maintain anti-virus software, a firewall, and anti-spyware software. Keep software, particularly your web browser, up to date. Do business with reputable vendors. Take advantage of security features like secure passwords and encrypting information between your computer and the vendor’s website (look for the “lock” symbol in the browser or the website address beginning with “https” rather than “http”. Use a credit card rather than a debit card. Check your statements for any unusual or unauthorized activity.

 

Hopefully these tips will help you and those around you to have a happy holiday and reduce the risk of an unwelcome holiday event due to being uninformed. Please feel free to share these tips with your friends and family to help increase awareness and reduce risky behavior.

 

See the CERT Cyber Security Tips website for more information like this.

Nov 18 2008   1:15PM GMT

Did you see this? - Microsoft Exchange Online - Evaluating Software-plus-Services

Posted by: Troy Tate
administration, tools, Microsoft Windows, Development, CIO, DataCenter, email, Exchange 2007, Exchange, design, Microsoft, howto

New Infrastructure Planning and Design Guide-Now Available for Download

Exchange Online — Evaluating Software-plus-Services

 

The Infrastructure Planning and Design team has released a new guide, Exchange Online — Evaluating Software-plus-Services. Download the guide here.

 

In addition to the continuously growing collection of IPD guides focusing on architectural design configurations, Microsoft is now introducing a variation of these guides. This new type of guide is designed to help you make decisions about what’s best for your organization from both a business and a technology point of view.

Considering an online solution for your organization’s e-mail services? The Exchange Online — Evaluating Software-plus-Services guide provides a clear comparison of e-mail technologies across on-premises, standard hosting, and dedicated hosting scenarios. Use the guide as a framework for evaluating the technical feasibility of Microsoft Exchange Online. An overall scoring assessment is provided for each option, identifying key mail services and requirements for your organization. Understand the impact of adopting software-plus-services, weigh the importance of each topic to your organization, and learn which offering will serve you best.

Find other Infrastructure Planning and Design guides.

Oct 9 2008   3:00PM GMT

Alternatives to e-mail attachments - SharePoint is risky!

Posted by: Troy Tate
administration, Networking, Firewalls, Storage, Security, DataManagement, intellectual property, email, Data security, Policy, SharePoint, Exchange, design, website, risk, policy enforcement, vulnerability

I’m looking for some help on this topic and have posted a question to the ITKE community. Hopefully someone out there has had some experience with this service for your organization and can provide some valuable insight.

One group I participate in is a mailing list from SANS. If you have not attended a SANS event or education, then you should try to get to one of their events. They are one, if not, the premier non-vendor related security and systems administration group in the IT industry. I posed the same question to this peer group and have had some very good responses. Some suggestions for solutions have come back and include:

Microsoft Office SharePoint (http://www.microsoft.com/sharepoint/default.mspx)

OpenText – Livelink (http://www.opentext.com/2/sol-products/sol-pro-llecm10.htm)

Webex Connect – (http://webex.com/enterprise/index.html) (There are other flavors for small & medium business)

 Accellion -  http://www.accellion.com)

 

These are very interesting solutions and I will certainly be looking at all potential candidates. One thing that bothers me about the SharePoint option is its security capabilities. SharePoint is typically Microsoft Active Directory integrated. This has major security implications and in fact CSO magazine has posted a recent article on this topic. I recommend that you read the article and understand what risks the SharePoint solution may open for your organization.

Why Security Pros Hate Microsoft SharePoint

Microsoft’s SharePoint collaboration platform is all the rage in today’s business world, especially since third parties gained the ability to plug security holes. But managing it can still be a nightmare for IT security shops.

I am still looking for more references and ideas for this solution, so please share what you are doing for your organization and it will be much appreciated by me and other readers.

Sep 3 2008   7:28PM GMT

Did you see (listen to) this? - Podcast on preventing spam

Posted by: Troy Tate
administration, Security, tools, reporting, internet, DataManagement, IT education, spam, email, Data security, Policy, Exchange, anti-virus, Performance, howto, Metrics, risk, awareness, podcast

An audio podcast on how SPAM is generated along with an examination on the frameworks and technologies that help manage and reduce SPAM.

This may be a great tutorial for you and/or your users.

CERTStation Media - Spam-Prevent.mp3

I just ran my monthly e-mail statistics and these are the results:

97,000 msgs/day inbound

8,800 msgs/day delivered to end users - 9%

22,200 msgs/day quarantined as spam - 23%

66,000 msgs/day blocked as spam - 67%

This month had higher than normal quarantine activity. Quarantine has been running about 15% and blocking around 75%.  How does your mail stack up?

Thanks for your time. Let’s be good network citizens together & practice safe networking!

Aug 20 2008   6:19PM GMT

Did you see this? - Need some Exchange advice/support

Posted by: Troy Tate
administration, tools, Microsoft Windows, web, CIO, DataCenter, DataManagement, WWW, CA, spam, certificate authority, digital signatures, email, RSS, wiki, Exchange 2007, Outlook Web Access, Policy, Exchange, design, OWA, website, anti-virus, Performance, Powershell, howto, policy enforcement, awareness, blog, toolkit

Maybe you have already read my post about implementing new Exchange 2007 mailboxes for over 2000 users. If not… look here. So, as you see from this event, ongoing support for these global users on a new messaging system is going to be a real challenge.

I found a great blog posting with links to some excellent Exchange resources. Keep this in your toolkit for those times you just can’t find the answer elsewhere to those nagging Exchange problems. I see lots of other IT people struggling with this system and looking for support here at IT KnowledgeExchange.

Some other Exchange resources I recommend are:

Microsoft Exchange Server Resource Site

E-mail archiving

Seven ways to organize your e-mail

MessagingTalk.org - Portal for Microsoft Exchange Messaging & Collaboration

Thanks for your time. Let’s be good network citizens together & practice safe networking!

Jul 25 2008   12:58PM GMT

I know who I am - Do you know my name?

Posted by: Troy Tate
administration, Security, Microsoft Windows, Database, Development, Browsers, reporting, internet, DataCenter, DataManagement, WWW, email, wiki, Exchange 2007, Policy, Exchange, blogging, design, website, troubleshooting, howto, online identity, research, policy enforcement, awareness, subscriptions

If you read my previous post then you know we recently went through a major e-mail system migration. Part of that e-mail migration included moving from various naming conventions ( firstname at domain.com,  firstname.lastname at domain.com,  FirstInitialLastName at domain.com, etc.) to a single naming convention of  firstname.lastname at domain.com. Of course this was a huge undertaking and also a political move. One thing I am sure of is that the users will never understand the discussions taking place behind the scenes and will continue to take place about names of other non-user specific mailboxes like a project engineering team or an application mailbox.

Another thing which struck me during this process is that we netizens are identified by our e-mail address in many places on the web. Have you ever looked to see how many places you are identified by your e-mail address? I had to take some time and go out and change my e-mail address wherever the old one was in use. That is not a easy task let me tell you! First of all I went through the mailing lists I subscribe to. I went to their websites and tried to find the area to change my profile’s e-mail address. There are some sites where I could never find this and/or could not change it. So, webmasters & publishers…. please make it easier for your subscribers to modify their e-mail address or credentials! There is this need for companies that may get purchased or change names. There is the need for the users who change names when getting married or divorced…. this should not be as difficult as I found it to be.

In the end, I’m not sure what I will be missing out on when we go back and clean out all of the non-standard names which we will likely do by the end of the year.

Thanks for your time. Let’s be good network citizens together & practice safe networking!

Jul 25 2008   12:41PM GMT

2000 users - new mailboxes - one weekend - DONE!

Posted by: Troy Tate
administration, Networking, tools, Microsoft Windows, internet, CIO, DataCenter, DataManagement, CA, antivirus, certificate authority, digital signatures, email, Exchange 2007, Outlook Web Access, Exchange, design, OWA, Microsoft, troubleshooting, Powershell

Well, we did it! We implemented new mailboxes on Microsoft Exchange 2007 for over 2000 users in one weekend. Of course it took lots of planning, testing and blood, sweat, tears during the process, but we are now on one e-mail platform where there were at least 5 before. We had more domains than we needed and now the company is on one domain. We had to plan and provide for inbound messages still to the old domains.

The implementation was not without a couple of minor glitches and learning how users use the application. One glitch was a mistyped IP address. This prevented e-mail flow for a short period of time, however that is not a huge issue since SMTP servers will continue to retry sending messages. Another issue that was encountered was administrative rights to “shared” mailboxes like customer service or supply buyers.  This has now been resolved and users are getting full use from the system.

We still have some work to be done on things like:

• proactive system monitoring to detect issues before the users do;
• alternatives to sending large attachments (our attachment limit is 15MB);
• running Outlook Anywhere so a mobile user can attach to their mailbox without having to use VPN;
• supporting mobile devices like smartphones (our focus is on Windows Mobile v6 and up);
• user certificates using private PKI to allow for digital signatures and encryption.

So, as you see, work in IT never finishes… it just continues to grow as more services and systems are implemented and change happens. Please feel free to leave a comment if you would like more information about our implementation process and decisions we made along the way.

Thanks for your time. Let’s be good network citizens together & practice safe networking!

Jul 2 2008   2:33PM GMT

Did you see this? - The Great SPAM diet results are in

Posted by: Troy Tate
Security, Monitoring, reporting, spam, antivirus, email, Data security, malware, anti-virus, Performance, botnet, online identity, Metrics, research, awareness

See my previous post on The Great SPAM Diet. The results are now in and darkReading has the scoop.

McAfee’s Great Spam Experiment, Unplugged

Many spam messages sent to participants in the study were phishing emails or contained malware or links to malware-ridden sites

Did anyone doubt that these would be the results? Thanks for your time. Let’s be good network citizens together & practice safe networking!