Google

Oct 15 2009   12:51PM GMT

Google’s Postini services restored - cascading issues caused message delivery issues

Posted by: Troy Tate
Google, cloud services, saas, antispam, antivirus, service outage, service level, incident report, root cause analysis, corrective actions

I recently posted about Google’s Postini - cloud email security service - delivery issues. This is a follow-on post about the incident root cause analysis and corrective actions. Maybe there’s some lessons learned here that you can use in your organization’s service delivery.

The impact on customer email services lasted more than 24 hours while Postini engineers worked to resolve the issues. So, this was not an insignificant event. During this period, messages were delayed and users were not able to get to their quarantines to release messages trapped by filters. Administrators were also unable to access the administration console. The Postini support portal was unreachable at times due to the high volume of users trying to get updates on the event. The support phone line queues were very long and it took a long time to reach a support agent. Nothing like this has happened before in all of the years we have been a Postini customer.

I just received the incident report about the service disruption and wanted to share some of the information with IT Trenches readers. Continued »

Oct 13 2009   7:59PM GMT

Google’s Postini - cloud email security service - delivery issues

Posted by: Troy Tate
Google, cloud services, saas, antispam, antivirus, service outage, service level

Since very early today, US Eastern Daylight Time, Google’s Postini services have been experiencing some service issues. It is unknown as of this writing as to the cause or full scope of the issue. However, when logging into the Postini support portal, an administrator is given the following status indicators:

We have been Postini customers over 4 years now and this is the first time an outage like this has happened. It’s not a full outage as messages are still coming in although at a trickling rate rather than normal expected volumes. This outage is so bad that my ability to login to the support portal is impacted. I receive either an internal 500 server error or “Too many connectionsCould Not Select DB”. A recent update notification said that a secondary Postini secondary data center has been enabled.

The recent GMAIL outage raised some concerns about cloud computing. I wonder if today’s Google Postini outage is a symptom of some deeper Google service delivery problem.

Thanks for reading & let’s continue to be good network citizens! Hopefully you are not trying to send me any messages, who knows how long it might take for the message to reach me today. Otherwise, let me know what you think here in the comments.

Sep 16 2009   6:41PM GMT

Google search results serve up malware - I’ve had the crime of my life

Posted by: Troy Tate
malware, Google, search results, malicious software, drive-by attack, browser security, information security, software security, software

Well, that may not be news to you. However, there is a recent trend in malware propagation that uses Google as the portal to deliver payloads to visitors. Unsuspecting users go to Google and search for topics such as Patrick Swayze’s death or the controversy about Serena Williams cursing at the line judge in her recent US Open tennis match. When a user selects one of the Google search results and visits the page, malware is downloaded to the client computer since the referrer is Google. However, if someone were to just visit the page on their own or through another search engine, the website does not serve up malicious software.

For more information see this Register.com article Swayze death exploited to serve up fake anti-virus - I’ve had the crime of my life. Seems like malware is bombarding us from all directions now. You can’t even trust ads on the NY Times these days.

Thanks for reading & let’s continue to be good network citizens!

Feb 3 2009   7:41PM GMT

Will Microsoft ever get search right?

Posted by: Troy Tate
Microsoft, Google, search, Live search, Microsoft Live, documentation, Powershell

I am looking for some documentation on Powershell to better understand how to use it. Per Wikipedia: Windows PowerShell is an extensible command-line shell and associated scripting language from Microsoft. So, I went to the Microsoft.com home page at www.microsoft.com. I typed powershell into the Search field at the very top of the page. I clicked the magnifying glass… waited a few seconds… and NOTHING was returned! So, I clicked on the Live Search option and 39,500 results were returned. So, now when I go to the main Microsoft page and then enter powershell into the same search term field as before and press Enter, the Live search results get returned - filtered for Microsoft.com only. It seems like my Live search excursion “woke up” the main Microsoft website search into knowing some powershell content does exist at Microsoft.com

I have often been frustrated in the past when searching Microsoft support using the exact error or event code from a Microsoft system or application log and nothing gets returned. It just seems like Microsoft is still missing the boat when it comes to search.

So, I guess I will continue to Google for Microsoft support information until I can see that Microsoft is better able to search their own website from their homepage.

Thanks for reading & let’s be good network citizens out there!

Oct 27 2008   8:52PM GMT

Did you see this? - (Wire)Sharkfest 2008 videos - including Vint Cerf - now available

Posted by: Troy Tate
Networking, forensics, Security, tools, Microsoft Windows, Linux, Monitoring, web, reporting, Google, internet, IT education, WAN, LAN, performance monitoring, troubleshooting, Performance, Network TAPs, howto, network analysis, Metrics, wireshark, packet capture, research, education, toolkit, man-in-the-middle, analysis

Checkout the Sharkfest 2008 videos at LoveMyTool.com. If you use Wireshark or want to learn network troubleshooting, this is one of the best resources you can have in your toolkit. The videos will give you a better understanding of this tool and other tools out there.

There is even a video of Dr. Vinton G. Cerf, vice president and Chief Internet Evangelist for Google. He is responsible for identifying new enabling technologies and applications on the Internet and other platforms for the company. Widely known as a “Father of the Internet,” Vint is the co-designer with Robert Kahn of TCP/IP protocols and basic architecture of the Internet.

Have a great day and thanks for stopping by!

Jul 8 2008   5:12PM GMT

Browser warnings - Danger Will Robinson! - or did it just cry “Wolf!”?

Posted by: Troy Tate
forensics, Security, Development, web, reporting, Google, WWW, IT education, antivirus, Data security, malware, Policy, Firefox, website, anti-virus, honeypot, botnet, online identity, Metrics, honeynet, policy enforcement, awareness

I sometimes browse the internet using Firefox. I say sometimes because Internet Explorer is the standard browser at my company and Firefox is not supported by IT. Well, since I work in IT, sometimes you have to test things on behalf of users and also to see how certain sites are different depending on the client browser.

Well, I recently upgraded Firefox to v3. It does seem much better than v2 although some of my useful addins are now broken (when will YSlow get fixed for v3?). One of the new features of Firefox v3 is the ability to report to the user if the visited website is a known potential malware site. This is a good feature! It provides the user with some useful information and education about the dangers on the internet. However, how accurate is this feature? What if you are visiting a trusted website that you frequently visit and now get this message?

For your information, this is the message that you will see when you attempt to visit a site deemed as risky.

Reported Attack Site!

This web site at certification.xxxxxxx.org has been reported as an attack site and has been blocked based on your security preferences.

Attack sites try to install programs that steal private information, use your computer to attack others, or damage your system.

Some attack sites intentionally distribute harmful software, but many are compromised without the knowledge or permission of their owners.

I blanked out the actual website address above. However, those of you with a bit of detective in you are likely going to figure it out.

What is interesting about this particular warning message is that it is referring to a website that has security as a guiding principle. When you see this message in Firefox, you have three options presented:

• Get me out of here!
• Why was this site blocked?
• Ignore this warning - in very tiny print at bottom of message.

I was curious as to why this site would be considered as a danger. I clicked on the Why was this site blocked? option. The report I received was interesting and as I mentioned earlier, could this be an example of someone crying “Wolf!”?

The report was as follows:

What is the current listing status for certification.xxxxxxx.org

Site is listed as suspicious - visiting this web site may harm your computer.

Part of this site was listed for suspicious activity 1 time(s) over the past 90 days.

What happened when Google visited this site?

Of the 6 pages we tested on the site over the past 90 days, 1 page(s) resulted in malicious software being downloaded and installed without user consent. The last time Google visited this site was on 07/06/2008, and the last time suspicious content was found on this site was on 07/06/2008.

Malicious software includes 1 scripting exploit(s). Successful infection resulted in an average of 3 new processes on the target machine.

Malicious software is hosted on 3 domain(s), including lokriet.com, clrbbd.com, catdbw.mobi.

1 domain(s) appear to be functioning as intermediaries for distributing malware to visitors of this site, including catdbw.mobi.

Has this site acted as an intermediary resulting in further distribution of malware?

Over the past 90 days, certification.xxxxxxx.org did not appear to function as an intermediary for the infection of any sites.

Has this site hosted malware?

No, this site has not hosted malicious software over the past 90 days.

How did this happen?

In some cases, third parties can add malicious code to legitimate sites, which would cause us to show the warning message.

Next steps:

• Return to the previous page.
• If you are the owner of this web site, you can request a review of your site using Google Webmaster Tools. More information about the review process is available in Google’s Webmaster Help Center.

This is great educational stuff, but did it really happen to this particular website? I don’t know, but apparently Google does. With the report of just one incident, does it make this site really worth the notification? How many incidents should it take before a site is considered malicious and who determines what malicious is?

Just something else to mull over in your copious time as you go perusing websites in Firefox.

Thanks for your time. Let’s be good network citizens together & practice safe networking!