The impact on customer email services lasted more than 24 hours while Postini engineers worked to resolve the issues. So, this was not an insignificant event. During this period, messages were delayed and users were not able to get to their quarantines to release messages trapped by filters. Administrators were also unable to access the administration console. The Postini support portal was unreachable at times due to the high volume of users trying to get updates on the event. The support phone line queues were very long and it took a long time to reach a support agent. Nothing like this has happened before in all of the years we have been a Postini customer.

I just received the incident report about the service disruption and wanted to share some of the information with IT Trenches readers.

The event started at about 6:25 PM GMT Tuesday, October 13. At this time customers began experiencing severe mail delays and disruption. Some senders were receiving delivery failure notifications after multiple resend attempts failed. About an hour later, automated monitoring systems detected the mail flow issues and traffic was automatically failed over to a secondary data center. Message flow was also poor through the secondary data center.

Trying to improve message flow, message traffic was directed across both primary and secondary data centers. Also, in an attempt to reduce impact on data center resources, access to the administration and other web consoles was disabled. The engineers were able to eventually discover the causes of the message flow issues.

• A message filter update inadvertantly caused performance issues.
• Unusual malformed messages caused increased scan processing and in tandem with the bad message filter update caused issues with mail delivery.
• Processing capacity was reduced due to a power supply failure on a database storage system. This increased latency in message processing.

As you can see, this event was caused by a series of issues. The hardware was repaired and the filter update was revoked, but not before a lot of messages were either deferred or not delivered.

The corrective actions to prevent future outages due to similar conditions include:

• Create a standard procedure for reverting message filter updates. Isn’t it always a good idea to be able to back out updates?
• Improve monitoring of database server power failures. Apparently this power failure was not detected by their current monitoring process.
• Improve communication with customers during service outage events. This would help relieve some frustration and help customers understand the severity and scope of the outage.

Fortunately email services have now returned to normal. However, during this 24 hour period, there was a high level of frustration and concern about how to work around the impact on email delivery. I think most IT organizations can learn some lessons and improve service delivery when reading of incidents like this and seeing the lessons learned. I know my team will have some discussions around this event and work to improve the resiliency of service delivery.

Thanks for reading & let’s continue to be good network citizens!

Postini system status on October 13, 2009

We have been Postini customers over 4 years now and this is the first time an outage like this has happened. It’s not a full outage as messages are still coming in although at a trickling rate rather than normal expected volumes. This outage is so bad that my ability to login to the support portal is impacted. I receive either an internal 500 server error or “Too many connectionsCould Not Select DB”. A recent update notification said that a secondary Postini secondary data center has been enabled.

The recent GMAIL outage raised some concerns about cloud computing. I wonder if today’s Google Postini outage is a symptom of some deeper Google service delivery problem.

Thanks for reading & let’s continue to be good network citizens! Hopefully you are not trying to send me any messages, who knows how long it might take for the message to reach me today. Otherwise, let me know what you think here in the comments.

According to Wikipedia, the three curses are:

• May you live in interesting times.
• May you come to the attention of those in authority (sometimes rendered May the government be aware of you)
• May you find what you are looking for

These curses have no time frame attached, so they could be true yesterday, today or tomorrow. The first curse about living in interesting times is very true for most IT shops. The challenges never seem to end but the funds do. The business requirements are fuzzy but the support requirements are real. What makes for interesting times in your role and your organization?

The second curse could be attached to information security services. It most assuredly should apply to those who employ botnets and trojans for unrighteous financial gain. The most recent reports of trojans and bots that can dynamically alter web pages during a banking transaction are very scary. Fortunately, information security professionals are aware of the threat and are working on addressing it.

The third curse may be applicable to those who support networks and need to answer the ever present question about “why is the network slow?” It seems like we are always pressed into identifying the reasons for slow application and service performance without having the toolset to do proper diagosis and troubleshooting. It then becomes a challenge to find what we are looking for.

Maybe IT is a cursed profession after all. What do you think? Is there a more cursed profession?

Thanks for reading and let’s continue to be good network citizens!