The Troposphere

Jun 30 2009   7:43PM GMT

Rackspace falls over in Dallas, tweets the whole thing

CarlBrooks Carl Brooks Profile: CarlBrooks

More than 24 hours after users began reporting Rackspace hosted services were unresponsive, and the main site went dark, Rackspace has possibly set a new record for transparency and accountability, if not customer satisfaction, by tirelessly tweeting the entire episode.

They also ran to update the company blog (how droll, so Web 1.0, right?) and blamed power outages in their Dallas data center.

For additional amusment, see the vulturescompetitors flock to #rackspacefail

An official statement has not been made and a request for comment has gone unanswered to date; so the root of the problem is still to be determined. Amazon’s recent calamity was exacerbated by lightning unaccountably penetrating a supposedly world-class data center- it’ll be interesting to see if Rackspace’s facilities have similar flaws.

UPDATE: Rackspace HAS NOT released their incident report, but it’s out in the wild. According to the report, which I won’t post, but will summarize from, since the content is fair game at this point: a mains breaker flipped and one line of generator backups had “excitation failure” which means they didn’t start up properly. Subsequently 3 banks of UPS batteries bled out and slammed a bunch of racks — which means they weren’t charging properly or worse, underdesigned for the load.

What this means in the simplest possible terms: “Heads Will Roll”. Between this and Amazon’s air-to-ground static electricity adventure, data center types are wagging their grimy, highly redundant fingers as hard as possible at these incidents.

UPDATE: the incident report is now public

2  Comments on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • Kr8tr
    Thanks for your post. We found the Twitter account to be very useful last night to reach large numbers of customers quickly. It is an essential tool. As for transparency - we believe in telling customers the truth - even when it hurts. We balance that with knowing that we need to be accurate. Last night we "tweeted" what we could confirm. Today the process of root cause analysis has started and we are working with our power infrastructure vendors to make that determination. When we do, we will make that information available on our blog. Disappointing customers with outages is bad enough. Not be honest with them is a recipe for a "going out of business sale". Rob La Gesse Director of Customer Development Rackspace 210-845-4440
    0 pointsBadges:
  • Brianw168
    Thanks for your comments! Naturally, I'm in favor of knowing as much detail as possible, always. Please don't hesitate to reach out. You can always reach me by email at
    40 pointsBadges:

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: