Yottabytes: Storage and Disaster Recovery

Oct 31 2013   7:29PM GMT

A Year After Hurricane Sandy, What Has IT Learned?

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

Where were you a year ago? If you were on the East Coast, chances are you were dealing with Hurricane Sandy, a storm that was unprecedented not so much for its size and damage but for the way it seemed to target New York data centers. As we come up on the anniversary, what have we learned?

As you may recall, a number of data centers shut down abruptly due to losing power — which was often situated in the basement. While some companies got generators, others were stymied due to a lack of diesel fuel for them — or having to take diesel fuel up flights of stairs in a bucket brigade. Salt water and other debris also damaged equipment at some data centers.

Companies such as PEER 1 Hosting, which set up the famous bucket brigade, have been talking about what they’ve learned and offering advice to other companies that find themselves in similar situations.

As PEER 1’s Ryan Murphey notes, an important factor is people. “If you can’t ‘staff up’ before the storm, think about how you’ll get additional support to the facility if it’s needed,” he recommends such as by setting up emergency response teams near data centers.”

“Focus on the people, stupid,” agrees Barb Darrow of GigaOm. “Before Sandy, nobody seemed to imagine that highways, tunnels and subways could be out for days on end. Now there have to be plans in place for how personnel can get to the affected area, and for how other personnel can work remotely as effectively as possible.”

Murphey also suggests stocking up on equipment and setting up contracts ahead of time for items such as fuel. For example, the organization now has a pump that can reach the 18th floor, as well as fuel hoses on-site — which fit the generators. And for stored diesel, organizations need to set up filters and other systems to remove any potential water from the fuel, which could keep generators from running, warns Alastair Trower in Data Center Knowledge.

At the same time, Murphey notes that stuff happens and you can’t always count on being able to get what you need when you need it, contracts or no.

Other people and entities are also making preparations. For example, the state of New York is setting up a strategic gasoline reserve of as much as 3 million gallons, though it isn’t clear how much of that would be regular unleaded gasoline for vehicles vs. diesel fuel that could be used in generators. The New York Stock Exchange has devised a plan takes advantage of the company’s data centers in New Jersey and in Chicago.

Some organizations are also working on getting better, more site-specific weather prediction in place so they have a better idea of what can happen in their own locations, Darrow writes.

At the same time, some things haven’t changed. While some organizations are looking at backup data centers in less hurricane-prone regions, such as Omaha, Nebraska, Darrow writes, an April survey found that two-thirds of data center managers would rather see the data center in the city where they worked — and even potential alternative locations tended to be vulnerable to natural disasters themselves. The most important reasons given for data center expansion, Digital Realty noted, were (in order of priority) the need for increased security, energy efficiency, new applications/services, and more space. It isn’t clear whether “Not Being Under Water,” “Not Being on Fire,” or other variations on “Not Being Destroyed” were choices.

Time-critical organizations such as stock exchanges and other financial companies are also concerned about latency, or the additional seconds involved in getting data from places like Nebraska rather than New York.

And Murphey notes that, despite his company’s experience, it still is unlikely to put its electrical equipment anyplace other than the basement. First, real estate on higher levels is more expensive. Second, there are structural issues associated with supporting the weight of the equipment, as well as practical issues with storing diesel fuel anywhere other than a basement.

If nothing else, maybe you’d better stock up on buckets.

1  Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • Tony F
    The trouble as I see it is, that even the best defined, designed, and constructed facilities will fail at some time, for some reason. Generally I have found that in the bloom of life there are good intentions to continually monitor, assess, test and verify emergency systems. As time goes by, and nothing fails, the good intentions are replaced with apathy. Things other than planned testing and verification become more important. Leaders often point to failures that occurred when testing goes wrong and brings down the systems, and use this as an excuse not to bother with live tests of power system backups. Acts of God rarely are the sole reason for a system failure, but at least the costs can be recouped from insurance claims, whereas "routine maintenance" cannot be. Sad but true. So, take your chances by all means, but don't expect a different result. PS: I have had 40 years experience with power and UPS, military and civil. "Full Noise" 100% Live testing will uncover all of the issues you identified. Design backups and bypass systems properly to overcome Live Testing fail syndrome.
    50 pointsBadges:

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: