At least, that’s the observation GigaOm’s Barb Darrow recently made of a Digital Realty survey of managers’ data center plans. “Despite the angst that superstorm Sandy and Hurricane Irene caused data center providers and their customers in the New York metro area over the last two years, businesses still want to expand their data center capacity in that low-lying, suddenly storm-surge-prone area,” she writes.

Apparently the familiar is more comfortable than the unknown. According to the survey, two-thirds of respondents would rather see the data center in the city where they work, and target locations, other than New York were Los Angeles (earthquakes and fires), Dallas (tornadoes), Chicago (blizzards), the San Francisco Bay Area (earthquakes again), and Phoenix.

“Of course, when two 100-year storms hit the same area within two years of each other, you might start evaluating new locations,” Darrow writes. “Then the question becomes what areas are not susceptible to natural disasters,” echoing what she wrote at the end of last year about Fidelity Investments setting up a data center in the far-from-water, far-from-earthquakes, yet tornado-prone Omaha, Neb.

And recall that last June, just a little ol’ thunderstorm took out Amazon Web Services.

The most important reasons given for data center expansion, Digital Realty notes, are (in order of priority) the need for increased security, energy efficiency, new applications/services, and more space. It isn’t clear whether “Not Being Under Water,” “Not Being on Fire,” or other variations on “Not Being Destroyed” were choices. (To be fair, when respondents were asked to provide multiple reasons for expanding data centers, “disaster recovery/Sarbanes-Oxley” came in second after “security.”)

The other interesting factor that cropped up was the data sovereignty issue. As you may recall, this is becoming more of a thing as an increasing number of countries, including the U.S., claim some degree of access to data stored on their shores, regardless of the data’s country of origin or the residence of the company that owns it.

“Geopolitical location of data” was extremely important to 50 percent of respondees, though it was slightly beaten out by data authenticity and security, physical security, control over the facility, and the total cost of the technology. “The two factors in data sovereignty (data authenticity/security and geopolitical/legal location) are the most important considerations for 29% of the respondents,” Digital Realty noted.

Physical security — that is, Not Being Underwater, Not Being on Fire, and Not Being Destroyed — was apparently a consideration for only 14 percent of respondees.

While several data centers were knocked offline due to flooding — most notably Datagram, which hosts Gawker, Gizmodo, Buzzfeed, The Huffington Post, and Media — many stayed on, often through generators running on diesel fuel. (The New York Times – which criticized data centers just last month about their use of diesel backup generators — was strangely silent on the subject this week.)

The problem then switched to getting fuel delivered, since typically generators would keep only three days’ fuel on-site. That time, however, did give users of those data centers time to find other alternatives.

Though data centers went through extensive preparation, the ones that were knocked offline typically had either the data center, or the fuel systems, or both, in the basement, which flooded. Some sites went offline after they weren’t able to get fuel delivered to the island of Manhattan.

“The situation shows that in many ways, Lower Manhattan is one terrible place to put a data center,” noted Cade Metz in Wired. On the other hand, he said, data centers need to be near where the business action is to provide low-latency data transmission.

In one case, Fog Creek Software and Squarespace  – with fuel pumps in the flooded basement and a generator on the 17th floor — employees used a bucket brigade to get fuel up the stairs to run the generator.

Other customers were migrated to cloud services such as Amazon Web Services — ironically, since it has suffered a number of outages over the past few months.

Not only was the Internet relatively resilient to the hurricane itself, but to the increased load of all the East Coast people who stayed at home, watched Netflix, and chatted with their loved ones over Skype.

Numerous other posts over the years have described how data centers have handled earthquakes, wildfires, and other disasters. That said, there’s a few other lessons to have been learned from Sandy:

• Don’t put data centers, or their diesel backup, in the basement. On the other hand, it’s not like you want it up over your head, either — especially if you end up needing to do a bucket brigade to the roof.
• Have data centers, or backup data centers, located in separate geographic regions.
• Plan, plan, plan. And don’t wait until an actual emergency to test the plan. “You can’t wait ’til folks’ hair is on fire to plan these things,”  Shannon Snowden, a data center professional who is now senior technical marketing architect for Zerto, a company with technology that helps companies move and failover applications, told GigaOm. “What you should be doing from the data center perspective is [always] make sure the power has been tested, that you can fail over to generators, that those those generators are tested to make sure they’re functional and that they have enough fuel,” he said.

Finally, the vendor I lambasted in April for exploiting natural disasters to promote their product was at it again — press release issued right on schedule on Monday afternoon.

Virtualization. In talking about how under-utilized data center servers are, and in appearing to limiting himself to less than state-of-the-art facilities, Glanz failed to notice how prevalent virtualization is becoming, which enables an organization to set up numerous “virtual servers” inside a physical server — which, in the process, results in much higher utilization.  ”[V]irtualized systems can be easily run at greater than 50% utilization rates, and cloud systems at greater than 70%,” writes Clive Longbottom in SearchDataCenter.

“[I]n many cases the physical “server” doesn’t even exist since everyone doing web at scale makes extensive use of virtualization, either by virtualizing at the OS level and running multiple virtual machines (in which case, yes, perhaps that one machine is bigger than a desktop, but it runs several actual server processes in it) or distributing the processing and storage at a more fine-grained level,” writes Diego Doval in his critique of the New York Times piece. “There’s no longer a 1-1 correlation between “server” and “machine,” and, increasingly, “servers” are being replaced by services.”

“Although the article mentions virtualization and the cloud as possible solutions to improve power utilization, VMware is not mentioned,” agrees Dan Woods in Forbes‘ critique of the piece. “If the reporter talked to VMware or visited their web site, he would have found massive amounts of material that documents how thousands of data centers are using virtualization to increase server utilization.”

Storage. Similarly, Glanz appeared to not be aware of advances in storage technology, even though some of them are taking place in the very data centers he lambasted in his articles. In Prineville, Ore., for example, not all that far from the Quincy, Wash., data centers he criticized, Facebook is working on designing its own storage to eliminate unnecessary parts, as well as setting up low-cost slow-access storage that is spun down most of the time.

Facebook — which does this research precisely because of the economies of scale in its massive data centers — is making similar advances in servers. Moreover, the company’s OpenCompute initiative is releasing all these advances to the computer industry in general to help it take advantage of them, too.

In addition, Glanz focused on the “spinning disks” of the storage systems, apparently not realizing that increasingly organizations like eBay are moving to solid-state “flash” storage technology that use much less power.

Also, storage just isn’t as big a deal as it used to be and as the story makes out. “A Mr Burton from EMC lets slip that the NYSE ‘produces up to 2,000 gigabytes of data per day that must be stored for years’,” reports Ian Bitterlin of Data Center Dynamics in its critique of the New York Times piece. “A big deal?  No, not really, since a 2TB (2,000 gigabytes) hard-drive costs $200 – less than a Wall Street trader spends on lunch!”

Disaster recovery. Glanz also criticized data centers for redundancy — particularly their having diesel generators on-site to deal with power failures — apparently not realizing that such redundancy is necessary to make sure the data centers stay up.

And yet, even with all this redundancy, there have been a number of well-publicized data center failures in recent months caused by events as mundane as a thunderstorm. Such outages can cost up to $200,000 per hour for a single company — and a data center such as Amazon’s can service multiple companies. If anything, one might argue that the costs of downtime require more redundancy, not less.

Of course it’s important to ensure that data centers are making efficient use of power, but it’s also important to understand the context.

Remember that?

It turns out, not so much.

Major Internet services such as Netflix, Pinterest, and Instagram were taken offline on Friday night due to thunderstorms that took out power to the Amazon Web Services site in Virginia.

“Amazon’s Cloud services status page was full of power-related error messages,” wrote MSNBC’s Bob Sullivan. “Amazon’s ElastiCache, for example, indicated that starting at 8:43 p.m., the service was “affected by a power event.”  At 9:25 p.m., this message was posted: “We can confirm that a large number of cache clusters are impaired. We are actively working on recovering them.”"

But, oh, Sullivan continued, this wasn’t really Amazon’s fault, because the storm was just so very bad. Why Amazon didn’t have backups or replicated copies elsewhere on the network, he didn’t say. (Apparently the problem may have been a routing issue.)

“Outages like this morning’s are a reminder of how fragile, still, our digital architectures actually are,” intoned The Atlantic‘s Megan Garber. “As much as we try to bolster them against the elements, they are made of sand, not stone. Buildings can be brought down by storms; but so, today reminds us, can their digital counterparts. Even the structures that lack structures can be torn by nature’s whims. That is, in its way, terrifying. And yet — here’s the other sliver — it is also, just a tiny bit, reassuring. No matter how advanced we get, today reminds us, nature will always be one step ahead.”

This is probably not a great consolation to those companies that have moved their operations to the cloud because they were assured it would still be there in a disaster. And this is a disaster? A thunderstorm (albeit one that has caused at least a dozen deaths)? Are those companies feeling “reassured” today by discovering that their disaster recovery systems are, in fact, vulnerable to an outage that might be in a completely different part of the country?

What happens if there’s an earthquake, hurricane, or some more severe natural disaster? If the Red Cross or FEMA loses its connectivity because it depended on the cloud, will its managers philosophically fold their hands and talk about how in the great scheme of things this shows just how little we all are? Will the people asking to be saved or helped see it this way?

Instead, hopefully Amazon and other cloud providers will take this as a wake-up call before hurricane season gets going, and ensure that the virtual cloud can stand up to the real thing.

Last summer, the data center at the Los Alamos National Lab in New Mexico was surrounded on two sides by a 60,000-acre fire, while in 2007, the data center at Pepperdine University was threatened by a 1,200-acre fire in Malibu, Calif., that came within 100 feet of it.

More recently, we’ve had the fires in Colorado, which led one data center manager to post to Slashdot asking for advice. While there were the usual number of jokes, tangents, and speculation about his motives, there was also useful advice for data center managers as fire season approaches. (And most of this advice is useful for disasters in general.)

• Have a disaster recovery plan and make sure it’s updated — for example, are all the contacts and their phone numbers correct? “DR plans are a living document that should be updated for every significant change to your infrastructure,” noted Slashdot user Macgrrl. “They should have an annual ‘trial run’ to see if they work. The worst time to find out your DR plan doesn’t work is in an actual disaster event.”
• Priorities are people, data, equipment.
• This is one advantage of using the cloud — data is by definition offsite.
• Perform regular offsite backups.
• Make sure the network is documented and up-to-date, with the documentation available electronically and offsite. Save configuration settings to a text file and store it both electronically and on paper.
• Label everything — including AC adapters to keep from zapping things afterwards.
• Take pictures of the cabling for documentation purposes.
• If you have to save equipment, focus on disk drives and servers first. And keep in mind that insurance that reimburses for equipment lost in a fire might not reimburse for equipment damaged in a bugout.
• To save time, use wire cutters to disconnect cables (*not* power cables!).
• Cover things you’ve left behind with plastic or trash bags to help protect them from water and smoke.
• Consider setting up your data center to be portable in the first place — set it up in a shipping container, put racks on wheels (and make sure doors are wide enough to move them through and you have a forklift if necessary), use quick-disconnect hard drive enclosures, buy a truck or van to store onsite, etc.

“Any disaster plan should be able to cope with ‘and then a giant foot appeared above the building and squished it flat,’” noted Slashdot user GirlInTraining. “Yours should be no different. It might not be a wild fire that threatens your servers… it could be a UPS that shorts out, or a tornado, flood, a failed fire suppression unit, or simple human incompetence.”

Take for example Microsoft, which came under a barrage of criticism during the Japan earthquake last year for offering to donate a dollar for every retweet of its message promoting its Bing search engine; after being attacked, the company swiftly backpedaled and just made a straight donation, no retweeting required.

That’s why I hesitated at posting about this upcoming vendor — my initial reaction was negative, and it’s only several days later, after checking out the coverage elsewhere, that I can look at its announcement more objectively.

With Tornado Alley this spring looking more like Tornado Interstate, and numerous regions and businesses affected, it’s not surprising that some vendors would want to use it as a news hook — though perhaps waiting until the twisters had actually stopped forming might have been more tasteful timing.

In response to the severe damage caused by tornadoes touching down in the Dallas area, Nirvanix, the leading provider of enterprise-class cloud storage services, today announced that it is expanding its Disaster Avoidance Program to customers currently storing data in its Node 3 data center in Dallas enabling them to exercise the option of moving their data to other locations in the Nirvanix Cloud Storage Network—either on a temporary or full-time basis—free of charge.”

(Ironically, the Storage Networking World show was being held in Dallas at the same time and was itself disrupted by the severe weather, although Nirvanix did not appear to attend that event.)

On its face, this is a reasonable offer. Users in a disaster area can store their data outside the area. Great. So what’s the problem?

Perhaps it’s the use of the phrase “the leading provider of enterprise-class cloud storage services” in the first sentence. Really, did Dallas people need to have that pointed out to them just then?

Perhaps it’s the Johnny-on-the-spot nature of the announcement, which was issued the same day the tornadoes actually occurred. By Googling, one can ascertain that such an announcement is not unusual for Nirvanix, with the company making similar offers during disasters such as the Japan earthquake and Hurricane Irene.  Pull out the boilerplate press release, drop in the name of the disaster and its location, and you’re good to go.

One does wonder at what point the trigger occurs to send out such a release. After a certain amount of property damage occurs or a certain number of people are killed? Does it depend on how many customers Nirvanix has in the affected area? Will Nirvanix issue a similar press release and offer this week regarding the Midwest tornadoes, or did they not come up to snuff?

While some disasters, such as the Japan earthquake, are unpredictable, it’s no secret to anybody that we have tornadoes in the spring and hurricanes in the fall. If one wants to offer such a service to one’s clients, how about issuing a generic press release at the beginning of the disaster seasons so that it looks less like a vendor exploiting a particular tragedy? The anniversary of the Great San Francisco earthquake is coming up, too; that might be a marketing opportunity as well that is far enough removed from actual events and tragedies that it won’t appear so opportunistic.

Disaster recovery?

Yep. A new data center, Granite Block Global Data Center Inc., which is intended to provide application hosting, colocation and disaster recovery services to businesses in the financial services, healthcare and high technology industries, has opened up in a former textile mill in the venerable Massachusetts city.

And “Granite Block” isn’t just a colorful New England name. The walls of the facility are made of four-foot-thick solid granite, which the company says is virtually impenetrable.

Repurposing former industrial facilities into data centers has been an increasing trend in the past few years. Companies and cities looking for data center facilities are realizing that their business-convenient downtowns are often already the home of large, frequently empty, buildings. And not only do the buildings typically have a great deal of power capacity but companies can often get tax breaks for investing in what are considered distressed urban areas.  Around the world, data centers are now moving into paper mills, tanneries, department stores, warehouses, and even churches and bomb shelters.

Now a 163,000-sq. ft. former textile mill in Fall River — a city previously famous for being the home of accused ax murderer Lizzie Borden — is joining them, with Granite Block investing $5 million in the project, amid hopes that other abandoned textile mills in the New England area could serve the same purpose. Open Cape, a fiber company, has already said it intends to move into the facility — which will also give other tenants an additional connectivity option.

In addition to making use of the large power capacity the building already has, Granite Block intends to install wind turbines for energy and 400-foot artisan wells for geothermal cooling. The company has also added two 2-megawatt generators and says it has enough fuel on-site to run them for six days.

The facility is starting with 20 jobs, but could bring up to 70 jobs to the area, as well as visitors. A similar data center in Boston, which is just 40,000 square feet, brings in 500 to 700 visitors monthly, meaning the Fall River site could see three times that amount, said Granite Block President Roland Patenaude.

“You may laugh now, but when it happens you’ll be happy you read this,” the circular warns. It goes on to describe the zombie threat, and what people can do to be prepared in the event of a zombie apocalypse.

No, it wasn’t issued on April Fool’s Day, and no, it wasn’t a joke. Well, sort of. Does the CDC really expect a zombie apocalypse anytime soon? No, probably not. But its purpose in creating the alert was deadly serious, and it’s something we can all take lessons from in developing disaster recovery plans:

• It got attention. The CDC had more than 1,000 articles published about the Zombie Apocalypse circular, and got so many Internet hits that its server crashed. Even so, the zombie circular got 60,000 hits in that first day. In contrast, a typical CDC blog post might get between 1,000 and 3,000 hits, and the most traffic on record had been a post that saw around 10,000 visits, a CDC spokesman told Reuters.
• Disasters are all the same. Seriously, there’s not going to be that much difference in preparing for an earthquake vs. preparing for a pandemic vs. preparing for…well, a zombie apocalypse. The CDC suggested list of preparations included getting together food and water supplies, making arrangements to meet with loved ones, etc. — all the same sorts of things you’d do if you were making *any* disaster plan. True, some of it was with the theme (“Plan your evacuation route. When zombies are hungry they won’t stop until they get food (i.e., brains), which means you need to get out of town fast!”) but generally the suggestions were generic and could apply to any disaster.
• Bypasses the Critical Censor. As a former resident of the Bay Area, I can testify that people who live in an area prone to disasters can develop a certain kind of blinders. Yes, we all knew there’d be an earthquake sometime, and some of us even had some preparation, but in general people don’t worry about it all the time. ”Human beings are hard-wired to believe in their heart and soul that disasters don’t happen and won’t happen to them,” Dennis Mileti, a retired University of Colorado sociology professor and researcher, told MSN Money. Writing the circular about zombies allowed people to read it and absorb the lessons without them getting into the whole “Oh, I know all that, I don’t need that, lalala” reaction that a more realistic disaster could have elicited. (In point of fact, the CDC wrote the circular to help people prepare for hurricanes.)

It seems pretty self-evident that one could apply these same lessons to writing a disaster recovery plan — just write all the same preparations, but wrap it into another event that could get people’s attention and make them laugh a little as they read it.

I hear the next Rapture is scheduled for October 21.

Oh, yeah, and there was a Royal Wedding.

But ten minutes before that (not that they were trying to hide anything, of course), Amazon also released the post-mortem of its extended Elastic Compute Cloud (EC2) outage of the previous week.

In case you were under a rock, a number of major computer sites — including foursquare, Reddit, and Quota — were down for a day, sometimes more, on April 21, due to a problem with Amazon’s web hosting business. It wasn’t until Monday or Tuesday of this week that all the sites really recovered.

If you’re familiar with the concept of “thrashing,” where a too-full hard disk or computer memory is so busy trying to find places to work that it doesn’t get anything done, that’s basically what happened to Amazon, on a mammoth scale. Due to a configuration problem, the cloud went down, and the first thing all the servers did when they came up was try to re-mirror themselves — which they couldn’t do because all the other servers that were up were trying to do the same thing. The actual summary goes into a lot more detail, if you really want to know, but that’s basically it.

So now the Internet is seeing a storm of a different kind: A pundit storm where people talk about 1) What It All Means and 2) Where We Go From Here and 3) Could It Happen Again?

1) S*** happens. 2) Don’t have a single point of failure, duh. 3) Of course.

Oh, you wanted more detail?

What it all means is that people are human and machines are stupid. This does not change, and will not change. Count on it. Problems happen. Then we institute new systems that help us protect against the most recent problem, and wait for a new problem to happen.

You know, like the TSA.

Where We Go From Here is that Amazon is instituting a number of changes in processes and procedures, both human and machine, that are intended to keep this from happening again.

Organizations that use the cloud — anybody’s cloud, not just Amazon’s — should take this as a wake-up call. Even if you weren’t affected by this outage, you could be on the next one. Don’t just have a backup. Have a backup for the backup. Yes, it costs money. How much money does it cost for your business to be out for a day? (Even if Amazon did give all its affected customers a freebie.) Forrester analyst Rachel Dines wrote a blog post listing a number of questions organizations should ask their cloud provider about backups and failover strategies.

Finally, accept that it’s going to happen — whether it’s from a natural disaster like the earthquake in Japan or the tornadoes in the American South, government action to shut down the Internet like in Egypt, widespread electrical failures, or simply a flu pandemic. As Dines says, “Assume nothing” — check every step in the disaster recovery plan, and figure out what the alternative is for every component that could fail.

Think about it. How many places are really safe from natural disasters? We’ve already seen how the Icelandic volcano shut down flights all over Europe. The Bay Area, Seattle, and Portland are all geologically active; many parts of the Southeast are vulnerable to hurricanes; the central U.S. is prone to tornadoes. In addition to earthquakes and volcanoes in the Asia Pacific area, the region is also subject to typhoons.

It’s easy to think that having a backup or replication system in place is enough, but after watching the widespread devastation in Japan, it’s clear that we need to be thinking about how to scale up our ideas of what kind of disaster we’re planning for.

1. Where are your backups, replicated servers, etc. located? Same building? Same city? Same state? If you didn’t realize it before, it’s clear now that a disaster can cover a massive distance and that backups need to be geographically dispersed, perhaps through the cloud. Also, even if you’re using the cloud, where is the data center actually located? If it’s someplace subject to natural disasters, such as earthquake- and wildfire-prone areas in California, it may not help you much. I know some companies that choose to have their backup sites located near Spokane, Wash., because it’s geologically boring.

2. And while you’re at it, how well is your company set up for remote employees? If employees are evacuated, is there a way they can work from where they are? Can employees in other parts of the world pick up the slack?

3. How well is your site and your backup site set up for emergency power? A big part of the problem with the Japanese nuclear reactors was that they didn’t plan for an extended power outage. While there were batteries to operate the cooling system, they lasted only a few hours. Some colocation facilities keep diesel fuel on hand to run generators; does yours? How long will it last?

4. The good news — and there is some — is that the Internet reportedly held up remarkably well. Renesys, which has performed some interesting analyses of Internet shutdowns in Libya, has observed that much of the country’s Internet traffic was unchanged. “It’s clear that Internet connectivity has survived this event better than anyone would have expected,” the company wrote in its blog. “The engineers who built Japan’s Internet created a dense web of domestic and international connectivity that is among the richest and most diverse on earth, as befits a critical gateway for global connectivity in and out of East Asia. At this point, it looks like their work may have allowed the Internet to do what it does best: route around catastrophic damage and keep the packets flowing, despite terrible chaos and uncertainty.”

Consequently, communication with people outside the disaster zone has been better than after some natural disasters, with many people able to check in with loved ones fairly quickly, using social media such as Facebook and Twitter.

So think about your plan. Meanwhile, there are many ways to follow the developing situation in Japan, and to help victims in the ravaged country. Google, in particular, has collected a list of resources to keep informed about what’s happening. It could just as easily have been any of us, so think about how you can help.