It’s difficult to write this, when the full extent of the earthquake damage to northern Japan isn’t yet clear and the nuclear crisis is still escalating. But now — when the images are fresh not only to you but also to the managers who approve your disaster recovery projects — is the best time to think about how your company would handle being in a similar situation, as overwhelming and impossible to believe it might be right now.
Think about it. How many places are really safe from natural disasters? We’ve already seen how the Icelandic volcano shut down flights all over Europe. The Bay Area, Seattle, and Portland are all geologically active; many parts of the Southeast are vulnerable to hurricanes; the central U.S. is prone to tornadoes. In addition to earthquakes and volcanoes in the Asia Pacific area, the region is also subject to typhoons.
It’s easy to think that having a backup or replication system in place is enough, but after watching the widespread devastation in Japan, it’s clear that we need to be thinking about how to scale up our ideas of what kind of disaster we’re planning for.
1. Where are your backups, replicated servers, etc. located? Same building? Same city? Same state? If you didn’t realize it before, it’s clear now that a disaster can cover a massive distance and that backups need to be geographically dispersed, perhaps through the cloud. Also, even if you’re using the cloud, where is the data center actually located? If it’s someplace subject to natural disasters, such as earthquake- and wildfire-prone areas in California, it may not help you much. I know some companies that choose to have their backup sites located near Spokane, Wash., because it’s geologically boring.
2. And while you’re at it, how well is your company set up for remote employees? If employees are evacuated, is there a way they can work from where they are? Can employees in other parts of the world pick up the slack?
3. How well is your site and your backup site set up for emergency power? A big part of the problem with the Japanese nuclear reactors was that they didn’t plan for an extended power outage. While there were batteries to operate the cooling system, they lasted only a few hours. Some colocation facilities keep diesel fuel on hand to run generators; does yours? How long will it last?
4. The good news — and there is some — is that the Internet reportedly held up remarkably well. Renesys, which has performed some interesting analyses of Internet shutdowns in Libya, has observed that much of the country’s Internet traffic was unchanged. “It’s clear that Internet connectivity has survived this event better than anyone would have expected,” the company wrote in its blog. “The engineers who built Japan’s Internet created a dense web of domestic and international connectivity that is among the richest and most diverse on earth, as befits a critical gateway for global connectivity in and out of East Asia. At this point, it looks like their work may have allowed the Internet to do what it does best: route around catastrophic damage and keep the packets flowing, despite terrible chaos and uncertainty.”
Consequently, communication with people outside the disaster zone has been better than after some natural disasters, with many people able to check in with loved ones fairly quickly, using social media such as Facebook and Twitter.
So think about your plan. Meanwhile, there are many ways to follow the developing situation in Japan, and to help victims in the ravaged country. Google, in particular, has collected a list of resources to keep informed about what’s happening. It could just as easily have been any of us, so think about how you can help.