The other, more important reason? He wanted to see how his IT staff reacted.

What DR and BC expert Paul Kirvan has found too often is that a lack of disaster recovery documentation is stymieing the best-laid and expensive – costing into the millions — DR strategies.

It’s not simply that they don’t have disaster recovery documentation, but if they do, people can’t understand it.

In one recent instance, a CIO ran through a disaster recovery scenario, and it went off smoothly, thanks to one all-star on the staff who knew how to recover everything off the top of his head.

“I asked, ‘What if he’s sick of on vacation?’” Kirvan said.

His point is that the documentation has to be simple enough and consistent enough for anyone on staff to be able to step in and recover a system — so simple that, even if your IT staff can’t perform the function for some reason, a non-IT person could.

To help get your staff on the same disaster recovery documentation page, Kirvan suggests checking out disaster recovery software, plan templates and guides, a list of which has been compiled by fellow industry expert Phillip Rothstein.

The funniest comment regarding my article this week on SearchCIO.com about Austin Powder Co.’s DR plan, post-virtualization, to make its HQ a hot site for its remote data centers came from “Richard” via email: “Sorry, but when I saw the tag line on this article ['A powder keg of BC and DR planning'] and the opening sentence, ‘blasting through,’ I thought it was April 1!” he wrote. “And later in the article, the reference to Symantec reminded me of Semtex, and made the article even more explosive! I just hope these folks don’t lose track of any of their explosives products in their quest for virtualization.”

I emailed Richard to share that my copyeditor and I had had similar chuckles. However, I should add how impressed I am by the IT vision at Austin Powder, a company that was founded in 1833 and has seen more general ledgers in various formats than most enterprises.

The best wisdom of the week came from another Richard — Dick Csaplar, of the Aberdeen Group in Boston. “A disaster plan ages very quickly,” he said; disaster recovery solutions are a constant effort: “You are never done. People in charge of doing this need to stay on top of it. The people who pay attention and update their disaster plans are among the best-of-class organizations.”

Companies of all sizes can benefit from this lesson, as demonstrated by another story source, Greg Schulz, author of The Green and Virtual Data Center, and founder of The Server and Storage I/O Group in Stillwater, Minn. His own BC/DR strategy involves three laptops: a small one he uses for travel, a bigger one in his office, and another one as a backup. He’s not into the “less laptop” trend, but is going toward a more robust laptop using virtualization to seamlessly move his workloads wherever he happens to be: on the road, on an airplane, in his office.

And Austin Powder Network Administrator Chris Benco described how shared storage, added to the company’s virtualized environment, enables him to do the same thing among the main site and two remote data centers. Now the company not only is protected, but also can proactively move workloads to avoid a storm in one area, perform maintenance in another, whatever.

Let us know what you think about the story; email Laura Smith, Features Writer.

“With DR, you need a Get Out of Jail Free card,” said Edward Haletky, CEO of The Virtualization Practice LLC, in Wrentham, Mass. Testing a disaster recovery system can be an opportunity to wave the IT flag and, rather than suffer the frustration of impatient users, spin the event to make them understand the miracle of recovery. IT needs to adopt and share the general hot-site mentality, he said. When you switch to a hot site, be sure to send an email to let users know that the system will be slower than normal but, remarkably, is running, he added, emphasizing the last word.

“Don’t expect your hot site to be as fast as your production environment, and reinforce it to your staff: ‘We’re running in a reduced-capacity environment — but we’re still running,’” he said.

Haletky, a virtualization evangelist since 2004, consults with enterprise companies and mentors other enterprise consultants. He is an architect — both physical and virtual (because “you can’t just be in the virtual world”) — who had an opportunity to test the beta version of VMware Workstation. He also helped to judge the products at VMworld 2010.

“I always dreamed of having a machine that, no matter what I had, would run everything. Now I can do that with virtualization,” said Haletky, who is also author of two books: VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment and VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers.

One of the keys to disaster recovery in a virtual server environment is reducing the amount of time it takes to make a backup, and that depends on such strategies as deduplication, as well as on bandwidth. Next week on SearchCIO.com, learn how to deal with the top concerns for virtual disaster recovery: bandwidth, testing the system, and deciding whether virtual DR should be located in the cloud.

Let us know what you think about the story; email Laura Smith, Features Writer.

But cloud uptime? Now that is an even larger trust issue that CIOs just can’t seem to get past. At least, not the CIOs attending a recent gathering of public cloud services providers sponsored by the trade and investment arm of the British Consulate-General.

The CIOs and cloud services providers came together to hash out what it’s going to take to get enterprises onto the cloud. Security was an issue, of course, with data transparency and knowing who has access to their data among the concerns.

As for performance, one CIO said he would FedEx a terabyte of data to a public cloud provider for fear that the provider’s network couldn’t handle a data transfer of that load. One attendee said performance uncertainties in the cloud could possibly weaken your disaster recovery plan.

The CIOs also didn’t trust that their public cloud providers wouldn’t go out of business. CIOs have a long memory and haven’t forgotten that seemingly well-established hosting providers can go out of business — think Exodus Communications.

In 2000, Exodus was the darling of the hosting industry, with revenue of $818 million, stocks worth $90 a share and 42 colocation facilities — not to mention nearly 5,000 customers, including Microsoft, Yahoo and the New York Stock Exchange. Many of the company’s customers, however, were dot-com startups that failed to pay their hosting bills, pushing Exodus further into debt as it continued to build and acquire more facilities. (Some experts believe that the next wave of winners in outsourcing will be the ones that have large infrastructures that can support the entire services layer, from software to hardware. That would require big investments in infrastructure, like those Exodus made.)

Public cloud providers are not immune — a few bad infrastructure and financial planning decisions could bring the multitenant house of cards down. What happens to customer data then? Just as they asked during the dot-com bomb and downfall of application service providers, CIOs want to know how public cloud providers will deal with porting data and services to another cloud provider, or back in-house.

They don’t want their data to end up as an asset in bankruptcy court.

But this is a nascent industry, and CIOs are willing to wait for public cloud providers to grow up a bit. And as they grow, CIOs would like the providers to keep these other capabilities in mind:

• The ability to work offline, as well as online.
• The ability to manage multiple cloud services and relationships under one umbrella.
• The ability to speed up, not slow down, change management.

CIOs are sending clear messages to public cloud providers. It will be interesting to see how the providers live up to these demands — or maybe private clouds are the way to go?

Let us know what you think about this blog post; email Christina Torode, News Director.

One of my colleagues attended a conference during the no-fly period, and saw first-hand how technology can aid business continuity: A keynote speaker who was stranded in Europe did his talk via live videoconference, even responding to questions that popped up live or via Twitter.

This story was still rolling through my mind as I read Senior News Writer Linda Tucci’s story this week about business continuity and the volcano. Prior to that, I’d been thinking of disaster recovery and business continuity more in terms of the physical limitations of people being unable to travel, but she investigates the supply chain management nightmare created by the lack of air travel for days on end.

The message? You could have an excellent supply chain management strategy – but if you don’t have the disaster recovery and business continuity plan to back it up, it won’t do you a whole lot of good when it matters the most.

Do you have a successful disaster recovery or business continuity story that worked — or didn’t, and you learned an important lesson from it? What’s your supply chain management strategy when things go wrong? I’d love to hear your story; please e-mail me at rlebeaux@techtarget.com.

According to the official WordPress blog (which I assume was also unavailable during the downtime), the problems were likely the result of an “unscheduled change to a core router by one of our data center providers [that] messed up our network in a way we haven’t experienced before, and it broke the site.” Worse, the outage broke all of the company’s mechanisms for failover in San Antonio and Chicago.

I’ll be interested to hear about WordPress’ stopgap solutions in the case of blog failure. In the meantime, I’m left to ponder — and point out to our readers — that having a disaster recovery and business continuity plan isn’t necessary only for such catastrophes as hurricanes — or a massive data breach that leaves you scrambling to explain to irate customers what went wrong (I gather that WordPress did an admirable job updating users via Twitter during the outage). But how do you prove this to the business so you can get the resources you need? Consider piggybacking disaster recovery efforts on other projects or mapping availability risk.

The WordPress problems also underscore the importance of properly vetting your providers, whose data center outage could negatively affect your business and its customers. For more information about assessing potential sourcing partners, check out our FAQ on getting started with IT outsourcing.

Configurations drift, or IT staff simply don’t have the time — or the budget — to mirror every aspect of their production environment. That’s where virtualization comes in.

Applications that may not have made it on the mission-critical DR list can now be put on a shared piece of hardware: a virtual server in the data center or hot site. The costs of maintaining hardware for both mission-critical and not-so-critical applications can drop considerably in this scenario.

The other day, Nelson Ruest, a principal with consulting firm Resolutions Enterprises Ltd., in Victoria, British Columbia, was telling me that one of his enterprise clients is projecting 60% to 70% savings, per year, across its infrastructure for disaster recovery. The company moved from mirroring the production environment — taking up three floors to do so in its backup data center — to using less floor space and hardware with a virtualized DR plan. It replaced many physical servers with virtual servers in its production environment as well.

Independent Bank, out of Iona, Mich., saved $1 million in hardware by replacing underutilized hardware with virtual servers, as part of a desktop-to-data-center virtualization DR strategy.

The shift to virtual desktops and servers allowed the bank to eliminate configuration drift between its hot site and production environment as well, according to its CIO, Pete Graves.

Still, enterprises are hesitant to use virtual disaster recovery for mission-critical applications, and are definitely not throwing out tape backups any time soon, according to John Humphreys, senior director of product marketing for the virtualization and management division of Citrix Systems Inc.

Humphreys is seeing the spread of virtualization for DR take the same path that the technology did on the server front: around the edges of the enterprise, starting with non-mission-critical applications that haven’t made it into the “critical” DR budget.

Others believe that virtualization DR will continue to evolve as server virtualization does. As the staff at SearchServerVirtualization.com point out in an article on predictions for 2010, there is still the potential for one virtual server to take down hundreds of other virtual machines with it.

So, it would seem that enterprises are testing out and moving forward with virtualization disaster recovery, but with a note of caution.

Tell us what your disaster recovery plans are and if virtualization will be part of them. Email ctorode@techtarget.com.

“The larger enterprise companies are beginning to take the H1N1 pandemic seriously and beginning to think about what they are going to do if 40% of their workforce gets sick this winter,” said Bob Boyd, CEO of Agility Recovery Solutions Inc., the Charlotte, N.C.-based provider that brings its ReadySuite trailers on-site in the event of a disaster or for testing purposes. “We are seeing a fairly significant increase in interest from those larger companies to have technology on schedule with us, so that if they sent 100 people home, we could ship them laptops, so they could work from home.”

Indeed, of the 41 recoveries Agility has done this year, seven have been for customer responses to the pandemic. One was in Montreal and the rest were in the U.S.

“In the seven cases we’ve done so far, their pandemic plans said that if an employee is diagnosed with H1N1 virus, the company takes the 100 people that employee is closest to, whether in the adjacent cubicles or by contact in meetings, and all those people go home for up to two weeks, because the incubation is thought to be about 10 days,” he said.

Companies like Agility can supply laptops loaded with the technology employees need to do their jobs along with any equipment, like headsets.

Agility is planning a series of webinars in September with avian flu expert John Lange to help its smaller member companies get ready for the expected upswing in swine flu cases this year. The emphasis will be on what companies can begin to do to protect their employees from the swine flu virus and how to handle cases, as they come up. Large companies with effective HR departments are typically well-prepared to deal with medical issues.

“You get to a small medium-sized company, that guy doesn’t even know if he has the legal right to send that worker home. They need help on how do they talk about H1N1 with their staff and encourage good practices,” Boyd said. He said Agility is spending a lot of time talking to customers about cross-training employees, so that more than one person knows how to perform a critical business function. “If they are right, a lot of people are going to be out sick.”

A business continuity plan is your company’s prescription for things that you can expect to go wrong: components will fail, servers will fail, network outages are going to happen, IT professionals make mistakes. Disaster recovery is your plan for the things you can’t anticipate. “If you’re doing this Calvin and Hobbes scenario, where planes are falling from the sky, that is when you’re talking a disaster recovery plan,” Shacochis said.

Savvis, with 29 data centers, likes to boast it is has built the architecture required to give companies business continuity, which in turn gets rolled into its standardized services. “Virtually all our products have a high-availability option that can be added on for pennies on the dollar,” he said. And many customers use those services as their DR site.

But the premise of a DR-in-a-box solution — promised by various providers — is, in his view, untenable.

“We don’t really know what your requirements are, we don’t really know what the nightmare scenario will be, you’re not really implementing anything with us, but trust us, when you pick up the phone we’ll be there, and we’ll get you a data center in a heartbeat,” Shacochis said. “Those sorts of services are not that difficult to sell because there are a lot of people who want to believe that they exist. But they are very difficult to execute on.”

In fact, Savvis has not gone to market with a catchall DR solution yet. “We don’t really believe that the process maturity across so many different customers is there, or the standardization across so many different architectures is there that would allow us to do it,” Shacochis said.

Shacochis, however, believes that there will be a really elegant solution that will ultimately be cheaper than the present multi-tiered DR solutions and better. His idea is that the kind of cloud computing platform that Savvis is building in its labs will enable it — eventually — to offer DR that is standardized, flexible and cost-attractive enough to customers to make it worthwhile. Shacochis envisions a platform that can function as a complete cloud data center.

“All the typical IT resources you get in a physical data center, we’re going to be building in a platform that will allow you to provision not just your compute resources on the network, but to provision actual data center topologies for routing, switching, security, load balancing and failover features, as well as computing, storage and storage lifecycle management resources that are all running in a software-based context that you can control over a portal, and eventually control via a software API,” he said.

The beauty of that model, he says, is that companies could provision their entire cloud application stack, get it up and implemented and then turn it off.

“That really would be a cloud DR model, where the customer is paying a small percentage of what they would ordinarily pay for production, and they would have a highly functional and easy to executive DR plan.”

When will we see this? He thinks it’s a year or two off. Sounds good to me.

Benchmarks for state-of-the-art RTO and RPO are one thing if your enterprise is already up to date with business continuity and disaster recovery. But these metrics — and affordable DR — can seem awfully abstract in the real world, I am discovering from CIOs, especially if the real world is the globe.

I recently got an email from Jiten Patel, CIO of the Foundation for International Community Assistance (FINCA International), an amazing not-for-profit that provides microfinance services to the world’s lowest-income entrepreneurs. You’d think with its 25-year successful history of village banking, not to mention high-profile support from uber-connected celebrities like actress Natalie Portman, FINCA would not be at a loss for good DR. And, indeed, setting an RPO and RTO benchmark for FINCA’s headquarters in Washington, D.C., “is an easy conversation.”

But there was Patel, on business in Baku, Azerbaijan, explaining the difficulty of providing affordable DR at its microfinance operations in 21 countries around the globe:

“DR options in the developing countries are fairly limited and expensive propositions, and not something we can afford to have in place in every country. And unlike in the US where we have the likes of Sungard and IBM who offer such services at a reasonable cost, this is not the case in countries where we operate — one has to buy a box, which may sit there ‘cobwebbed’ — which is not a viable option and it adds to the financial burden.

“In light of this my strategy has been to centralize our infrastructure on a region-by-region basis, to use 3rd-party hosting providers who can provide more robust redundancy options, and to try and negotiate more affordable DR pricing, which where we can makes it much more palatable for the likes of us.

“It would be a boon for all of us NGOs who operate globally, if someone like Sungard or IBM offered such services at very affordable rates around the globe: Offering it just in the US for far -lung global operations is not viable.”

The world may be getting flatter, but the playing field is far from level when it comes to DR. And it is just a hunch, but disasters that shred data must be common in countries where electricity on any given day is not a given.