Disaster Recovery

Preparing IT for Flu Epidemics

Question: Are there ways we can prepare for a large percentage of our technical support staff out with H1N1 flu?

With the heightened interest in how to respond to the H1N1 pandemic, every organization should be considering how to manage production support operations in the face of high absenteeism rates that could exceed 30%. Because the illness often hits suddenly, staff members could be sick or home caring for a family member and unable to work. Temporarily losing key individuals, such as system administrators and DBAs, can be traumatic without proper planning and redundancy. A good response plan should focus on assuring that necessary skill sets are available when needed. The following suggested approach is a good start towards making sure you are covered:

• Identify the critical functions and performance timeframe. This information may have already been gathered as part of a business impact analysis. If not, draw up a simple list of the functions or tasks and how time critical they are.
• List the skills and knowledge required to perform critical functions and the staff that possesses them. These might include UNIX administration or knowledge of a custom finance application. Management and the operational staff will know.
• Identify the primary and secondary staff members who can provide backup for each task or skill. In particular, identify critical skills that are possessed by only one staff member. Gaps such as these are the biggest risks.
• Develop a plan for backfilling those critical skills. This may include documenting procedures and training other staff members or locating an outside resource to provide the skill on a temporary basis.
• Practice running operations using backup staff and documentation. This validates the ability of the backup staff to perform the tasks and also provides on-the-job training and job enrichment opportunities
• Plan for working at home (WAH). In many organizations, technical staff members are already required to be available 7×24, so the mechanisms are in place.
• Develop a contingency plan for reducing workload when absenteeism is high. Discuss with senior management the possibility of performing only minimal system changes and delaying major deployments to reduce risk and maintain system stability. Given the possible business implications of such a plan, buy-in from all stakeholders is essential. Define conditions and triggers for putting the plan in action.

Having to deliver services without a full staff is a situation that every organization encounters sooner or later. It can be triggered by events other than a flu pandemic. Preparing for it will make your organization more resilient and provide unexpected benefits.

About the Author

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants

Disaster Recovery and Data Integrity

Question: When recovering an application at a remote disaster recovery site, how can the data’s integrity be verified before resuming production processing?

One of the primary goals of a disaster recovery (DR) plan is to protect business data.  The RTO (recovery time objective) component is often the most time consuming part of a DR plan.  The objective is to shorten the recovery task as much as possible while maintaining trust in your data so that the business can recommence regular operations quickly.  Since the complexity of many applications leave far too many places for bad data to hide, the following recommendations will help speed the recovery process.

Disaster Recovery Preparation

• Implement the fastest replication to your DR site that budget and technology allow. The best option to minimize data loss is synchronous replication of your business critical data to a remote backup site. This ensures full data integrity because the replication logs will record data currency while the transactions are copied to both sites simultaneously.
• The next best option is an asynchronous replication scheme. Even if the delay is measured in minutes or based on periodic file copies, you will still be more prone to some known level of data loss. However, since this option is considerably less expensive, the business owners might decide that the reduced costs will offset the increased risk of data lose. Make sure that everyone understands the impact of data loss on the business recovery process. This includes establishing data loss tolerances, and identifying likely types of data loss during the development of the DR plan.
• Copy verifying data to your DR site along with the primary data. This means finding supporting data from outside the application that corroborates your database. This might include data input forms, activity logs, emails, checksums, or input files. Some of these may be paper-based and could be scanned or copied to the DR site.

Recovery Plan Preparation

• Assess the data at the DR site by knowing both its currency (i.e. how old it is, as precisely as possible) and its consistency across multiple applications. Currency is important because it will identify any lost transactions or updates, while consistency is important because incomplete transactions can cause data corruption problems after you have resumed production.
• Review the replication logs for the last data transmission to establish currency.
• Compare verifying data at the DR site with the recovered data to follow specific transactions and identify the degree of data loss.
• Have the business application users access the data and determine if recent changes to production data are found in the recovered data.
• Execute a build acceptance test (BAT) like the ones used to verify application installation and integrity. These tests often point out data anomalies.
• Have a process and tools in place for tracking and resolving data problems after you return the applications to production.

The above recommendations may not identify all the issues, but they will go a long way toward meeting your RTOs and customer requirements, so that you will be able to start using the application while continuing the verification of the recovered data.

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants

Mapping Application Disaster Recovery to Business Requirements

Question: Now that my organization has acquired space at a  remote co-location data center and we’ve installed hardware, where do we need to consider in setting up recovery for our critical business applications?

While it would be impossible in this forum to go into all the possible strategies that you could employ for application recovery, it will describe the areas that you should consider when developing a recovery solution for your company.

Before thinking about any technology, disaster recovery is really more about business risk management.  As such it is important to start by meeting with the business owners of each application to identify the recovery requirements such recovery time objective (RTO), recovery point objective (RPO), end user workload, and whatever other applications or services are required by the application. In short, understand the main parameters of your recovery solution from the business perspective first. Keep in mind that the business owners may not be familiar with the technological underpinnings of the application, so involve the application support staff to ensure a full understanding of the recovery requirements so that the managers can make reasonable decisions based on what is achievable with the current technology and architectures.

From here, design your recovery solution while considering the following:

• Server power - How much processing power will be needed by the recovered application at the DR site? Will the DR site support production only or will development activities also be occurring there?
• Replication - How much data has to be available at the DR site, how fresh will it need to be, and how will it get to the DR site?
• Network - How much network capacity will be needed to support data replication and end user access to capacity and what protocols should the network support?
• End user access - How will the users of the application access it while running at the recovery site?
• Application installation and code management - How do you ensure that the latest version of the application is available at the DR site?
• Application recovery process - What will be the step by step process for recovering the application? Who will execute the recovery process?
• Change control - How do you ensure that changes to the production version of the application are reflected in the DR environment?
• Testing - How will you test the resources at the DR site and the recovery process?

In designing your recovery solution, think of it as an on-going resource that must be managed with the same attention as your production environment. That’s because it might someday be your production environment.

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants