In a previous discussion of cloud high availability, I covered in general terms what are some of the principals and approaches that make sense in a cloud environment. This time we will dive into some details of how this can be achieved in an Openstack environment.

The average published MTBF on switches seems to be between 100,000 and 200,000 hours. This number is dependent on the ambient temperature of the switch in the data center. I am assuming that most modern data centers are properly cooled for maximum switch life. This translates to between 11 and 22 years. Even in the worst case of poor ventilation and high ambient temperatures in the data center, the MTBF is still 2-3 years based on research found at http://www.garrettcom.com/techsupport/papers/ethernet_switch_reliability.pdf.

The mean time to replacement (MTTR) for a switch is going to be dependent on how exactly how the data center is staffed and what processes are used for replacing switches. Assuming that you would keep a few spares in the data center and that it is fully staffed 24 hours/day, the average time to replace a switch including configuration is going to be under 2 hours. Most modern switches are auto-configured so the actually provisioning time after the switch is powered up in the rack is under 5 minutes.

Let me walk through what will happen in the case of a top of rack (ToR) switch failure in the Swift cluster. Swift by its nature is fault tolerant at the rack level. That means that the system will continue to operate without data loss if an entire rack goes off-line. The cluster would detect the rack being off-line and send out a notification that the NOC staff would see within 5 minutes. In the case of rack going off-line Swift does not automatically move any data. The reason for this is that in fact, the NOC staff needs to make a decision about the cause of the rack going off-line and how long it will take for it to come back on line. In the case of a switch failure, the data in the rack is still intact, so it is far more efficient to just replace the switch then bring the rack back on-line without having to move the data. Even if the NOC staff decides to move data around, which they would only do if the fault is in the servers not the switch, the network overhead that it adds to the cluster is in the range of 3-5% for a large cluster with properly tuned ring rebuild cycle. Clearly taking a rack off-line is not considered a problem. I would argue that you should expect to be able to take racks off-line with no impact to the system as a whole as a matter of course for maintenance, upgrades and other reasons.

Nova behaves is slightly differently in the case of a rack failure. Unlike Swift the architecture does not have an assumed base unit of failure at the rack level. It does have the concept of a availability zone, which is quite different from a Swift zone just to confuse things. That doesn’t mean that you cannot create an equally fault tolerant Nova architecture, it just requires more development of high availability at the application level of the system combined with the use of the availability zone as a mechanism for balancing the applications in different locations. The assumption is that it is the responsibility of the application to build in fault tolerance, not the underlying infrastructure to keep track of the individual VM instances. Nova zones can be used to achieve this level of fault tolerance in combination. Combining this with the a live migration functionality and HA application design will allow you to build support for rack level failure. Again, the metrics for determining the next steps (replacement of switch only or rebuilding of entire rack) will be based on the specific component failure. See the recent discussion of this at http://lists.us.dell.com/pipermail/crowbar/2012-January/000643.html for more ideas on how to architect such a system.

Another approach would be to create high availability through redundant hardware. In this case you could provision the racks with two switches. However this is an expensive option in a large data center with hundreds of racks. It is clearly orders of magnitude more expensive to take this approach. From a risk perspective, you have substantially increased your per rack costs with little or no reduction in risks since the rate of failure is so low to begin with and the architected unit of failure for a cloud infrastructure should be at the rack level to begin with.

About the Author

Beth Cohen, Cloud Technology Partners, Inc. Transforming Businesses with Cloud Solutions

People have been managing data centers for 30-40 years now, so that should mean that there are a good set of standard best practices for building highly available resilient components.   That is true for the old style data center, but the old best practices are expensive and do not scale well for cloud architectures.  Duplicating hardware to protect against failure works well when you have hundreds of components but the costs are linear so it does not scale.  Unlike traditional IT operations, over-design to protect against obsolescence is not desirable when scaling to thousands of nodes.  For example, spending an extra $6000/rack for 10GB switches might seem to be a sensible way to protect against hardware obsolescence if you have 10 racks, but that extra cost is much harder to justify when you are provisioning a 100 racks and it has turned into an extra $6 million!

The principal of ‘replacement management’ takes on great importance when managing the thousands of physical devices required for a cloud deployment.  The advantage of the cloud is that you do not need to build expensive high availability redundant systems because an assumption that components will fail is built into the architecture.  By leveraging the huge pools of cloud resources, the level of redundancy can be considerably reduced.  If a component fails, the system will continue to work until someone replaces it.  Since commodity low price devices typically have a high rate of failure, the whole architecture needs to be based on “availability” and “partial failure”.

In a cloud environment, it makes much more sense to just replace a component than worry about what caused the failure and trying to troubleshoot it.  The most common components to fail are disks, since they are mechanical moving parts.  A typical disk failure rate in a cloud data center is about 10-15%.  However, fans, power supplies and memory will also fail less frequently.  For example, the OpenStack Swift architecture assumes that disks, systems and entire zones can and will disappear (fail) at any time.  Yet, there are only three copies of every file, and no additional redundancy in the hardware.

This approach to failure at scale can be very cost effective, but it takes different mindset from traditional operations.  Every cloud operations engineer for cloud should learn what is in the service, where the critical parts are located, and how to replace a failed component, then incorporate the knowledge into standard operations processes.  Automated tools need to be written to help identify the location of failed disks and other components so they can quickly be isolated from the environment and replaced.  To maintain a high level of robustness without sacrificing cost efficiency, the system needs to be designed to replicate data on the application/software level, not disk or network level.

In conclusion, the biggest paradigm shift is that development and operations groups need to work together to optimize the systems and drive down costs.  Tests and metrics need to be created to determine the optimum systems configurations.  By understanding how changes in the components affect the systems as a whole, it will allow you to flexibly configure the systems to meet the application requirements as they change.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

You are in a panic.  Suddenly, your primary data center is down and you are planning to failover your business critical application production to your carefully planned DR site.  Assuming a successful recovery, your first realization is that the DR site is now your production site, albeit temporarily.  All of a sudden you realize that you need to run your backup site in full production mode, i.e. it needs to be run as your ran your recently disabled production site.  Just because you are using the backup site, does not mean that all the normal business rules no longer apply.

When you were putting together your DR/BC plan, you figured that you only needed your backup site to be just a ‘bare bones’ operation that would only support critical functions, but the reality is that life at the DR site will include most, if not all, of the normal production operations headaches.  When putting together your BC/DR plan including the following considerations will make an actual disaster situation that much less painful:

• Full operations management of the DR environment is necessary to keep recovered production running. DR servers have all of the same issues that any server does. Is a full set of your administration and monitoring tools ready to use at the DR site?
• Backups will be required. Production data requires the same level of protection, especially if customer service level agreements are involved. Have you provided for these?
• Support of the ‘hands on’ variety may be needed even though you can manage your infrastructure remotely. If your DR site is far away from your primary site, getting staff there may be a challenge. Have you arranged for appropriate on-site assistance?
• Security controls have to be as stringent as ever because the risks are the same (and perhaps even worse) and all legal requirements still hold. Can you control and monitor access to your DR site?
• Applications still require support. Patches and emergency releases will inevitably be needed to keep the business running. Are all your code libraries and the tools needed for development and testing installed at the DR site?

The planning implications are clear – since your DR site is a substitute for your primary production site, think of it as such and outfit it to perform at the same level. Even though it will not be as large as the primary site, it should offer all of the same capabilities as the primary site.  In some situations, it just might be your home for a long period of time.

About the Author

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants

The pervasiveness of the Internet has put IT executives in a bind.  Nowadays, organizations rely so heavily on IT to run their businesses that users have become IT’s top priority.  IT is expected to deliver high availability and predictable performance for key user applications — the always on imperative.

Yet, many IT departments lack sufficient resources – skilled personnel, streamlined processes and effective technology – to keep IT operations running smoothly when needed.  Moreover, existing applications were often designed and deployed from IT’s perspective, not the business, or users’.  For example, IT generally concentrates on analyzing technical specifications, defining IT acceptance testing, and managing project deliverables.  User requirements get little attention which often leads to poor user adoption, negating the expected project ROI.

This technical mindset of designing, deploying and managing IT systems is no longer sustainable.  Application development needs to be prioritized for business requirements, user needs and business value.  The deployment phase should be concerned with user acceptance, training and usage, first.  Incorporating metrics for business impact (e.g. application availability & performance for the users; user adoption rates; and productivity gains) ensure that these goals are met.  Operations must concentrate on high availability and performance.  A through awareness and visibility into the applications and all other elements that must function optimally will make certain critical business services are enabled properly.

Adopting the following five tenets will deliver “always on” applications and lead to high availability and predictable performance of key applications for users:

• Design with the end in mind – Meet the objective of high availability and performance over the entire multi-year usage period
• Follow the money – Stay focused on financial and business benefits of IT systems instead of technical benefits
• Focus on user experience – Shift perspective from technology performance to user productivity
• Break the silos – Create cross functional teams to achieve high collaboration between IT and users
• Manage from the business process perspective – Monitor critical applications down through underlying equipment to understand business impact of all system components. This will hasten problem resolution and reduce unplanned downtime.

About the Author

Robert Johnson, Director of Product Marketing, Atrion Networking Corporation

With the heightened interest in how to respond to the H1N1 pandemic, every organization should be considering how to manage production support operations in the face of high absenteeism rates that could exceed 30%. Because the illness often hits suddenly, staff members could be sick or home caring for a family member and unable to work. Temporarily losing key individuals, such as system administrators and DBAs, can be traumatic without proper planning and redundancy. A good response plan should focus on assuring that necessary skill sets are available when needed. The following suggested approach is a good start towards making sure you are covered:

• Identify the critical functions and performance timeframe. This information may have already been gathered as part of a business impact analysis. If not, draw up a simple list of the functions or tasks and how time critical they are.
• List the skills and knowledge required to perform critical functions and the staff that possesses them. These might include UNIX administration or knowledge of a custom finance application. Management and the operational staff will know.
• Identify the primary and secondary staff members who can provide backup for each task or skill. In particular, identify critical skills that are possessed by only one staff member. Gaps such as these are the biggest risks.
• Develop a plan for backfilling those critical skills. This may include documenting procedures and training other staff members or locating an outside resource to provide the skill on a temporary basis.
• Practice running operations using backup staff and documentation. This validates the ability of the backup staff to perform the tasks and also provides on-the-job training and job enrichment opportunities
• Plan for working at home (WAH). In many organizations, technical staff members are already required to be available 7×24, so the mechanisms are in place.
• Develop a contingency plan for reducing workload when absenteeism is high. Discuss with senior management the possibility of performing only minimal system changes and delaying major deployments to reduce risk and maintain system stability. Given the possible business implications of such a plan, buy-in from all stakeholders is essential. Define conditions and triggers for putting the plan in action.

Having to deliver services without a full staff is a situation that every organization encounters sooner or later. It can be triggered by events other than a flu pandemic. Preparing for it will make your organization more resilient and provide unexpected benefits.

About the Author

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants

One of the primary goals of a disaster recovery (DR) plan is to protect business data.  The RTO (recovery time objective) component is often the most time consuming part of a DR plan.  The objective is to shorten the recovery task as much as possible while maintaining trust in your data so that the business can recommence regular operations quickly.  Since the complexity of many applications leave far too many places for bad data to hide, the following recommendations will help speed the recovery process.

Disaster Recovery Preparation

• Implement the fastest replication to your DR site that budget and technology allow. The best option to minimize data loss is synchronous replication of your business critical data to a remote backup site. This ensures full data integrity because the replication logs will record data currency while the transactions are copied to both sites simultaneously.
• The next best option is an asynchronous replication scheme. Even if the delay is measured in minutes or based on periodic file copies, you will still be more prone to some known level of data loss. However, since this option is considerably less expensive, the business owners might decide that the reduced costs will offset the increased risk of data lose. Make sure that everyone understands the impact of data loss on the business recovery process. This includes establishing data loss tolerances, and identifying likely types of data loss during the development of the DR plan.
• Copy verifying data to your DR site along with the primary data. This means finding supporting data from outside the application that corroborates your database. This might include data input forms, activity logs, emails, checksums, or input files. Some of these may be paper-based and could be scanned or copied to the DR site.

Recovery Plan Preparation

• Assess the data at the DR site by knowing both its currency (i.e. how old it is, as precisely as possible) and its consistency across multiple applications. Currency is important because it will identify any lost transactions or updates, while consistency is important because incomplete transactions can cause data corruption problems after you have resumed production.
• Review the replication logs for the last data transmission to establish currency.
• Compare verifying data at the DR site with the recovered data to follow specific transactions and identify the degree of data loss.
• Have the business application users access the data and determine if recent changes to production data are found in the recovered data.
• Execute a build acceptance test (BAT) like the ones used to verify application installation and integrity. These tests often point out data anomalies.
• Have a process and tools in place for tracking and resolving data problems after you return the applications to production.

The above recommendations may not identify all the issues, but they will go a long way toward meeting your RTOs and customer requirements, so that you will be able to start using the application while continuing the verification of the recovered data.

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants

While it would be impossible in this forum to go into all the possible strategies that you could employ for application recovery, it will describe the areas that you should consider when developing a recovery solution for your company.

Before thinking about any technology, disaster recovery is really more about business risk management.  As such it is important to start by meeting with the business owners of each application to identify the recovery requirements such recovery time objective (RTO), recovery point objective (RPO), end user workload, and whatever other applications or services are required by the application. In short, understand the main parameters of your recovery solution from the business perspective first. Keep in mind that the business owners may not be familiar with the technological underpinnings of the application, so involve the application support staff to ensure a full understanding of the recovery requirements so that the managers can make reasonable decisions based on what is achievable with the current technology and architectures.

From here, design your recovery solution while considering the following:

• Server power – How much processing power will be needed by the recovered application at the DR site? Will the DR site support production only or will development activities also be occurring there?
• Replication – How much data has to be available at the DR site, how fresh will it need to be, and how will it get to the DR site?
• Network – How much network capacity will be needed to support data replication and end user access to capacity and what protocols should the network support?
• End user access – How will the users of the application access it while running at the recovery site?
• Application installation and code management – How do you ensure that the latest version of the application is available at the DR site?
• Application recovery process – What will be the step by step process for recovering the application? Who will execute the recovery process?
• Change control – How do you ensure that changes to the production version of the application are reflected in the DR environment?
• Testing - How will you test the resources at the DR site and the recovery process?

In designing your recovery solution, think of it as an on-going resource that must be managed with the same attention as your production environment. That’s because it might someday be your production environment.

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants