To understand how to manage your high availability options, it is important to have a discussion about how high availability, risk and component failure work in a cloud environment.  High availability is very important for cloud environments; often the cloud provider is required to meet strict service level agreements for 99.99% or 99.999% (the so called 5-nines) availability.  In theory, that is the very reason that customers are interested in using cloud services.  It should be noted that most of the public cloud service providers have lots of methods to measuring availability that is in their favor, so even in the case of catastrophic systems failure, they are rarely accountable for the downtime.  This has been one of many sources of caution for enterprises that have wanted to leverage public cloud services.

Before going into the details of how to quantify the cost of risk mitigation for a cloud, a short discussion of the science of risk management will help with understanding how it all works.  The goal of business risk management is to detail what kinds of risks exist in your specific business and determine how to prevent them entirely or minimize their impact on the business as a whole. Business risk management is essentially quantifying the risk that a given system will fail multiplied by the cost.  Cost is further broken into two more categories.  Out of pocket costs, also referred to as sunk costs, and lost opportunity costs.  Sunk costs are costs that you will need to pay out to fix the problem, while lost opportunity costs are revenue lost due to the system unavailability.  For example, the risk that there will be regular earthquakes in Japan is high.  The Japanese have responded to this threat by having some of the strongest earthquake resistant building codes in the world.  However, as last year’s 9.0 tremor and following tsunami so dramatically demonstrated, it is impossible to prepare for such extreme and rare events.

High availability is best addressed by redundancy.  However, redundancy can be achieved at several levels of the IT infrastructure: hardware, software, network, or a combination.  Traditional IT organizations have reduced the risk of downtime by concentrating almost exclusively on hardware redundancy.  The scale of the cloud, where there are already thousands or hundreds of thousands of systems, hardware redundancy at the component level quickly becomes unsustainably expensive.   A telling scholarly article that looked at the reported hardware failures from several large data centers shows that by far the most likely failure at data center scales is as would be expected the components that have moving parts, such as hard drives and power supplies.

In practice, the enterprise needs to balance the cost of duplicating hardware throughout the cloud ecosystem, which is the traditional approach to solving the risk management problem, against the need for keeping operating expenses low. Another consideration is that duplicating everything at the hardware level does not automatically guaranty that you do not still have a single point of failure in the environment.  For example, you might have remembered to contract with multiple carriers to spread the risk of a network outage, but if they all come into the data center at a single location, the data center is still prone to a catastrophic “backhoe failure”, which is what happens when a backhoe has severed all the up-link cables in one fell swoop.  It is an expensive and time consuming repair that leaves many unhappy customers in its wake.  Yes, there are ways to mitigate this risk, but they are expensive and need to be balanced against the relative probability of such an event.

The best approach is to look at the probability of failure of each component in context of the entire ecosystem.  Since hard drives fail at such a high rate, the hardware approach is to mirror or RAID the drives across thousands of systems.  This translates to data redundancy and added costs that far exceed the optimum for availability and cost reduction.  At scale, building a storage system that handles the data redundancy at the software level is far more efficient.  Another examples, is planning for power supply (or more precisely fan) failures.  Again, since they are generally the next most common component to fail, instead of filling the cloud data center with thousands of extra power supplies and fans, it is better to build the cloud to be resistant to downtime if a server node fails.  In the end, addressing cloud high availability is not only about determining the MTTF of hard drives, cables or switch ports, it is also balancing it against the likelihood of a given failure at the data center macro level.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

If you are running a typical enterprise IT organization, you are probably struggling with an organization and processes that are not optimized for delivering cloud services.  Traditional IT operations are best designed to handle customized applications and a heterogeneous IT infrastructure, just the opposite of the skills and processes that are needed to support cloud services.  As an illustration of this, I recently had a conversation with a data center engineer about deployment automation.  He noted that his group was able to build a new server in four hours so he didn’t really see the point in further automation.  Hand building systems works fine when you are building 10 servers a week.  It does not scale when you are building 10 or 100 servers a day.  Deployment automation is designed to solve the problem of how to set up hardware and systems quickly when managing hundreds of racks and thousands of servers.  To achieve this level of automation requires the acquisition of new staff skills, building a factory approach to operations, and developing different types of processes.  What is often overlooked is that it will in turn drive significant changes to the enterprise IT organization to meet the new demands for supporting the cloud infrastructure.

Public cloud services are offered so cheaply is because they have both the economies of scale and more importantly, the operations expertise to support highly automated IT infrastructure. Amazon is estimated to have over 300,000 servers.  They do not provision them by hand; it would be an impossible task.   Any company that is managing cloud services, public or private, has faced this problem and has needed to build processes to allow data center administrators to quickly stand up new or replacement racks and servers.

With automation, racks and servers can be provisioned with a minimum of error-prone human labor in a few minutes or hours.  In the case of hardware failure, the administrators simply install new hardware, power it up and allow the auto-provisioning systems to complete the loading of the operating systems and applications.  The hardware is pre-wired into the rack, so that it can be easily plugged in and then automatically configured using the deployment automation.  Ideally, systems are configured and monitored to send out alarms or even automated orders directly to the vendor for new hardware when a certain usage level is reached or there is a hardware failure in the system.

If you are serious about reducing your operational costs for your cloud investment, the smartest thing you can do is invest in some serious automation for all your operations.  This includes not only building staff skills and developing the capacity for automated virtual machine deployment, but also automating deployment of server nodes, network gear and even entire racks.  I would recommend creating automated deployment processes to allow daily or even hourly system builds for faster systems development, test cycles and production deployments.  Leverage automation processes and framework to automate deployment of all modules across the entire cloud architecture.

Deployment and operations automation not only allows for appropriate expansion, but it also cuts the costs of delivering high availability by reducing the need for expensive hardware redundancy.  Service level agreements, growth and scaling can all be addressed by deployment and operations automation.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

Since the dawn of the Internet there has been an on-going debate over whether to use Layer 2 (Ethernet) or Layer 3 (IP) networking inside the data center.  In the beginning, yes, we are talking about the 1990′s, for the most part networks were built on Layer 3 protocols.  L2 switches were only used for internal LAN’s or very small installations.  Does anyone really fondly remember NetBIOS or SPX/IPX?  While Layer 2 switches were easy to deploy – one brand was appropriately named Black Box — they were impossibly slow and unreliable once you scaled past more than a couple hundred machines.  Before the development of public IP address sparing protocols such as CIDR, DCHP and NAT, if you wanted to have an internet connection you had to assign each system a public IP address anyway.

Fast forward 20 years and many network protocols later, data centers are now typically architected to use L2 switches rather than L3 routers.  The reasoning seems to be that Ethernet is faster because you don’t have the overhead of the IP hierarchy, and you don’t have to worry about reconfiguring IP addresses as systems get moved around.  For me the second argument doesn’t hold up, since that is exactly what DHCP and DNS are designed for!  The simplicity of Layer 2 protocols might work well in a data center with hundreds of physical machines, but cloud data centers have the additional burden of needing to keep track of all the virtual machine addresses and networks as well.  It is not uncommon for one physical node to support 30-40 VM’s.  Layer 2 switching protocols have improved mostly by adding “bolt-ons” such as VLAN’s, RBridges, or Cisco’s L2MP.  I would argue that these are all proprietary patches to the fundamental scale and complexity problem. They still don’t have the built-in hierarchy and resiliency of a fully routed IP network.

A better paradigm is to think of cloud data centers as miniature (or in the case of Amazon, not so miniature) versions of the Internet.  Thus applying the inherent scalability and flexibility of the IP address based Internet to a cloud data center network architecture makes perfect sense.

Cloud Network Architecture Basic Principles

• The cloud means, “Any server, any service, any time”
• Scalability through hierarchy
• Simplified network management
• Maximum network traffic flexibility
• Flattened traffic flow over the entire network mesh
• Minimize amount of state information maintained in network by keeping VM state (VM MACs and IPs) out of core network
• Reduce number of protocols to manage

Layer 2 Architecture Limitations

• Number of VLANs is limited to 4096
• Number of MACs stored in switch tables is limited
• Need to maintain a set of Layer 4 devices to handle traffic control
• MLAG (which is used for switch redundancy) is a proprietary solution that doesn’t scale beyond two devices and forces vendor lock-in
• Difficult to troubleshoot network without IP addresses and ICMP
• Configuring ARP is tricky on large L2 networks
• All network devices need to be aware of all MACs, even VM MAC’s, so there is constant churn in MAC tables and network state changes as VM’s are started or stopped
• Migrating MACs (VM migration) to different physical locations could be a problem if ARP table timeouts aren’t set properly

Layer 3 Architecture Advantages

• Provides the same level of resiliency and scalability as the Internet
• Easy to control traffic with routing metrics
• Can use BGP confederation for scalability, so core routers have state proportional to number of racks, not to number of servers or VMs
• It keeps per VM state (VM MACs and IPs) out of the network core, to reduce state churn. The only routing state changes are in case of a ToR failure or backbone link failure
• Uses ICMP to monitor and manage traffic

In future articles, there will be further dives into some of the ways that Layer 3 networks, virtual networks, and the most exciting new networking development, software only networks can be used to successfully address the needs of a cloud data center network.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

The wonderful thing about cloud architectures is that they are designed to be cost effective at massive scales.  The major cloud providers are profitable not only because they can aggregate customers and use the available equipment more efficiently, but they can leverage their considerable market muscle to purchase truckloads of components at steep discounts.  As Google discovered and published in Failure Trends in a Large Disk Drive Population , the brand and cost of a hard drive had little to do with its reliability.  Another paper delivered at the same 2007 Usenix Conference, Disk Failures in the Real World: What Does an MTTF of 1,000,000 Hours Mean to You?, came to similar conclusions.  The key to building reliability in the cloud is not quality components; it is building a hardware architecture that assumes that the components will fail and plan for that failure.  Since the individual components are essentially interchangeable, it stands to reason that a good cloud architecture should be completely hardware agnostic.

The fallacy of that kind of thinking is that failure rates are the only criteria for choosing a given component.  As you know hardware is a moving target, new and better hardware is always coming around the next corner.  Any good storage engineer knows that enterprise customers do not pay the EMC or NetApp premium just because they feel more comfortable buying from a known brand.  They are typically paying for the better tools, faster performance or bigger capacity that they need for their high performance applications.

It turns out that this applies to cloud hardware architectures as well.  Hardware does in fact matter if a cloud is going to run at peak efficiency.  Which hardware components are chosen can make a significant difference under stress conditions.  When the objective is to optimize the environment, the ideal cloud environment should be running at close to peak capacity – essentially under some stress — most to the time.  For example, in a storage array, the two constraints are always going to be system network bandwidth and disk I/O, i.e. how fast the disks can push the data around.  By specifying a faster disk controller and tweaking the configuration to boost the throughput by eliminating disk write caching for example, the entire system will run that much more efficiently.  Yes, in this case you will be reducing disk reliability, but since you already have a mechanism that provides disk failure resiliency in other ways, that risk can be tolerated in exchange for the faster throughput.

In conclusion, at the proof of concept and small system level, cloud hardware agnosticism works just fine, but for massive cloud installations that want to run at peak efficiency, paying attention to specifying the right hardware components to eliminate the throughput bottlenecks, has the potential to boost overall performance significantly.  The trick is determining if the hardware cost differential is worth the increased performance.  Of course at truly Amazonian scales, that cost differential essentially disappears.  However at more modest enterprise scales, in my opinion, in most cases the TCO business case for the better hardware will prevail.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Moving companies’ IT services into the cloud the right way, the first time!

You are in a panic.  Suddenly, your primary data center is down and you are planning to failover your business critical application production to your carefully planned DR site.  Assuming a successful recovery, your first realization is that the DR site is now your production site, albeit temporarily.  All of a sudden you realize that you need to run your backup site in full production mode, i.e. it needs to be run as your ran your recently disabled production site.  Just because you are using the backup site, does not mean that all the normal business rules no longer apply.

When you were putting together your DR/BC plan, you figured that you only needed your backup site to be just a ‘bare bones’ operation that would only support critical functions, but the reality is that life at the DR site will include most, if not all, of the normal production operations headaches.  When putting together your BC/DR plan including the following considerations will make an actual disaster situation that much less painful:

• Full operations management of the DR environment is necessary to keep recovered production running. DR servers have all of the same issues that any server does. Is a full set of your administration and monitoring tools ready to use at the DR site?
• Backups will be required. Production data requires the same level of protection, especially if customer service level agreements are involved. Have you provided for these?
• Support of the ‘hands on’ variety may be needed even though you can manage your infrastructure remotely. If your DR site is far away from your primary site, getting staff there may be a challenge. Have you arranged for appropriate on-site assistance?
• Security controls have to be as stringent as ever because the risks are the same (and perhaps even worse) and all legal requirements still hold. Can you control and monitor access to your DR site?
• Applications still require support. Patches and emergency releases will inevitably be needed to keep the business running. Are all your code libraries and the tools needed for development and testing installed at the DR site?

The planning implications are clear – since your DR site is a substitute for your primary production site, think of it as such and outfit it to perform at the same level. Even though it will not be as large as the primary site, it should offer all of the same capabilities as the primary site.  In some situations, it just might be your home for a long period of time.

About the Author

John McWilliams, JH McWilliams & Associates, Business Continuity Consultants