What does this have to do with the just concluded April 2013 OpenStack Summit? Everything! The Summit was a big turning point for the just under three year old Open Source Cloud Platform Project. Never before has the Summit felt more like a real business conference with a clear-eyed agenda to architect a stable and viable enterprise ready cloud platform, rather than just a bunch of wild developers on a mission to save the world. After several years of intense development and shakedown, OpenStack is finally more than ready to take on the unique challenges of enterprises that want to deploy private clouds.

There were plenty of great examples of respected organizations, public and private that have crossed the innovation chasm by deploying OpenStack clouds into their production environments. The keynote sessions all highlighted large enterprise case studies to emphasis the message that OpenStack is a strong and very viable option. Here are just a few to whet your appetite:

• • Bloomberg has deployed a large scale internal cloud, for a variety of applications including analytics and support of their backend systems.
• • BestBuy built an OpenStack cloud to support their extremely spiky holiday rush traffic. In just eight months, 25% of their systems ran on their new cloud in time for the 2012 Christmas buying season. They found the new platform, combined with a move to a PaaS push button development environment and a server side delivery architecture, allowed them to speed delivery of their catalog content pages down to under 2 seconds, while saving over $500K per platform upgrade. How is that for some serious discounting!
• • Comcast Cable and NBC/Universal built an OpenStack cloud to move content delivery out of proprietary limited function set-top boxes and into backend servers, giving them more flexibility and capabilities for the development of new interactive features such as live streaming and gaming.
• • The NSA, yes the spies in DC, are using OpenStack. Of course, what, how and why they are using it were not shared, but clearly they are not overly concerned with reports of security deficiencies.
• • Samsung is rolling out a production grade platform that will support millions of users across the globe. If the success of the new Android based smartphones and tablets are any indication, Apple should be looking over its shoulder very carefully.

In conclusion, if your company is contemplating a public or private cloud deployment, you should be putting OpenStack at the top of your shortlist. The rich ecosystem of big names, such as Dell, IBM, RedHat and Cisco is a clear signal that OpenStack has arrived!

About the Author
Beth Cohen is a senior cloud architect for Cloud Technology Partners, Inc., focused on delivering solutions to help enterprises leverage the efficiencies of cloud architectures and technologies. Previously, Ms. Cohen was the director of engineering IT for BBN Corporation, where she was involved with the initial development of the Internet, working on some of the hottest networking and web technology protocols in their infancy.

From the Enterprise perspective, now has never been a better time to invest in cloud services. Enterprises are broadly adopting all types of cloud services at multiple levels in the organization. Initial predictions were for the enterprise to favor private and community clouds over public services, but the hottest trends in Cloud adoption has been Software as a Service (SaaS) and cloud-based business applications which are projected to grow from $13.4 billion in 2011 to $32.2 billion in 2016, a 19.1% five-year CAGR! The rapid adoption of SaaS applications of all flavors by the enterprise has been a surprise to many, but the vastly reduced costs due to the pay as you go pricing models, and high degree of service delivery flexibility has overcome any perceptions of needing to trade price for reduced feature sets. Unsurprisingly, the most often added cloud-based application services are Customer Relationship Management (CRM), Supply Chain Management (SCM) and Enterprise Content Management (ECM), with Web Conferencing, teaming platforms and social software suites nipping at their heels.

Infrastructure as a Service (IaaS) continues to appeal to companies that want to replace in-house and traditional data center models for the cloud hardware abstraction approach. Gartner is predicting Infrastructure-as-a-Service (IaaS), cloud management & security devices, and Platform-as-a-Service (PaaS) are growing from $7.6B in 2011 to $35.5B in 2016, a CAGR of 36%. A word of warning for companies that think just because the Amazon cloud is easy to use, it is easy to build and manage. Building a private cloud still requires a degree of expertise that few enterprises have in-house. In the next year or so expect more companies to see the value of using the services of cloud consultants to avoid painful and expensive mistakes when building private clouds; or alternatively using emerging enterprise focused public cloud services such as Bluelock or Terremark.
Platform as a Service (PaaS) which still does not quite know what it wants to be when it grows up, is lagging with only about $1B in revenue in 2012, but as the market matures, expect to see rapid uptake as companies recognize the value of standardized tools that can ease the pain of Cloud application deployments. Some of the newer PaaS tools like the ServiceMesh Agility Platform combine the features of a SDLC workflow engine, production support and orchestration across different cloud platforms.

New Cloud Technology Directions

Ultimately, innovation is the marriage of technology and organization change. The dilemma is how to pull innovation into IT core functionality without disrupting the flow of new ideas, when the modern enterprise understanding of IT is focused on operational excellence and cost control. The CIO should be leading the cultural change to the new flat organization by leveraging cloud and mobile application in new and interesting ways. The trick is to create tools for people to quickly test and validate their ideas so they can implement new ideas that work within the enterprise framework. Cloud tools can be used to deliver on that promise, but is the enterprise up to that challenge?

This is not the first time wrenching technology innovation has changed how companies do business, and it certainly will not be the last. Companies able to use disruptive technologies, such as cloud computing, effectively will leave companies who do not have that ability in the dust. By recognizing the seeds of change and embracing them, the smart company can leverage cloud services by using the efficiencies of pooled IT resources at the same time allowing greater flexibility to meet the challenges of the global economy. This combination of combining commodity utilities with innovation allows companies to compete effectively using the efficiency and flexibility strategies simultaneously.

Challenges, Always Challenges…

Consumers intuitively get cloud and have been more than willing to embrace it warts and all as it has matured. They are highly price sensitive and will easily sacrifice features in exchange for low cost. On the other side of the market spectrum, the more conservative enterprise market is still struggling with the basic model. Some companies worry that the emerging cloud companies are too small to do business with, while others are concerned with how to incorporate new technologies into existing technology portfolio investments. To address their concerns, at the same time as Cloud technology continues to mature and standards develop , innovation around service delivery and breakthroughs in storage technologies are making it ever more enterprise ready. The pace of cloud vendor consolidation has already picked up as the traditional enterprise vendors such as HP and IBM has rushed to add enterprise ready cloud services to their portfolios. This should alleviate the fears of even the most technology adverse companies.

A final word for any remaining cloud technology skeptics, as with any innovative force, the cloud is just a tool to drive change, not an embodiment of change itself. Enterprise cloud consulting leaders, have firsthand experience with how companies willing to ride the Cloud revolution will not only survive in today’s hyper-competitive world, but thrive. I cannot wait to see where the next wave takes us!

About the Author
Beth Cohen, Beth Cohen is a senior cloud architect for Cloud Technology Partners, Inc., focused on delivering solutions to help enterprises leverage the efficiencies of cloud architectures and technologies. Previously, Ms. Cohen was the director of engineering IT for BBN Corporation, where she was involved with the initial development of the Internet, working on some of the hottest networking and web technology protocols in their infancy.

As I mentioned my previous post on the recently concluded OpenStack Summit held in San Diego October 2012, OpenStack needs to be taken seriously by anyone who is interested in building a public or private cloud.  For the more technically inclined the latest Folsom release has several new modules and features of interest to the enterprise and cloud service provider alike:

Quantum – Software Defined Networking (SDN), the latest darling of the cloud world is moving forward quickly with some really valuable features including L2 to L3 tunneling and a network API.  Expect to see lots of new development here.  On a side note, there was a great panel on the future of SDN moderated by Ken Pepple from Cloud Technology Partners, with people from Midokura, Big Switch and HP Cloud Services talking about their vision of the future for SDN.

Cinder – Now that Cinder has been spun out as its own named project, new features include intelligent location of Virtual Machine image storage, real snapshots, live migration, and more capability for large scale simultaneous VM initiations.

Keystone – New features includes support for role-based identify management (RBAC), PKI functionality, tools to add integration capability with enterprise grade account management systems such as Active Directory and other LDAP based systems.  Rumors of Kerberos support coming were floating around.

Documentation and security are both finally being taken seriously, with a half day documentation track on Monday morning and a full day of security related sessions on Thursday.  There was recognition among the lead techs that the developers who are creating OpenStack are not the users.  This resulted in much discussion regarding new features to make it easier to do deployments, upgrades (finally!) and manage it by operations folks.  There even was some discussion of IaaS/PaaS integration and how the VM’s work with the platform, a long overdue recognition that the IaaS and PaaS layers of the cloud stack are intimately related.

On the subject of operations, the tools to manage OpenStack are still weak, but they are no longer non-existent.  Ceilometer and heat among others look promising, but the better operations management tools are still mostly part of separate distributions such as Cloudscaling, Nebula, StackOps and others rather than being part of the product core.  While I do see the reasoning behind this thinking, at the very least we really need to have some standardized installation tools.  There is little benefit for every instance and distribution to reinvent the wheel.

One final note, one thing that was quite noticeable was that the Grizzly design sessions had a very different vibe from previous summits.  While some were packed with lots of discussions about new features, more often the Technical Project Lead and a few others ran the show with little input from the other participants in the room.  As the number contributors grow, the community is going to need to work hard to avoid having the immediacy of the Summit be diluted.

That doesn’t mean there isn’t plenty of meat already, but as can be seen by this tiny sampling of the current hot projects, there is still much work to be done.  The good news is the OpenStack community is ready willing and capable of delivering the goods in the coming months and years.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

From CTP’s perspective, since the majority of our customers are the big enterprise, we need to evaluate IaaS providers against a very different set of criteria. The enterprise is going to be more cautious – partially because they need to be and partially because they can afford to be — about putting anything beyond dev/test in these types of public IaaS services.  Gartner only touched on the IaaS requirements that are really important to the enterprise, regulatory compliance, true high availability SLA’s, support for large scale global deployments and a cost structure that isn’t essentially linear.  From that perspective, the Magic Quadrant would look very different: CSC, IBM, HP, Savvis and Terremark would all be top tier players, with Rackspace and Amazon in the niche quadrant.

On a side note, it is quite noticeable just how many of the providers on the list have at least some OpenStack built into their infrastructure.  HP Cloud Services is 100% based on OpenStack, while RackSpace is migrating from their legacy infrastructure as fast as they can convert their customers.  Internap, AT&T and others have either already stood up OpenStack services or have announced that they are in the works.  Definitely validates the OpenStack approach.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

After four extraordinary days of immersion into all things OpenStack at the just concluded OpenStack Summit in San Diego, I can heartily say without reservations that OpenStack is not only very real, it is truly a game changer for the IT industry.  With that being said, yes, there is certainly plenty of hype about this Open Source Infrastructure as a Service (IaaS) offering – one wag even quipped that OpenStack was at the peak of the Gardner hype cycle, so it must be real.  Here are just a few highlights for the terminally busy:

• The technology has come a long way since the project’s August 2010 inception.  With 13 major corporate foundation contributors, it seems like the only major technology companies not involved are Amazon and Apple.
• There was much discussion about the newly created OpenStack Foundation as an organization and its long-term viability.  However, compared to where the Apache Foundation was at the same time in its lifecycle, the OpenStack Foundation is far ahead of the curve.
• OpenStack is definitely gaining market traction.  There were plenty of academic, enterprise and service provider IT folks kicking the tires.  To encourage more market adoption, the website is featuring user stories.  A new one from WebEx was inspired by a visit to the Boston conference where the engineers saw the Mercardo Libra success story.
• While there was still a preponderance of developers in attendence (it was a design summit after all), there were more operations and business types than ever.  The vendor area was busier than ever.
• The biggest technical buzz was around Quantum and virtualized Software Defined Networking (SDN) — standing room only at the technical update and all the design sessions, but Cinder, and Keystone were not far behind in incorporating new features and functionality.
• Every time I turned around there seemed to be a new distribution available from both the usual suspects, such as Piston and Nebula, and the big players such as Cisco, SuSe and ???.  Niki Acosta’ Vapor Trail blog has a nice summary of the many vendor announcements.  More on the case of the proliferating distributions coming…
• And finally, the parties this time were for the most part more subdued.  HP’s soirée at the New Children’s Art Museum was cool, but the endless Techno Musak at all of them just gave me a pulsating (pun intended) headache.

As can be seen from this tiny sampling of the Summit activities, an enormous amount of energy and momentum has built up in just over two years.  For a project that will likely eventually be as important as Apache and Linux, every organization and person who has contributed to it should be justifiably proud of their accomplishments so far.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

In a previous discussion of cloud high availability, I covered in general terms what are some of the principals and approaches that make sense in a cloud environment. This time we will dive into some details of how this can be achieved in an Openstack environment.

The average published MTBF on switches seems to be between 100,000 and 200,000 hours. This number is dependent on the ambient temperature of the switch in the data center. I am assuming that most modern data centers are properly cooled for maximum switch life. This translates to between 11 and 22 years. Even in the worst case of poor ventilation and high ambient temperatures in the data center, the MTBF is still 2-3 years based on research found at http://www.garrettcom.com/techsupport/papers/ethernet_switch_reliability.pdf.

The mean time to replacement (MTTR) for a switch is going to be dependent on how exactly how the data center is staffed and what processes are used for replacing switches. Assuming that you would keep a few spares in the data center and that it is fully staffed 24 hours/day, the average time to replace a switch including configuration is going to be under 2 hours. Most modern switches are auto-configured so the actually provisioning time after the switch is powered up in the rack is under 5 minutes.

Let me walk through what will happen in the case of a top of rack (ToR) switch failure in the Swift cluster. Swift by its nature is fault tolerant at the rack level. That means that the system will continue to operate without data loss if an entire rack goes off-line. The cluster would detect the rack being off-line and send out a notification that the NOC staff would see within 5 minutes. In the case of rack going off-line Swift does not automatically move any data. The reason for this is that in fact, the NOC staff needs to make a decision about the cause of the rack going off-line and how long it will take for it to come back on line. In the case of a switch failure, the data in the rack is still intact, so it is far more efficient to just replace the switch then bring the rack back on-line without having to move the data. Even if the NOC staff decides to move data around, which they would only do if the fault is in the servers not the switch, the network overhead that it adds to the cluster is in the range of 3-5% for a large cluster with properly tuned ring rebuild cycle. Clearly taking a rack off-line is not considered a problem. I would argue that you should expect to be able to take racks off-line with no impact to the system as a whole as a matter of course for maintenance, upgrades and other reasons.

Nova behaves is slightly differently in the case of a rack failure. Unlike Swift the architecture does not have an assumed base unit of failure at the rack level. It does have the concept of a availability zone, which is quite different from a Swift zone just to confuse things. That doesn’t mean that you cannot create an equally fault tolerant Nova architecture, it just requires more development of high availability at the application level of the system combined with the use of the availability zone as a mechanism for balancing the applications in different locations. The assumption is that it is the responsibility of the application to build in fault tolerance, not the underlying infrastructure to keep track of the individual VM instances. Nova zones can be used to achieve this level of fault tolerance in combination. Combining this with the a live migration functionality and HA application design will allow you to build support for rack level failure. Again, the metrics for determining the next steps (replacement of switch only or rebuilding of entire rack) will be based on the specific component failure. See the recent discussion of this at http://lists.us.dell.com/pipermail/crowbar/2012-January/000643.html for more ideas on how to architect such a system.

Another approach would be to create high availability through redundant hardware. In this case you could provision the racks with two switches. However this is an expensive option in a large data center with hundreds of racks. It is clearly orders of magnitude more expensive to take this approach. From a risk perspective, you have substantially increased your per rack costs with little or no reduction in risks since the rate of failure is so low to begin with and the architected unit of failure for a cloud infrastructure should be at the rack level to begin with.

About the Author

Beth Cohen, Cloud Technology Partners, Inc. Transforming Businesses with Cloud Solutions

To understand how to manage your high availability options, it is important to have a discussion about how high availability, risk and component failure work in a cloud environment.  High availability is very important for cloud environments; often the cloud provider is required to meet strict service level agreements for 99.99% or 99.999% (the so called 5-nines) availability.  In theory, that is the very reason that customers are interested in using cloud services.  It should be noted that most of the public cloud service providers have lots of methods to measuring availability that is in their favor, so even in the case of catastrophic systems failure, they are rarely accountable for the downtime.  This has been one of many sources of caution for enterprises that have wanted to leverage public cloud services.

Before going into the details of how to quantify the cost of risk mitigation for a cloud, a short discussion of the science of risk management will help with understanding how it all works.  The goal of business risk management is to detail what kinds of risks exist in your specific business and determine how to prevent them entirely or minimize their impact on the business as a whole. Business risk management is essentially quantifying the risk that a given system will fail multiplied by the cost.  Cost is further broken into two more categories.  Out of pocket costs, also referred to as sunk costs, and lost opportunity costs.  Sunk costs are costs that you will need to pay out to fix the problem, while lost opportunity costs are revenue lost due to the system unavailability.  For example, the risk that there will be regular earthquakes in Japan is high.  The Japanese have responded to this threat by having some of the strongest earthquake resistant building codes in the world.  However, as last year’s 9.0 tremor and following tsunami so dramatically demonstrated, it is impossible to prepare for such extreme and rare events.

High availability is best addressed by redundancy.  However, redundancy can be achieved at several levels of the IT infrastructure: hardware, software, network, or a combination.  Traditional IT organizations have reduced the risk of downtime by concentrating almost exclusively on hardware redundancy.  The scale of the cloud, where there are already thousands or hundreds of thousands of systems, hardware redundancy at the component level quickly becomes unsustainably expensive.   A telling scholarly article that looked at the reported hardware failures from several large data centers shows that by far the most likely failure at data center scales is as would be expected the components that have moving parts, such as hard drives and power supplies.

In practice, the enterprise needs to balance the cost of duplicating hardware throughout the cloud ecosystem, which is the traditional approach to solving the risk management problem, against the need for keeping operating expenses low. Another consideration is that duplicating everything at the hardware level does not automatically guaranty that you do not still have a single point of failure in the environment.  For example, you might have remembered to contract with multiple carriers to spread the risk of a network outage, but if they all come into the data center at a single location, the data center is still prone to a catastrophic “backhoe failure”, which is what happens when a backhoe has severed all the up-link cables in one fell swoop.  It is an expensive and time consuming repair that leaves many unhappy customers in its wake.  Yes, there are ways to mitigate this risk, but they are expensive and need to be balanced against the relative probability of such an event.

The best approach is to look at the probability of failure of each component in context of the entire ecosystem.  Since hard drives fail at such a high rate, the hardware approach is to mirror or RAID the drives across thousands of systems.  This translates to data redundancy and added costs that far exceed the optimum for availability and cost reduction.  At scale, building a storage system that handles the data redundancy at the software level is far more efficient.  Another examples, is planning for power supply (or more precisely fan) failures.  Again, since they are generally the next most common component to fail, instead of filling the cloud data center with thousands of extra power supplies and fans, it is better to build the cloud to be resistant to downtime if a server node fails.  In the end, addressing cloud high availability is not only about determining the MTTF of hard drives, cables or switch ports, it is also balancing it against the likelihood of a given failure at the data center macro level.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

The magic of the cloud is that it can do anything.  It is both robust and flexible, the best of both worlds.  Ok, I admit that I have been spending far too much time reading cloud marketing materials lately.  Now back to reality.  Yes, the cloud is highly flexible and it can do almost anything, but if you want to get the most out of your private cloud investment, you need to pay attention to the underlying hardware as I discussed previously, and you need to define what you are planning on using it for by creating and testing use cases.

Use case planning seems counter-intuitive.  After all you can sign up for a web server with Amazon in about 5 minutes.  Amazon does not know what you are planning on doing with it.  Wrong.  Amazon’s product management department spends plenty of time figuring out exactly what would be attractive to their typical customer and honing the service to deliver it.  For the enterprise, the planning process is no different, but instead of planning for an external paying customer, for example, the use could be for internal application developers or a web portal.

To give you an idea of how this works, let us say, you are planning on using the cloud for the company’s E-commerce website.  This means that you will need to plan for applications that will support thousands of sessions per second, variable workloads and lots of complex and changing data.  By identifying the key metrics such as number of concurrent transactions per second, size of database, etc. you can then build a method for testing your assumptions.

To get the conversation started here is a short list of possible use cases for a private cloud.  Over the next few weeks I will be digging deeper into how to leverage the cloud model in the enterprise.

Archive storage — Many companies have moved to keeping their archives on line instead of on backup tape for many excellent reasons.  Using SAN or near-line storage is still expensive.  Cloud object or block storage is an attractive alternative because of its optimized approach to high availability.  It also scales nicely as archives grow over time.

Federated hypervisor/VM management – This is one of the main reasons that the enterprise is interested in the cloud in the first place – any server, any service, any time.  Adding self-service, charge back and transparent delivery of the right resources from a federated pool can be very cost effective.  Look for a cloud that provides cross platform hypervisor support and robust VM management tools.

Development and test – One of the best use cases for an enterprise cloud is a shared development and test environment.  Self-service is essential, but the private version allows much more control on resource use by using a rules based delivery model to optimize IT investments.  Creating an enterprise PaaS environment is also desirable because it allows better integration across applications and more standardized application development.

Application spaghetti rationalization – An enterprise cloud delivers better application portfolio management and more efficient deployment by leveraging self-service features, rules for deployments based on types of use.

Web services, portals and e-commerce – Web services of all sorts are a natural for the enterprise cloud.  They are well suited to being able to take advantage of the inherent elasticity and automated workload based provisioning and deployment capabilities.

VDI Support – VDI is another natural for an enterprise cloud.  VDI is often used to better maintain control over workers’ compute environments, but the workloads are inherently highly variable, which is an excellent reason for implementing such systems on the cloud.  An obvious extension is mobile application support which is a growing part of the enterprise service portfolio.

Disaster Recovery/Business Continuity — Again the cheap storage and VM management makes a good case for using the cloud as a secondary site.  The public cloud is already heavily used for these purposes, but moving the function in-house could be cost effective for a very large enterprise.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions