From the Enterprise perspective, now has never been a better time to invest in cloud services. Enterprises are broadly adopting all types of cloud services at multiple levels in the organization. Initial predictions were for the enterprise to favor private and community clouds over public services, but the hottest trends in Cloud adoption has been Software as a Service (SaaS) and cloud-based business applications which are projected to grow from $13.4 billion in 2011 to $32.2 billion in 2016, a 19.1% five-year CAGR! The rapid adoption of SaaS applications of all flavors by the enterprise has been a surprise to many, but the vastly reduced costs due to the pay as you go pricing models, and high degree of service delivery flexibility has overcome any perceptions of needing to trade price for reduced feature sets. Unsurprisingly, the most often added cloud-based application services are Customer Relationship Management (CRM), Supply Chain Management (SCM) and Enterprise Content Management (ECM), with Web Conferencing, teaming platforms and social software suites nipping at their heels.

Infrastructure as a Service (IaaS) continues to appeal to companies that want to replace in-house and traditional data center models for the cloud hardware abstraction approach. Gartner is predicting Infrastructure-as-a-Service (IaaS), cloud management & security devices, and Platform-as-a-Service (PaaS) are growing from $7.6B in 2011 to $35.5B in 2016, a CAGR of 36%. A word of warning for companies that think just because the Amazon cloud is easy to use, it is easy to build and manage. Building a private cloud still requires a degree of expertise that few enterprises have in-house. In the next year or so expect more companies to see the value of using the services of cloud consultants to avoid painful and expensive mistakes when building private clouds; or alternatively using emerging enterprise focused public cloud services such as Bluelock or Terremark.
Platform as a Service (PaaS) which still does not quite know what it wants to be when it grows up, is lagging with only about $1B in revenue in 2012, but as the market matures, expect to see rapid uptake as companies recognize the value of standardized tools that can ease the pain of Cloud application deployments. Some of the newer PaaS tools like the ServiceMesh Agility Platform combine the features of a SDLC workflow engine, production support and orchestration across different cloud platforms.

New Cloud Technology Directions

Ultimately, innovation is the marriage of technology and organization change. The dilemma is how to pull innovation into IT core functionality without disrupting the flow of new ideas, when the modern enterprise understanding of IT is focused on operational excellence and cost control. The CIO should be leading the cultural change to the new flat organization by leveraging cloud and mobile application in new and interesting ways. The trick is to create tools for people to quickly test and validate their ideas so they can implement new ideas that work within the enterprise framework. Cloud tools can be used to deliver on that promise, but is the enterprise up to that challenge?

This is not the first time wrenching technology innovation has changed how companies do business, and it certainly will not be the last. Companies able to use disruptive technologies, such as cloud computing, effectively will leave companies who do not have that ability in the dust. By recognizing the seeds of change and embracing them, the smart company can leverage cloud services by using the efficiencies of pooled IT resources at the same time allowing greater flexibility to meet the challenges of the global economy. This combination of combining commodity utilities with innovation allows companies to compete effectively using the efficiency and flexibility strategies simultaneously.

Challenges, Always Challenges…

Consumers intuitively get cloud and have been more than willing to embrace it warts and all as it has matured. They are highly price sensitive and will easily sacrifice features in exchange for low cost. On the other side of the market spectrum, the more conservative enterprise market is still struggling with the basic model. Some companies worry that the emerging cloud companies are too small to do business with, while others are concerned with how to incorporate new technologies into existing technology portfolio investments. To address their concerns, at the same time as Cloud technology continues to mature and standards develop , innovation around service delivery and breakthroughs in storage technologies are making it ever more enterprise ready. The pace of cloud vendor consolidation has already picked up as the traditional enterprise vendors such as HP and IBM has rushed to add enterprise ready cloud services to their portfolios. This should alleviate the fears of even the most technology adverse companies.

A final word for any remaining cloud technology skeptics, as with any innovative force, the cloud is just a tool to drive change, not an embodiment of change itself. Enterprise cloud consulting leaders, have firsthand experience with how companies willing to ride the Cloud revolution will not only survive in today’s hyper-competitive world, but thrive. I cannot wait to see where the next wave takes us!

About the Author
Beth Cohen, Beth Cohen is a senior cloud architect for Cloud Technology Partners, Inc., focused on delivering solutions to help enterprises leverage the efficiencies of cloud architectures and technologies. Previously, Ms. Cohen was the director of engineering IT for BBN Corporation, where she was involved with the initial development of the Internet, working on some of the hottest networking and web technology protocols in their infancy.

Enterprise SaaS takes off – With $14.5 billion in SaaS sales (an increase of 17.9% from 2011), lots of providers are getting it right. The biggest news is the boom in enterprise SaaS application adoption. Originally touted as the great leveler, allowing small businesses to take advantage of sophisticated systems hereto only affordable by big companies, the enterprise is rapidly dumping their high maintenance in-house systems and deploying a variety of SaaS services instead. Salesforce of course has long been a big player in this space, but the Workday’s spectacular IPO in October indicates a bright future. Look for lots of interest in Microsoft cloud products such as Office 365 and Azure as companies realize that these are cheaper and more flexible alternatives to traditional desktop tools. The rapid adoption of mobile devices in the workplace and its demand for more business customized apps is only going to accelerate this trend.

OpenStack grows up – While it remains to be seen if OpenStack wins the cloud infrastructure wars against VMware, CloudStack and Eucalyptus, given the number of cloud service providers lining up behind HP and Rackspace to roll out Openstack commercial services, it is not difficult to predict that 2013 will be another banner year for OpenStack. Practically the every major tech company in the world with the notable exception of Amazon is throwing their support behind OpenStack. On the technology side, more tools and functionality than ever makes the future of the largest cloud Open Source project ever definitely rosy.

Cloud Hardware Architectures get real – Over the last year or so, several vendors including VCE, the uneasy coalition of EMC, Cisco and VMware, Dell, and NetApp, have announced prepackaged cloud hardware stacks. On the surface the idea is appealing to enterprise IT infrastructure teams unprepared for the cloud revolution. However, as companies quickly found out, there is a big difference between dropping in a rack of hardware and building a productive enterprise cloud infrastructure. Since a primary cloud objective is hardware and software abstraction, more vendors will be developing infrastructure architectures tolerant of commodity hardware and supportive of transparent upgrades.

VMware Cloud gets it right – All indications are that 2013 will be the year that VMware finally gets it right after years of passing virtualization off as cloud. Enterprises that have been patiently waiting for a full suite of cloud features and tools will be rewarded with a system that will be expensive (what else is new), but actually delivers the goods.

Cloud Tools mature – With more offerings than ever from startups and mature companies alike, the market for sophisticated tools will be heating up as companies realize that they need orchestration, brokering, PaaS and cloud management suites. There will be lots of activity, new offerings, acquisitions, and of course, the inevitable hype.

About the Author
Beth Cohen is a senior cloud architect for Cloud Technology Partners, Inc., focused on delivering solutions to help enterprises leverage the efficiencies of cloud architectures and technologies. Previously, Ms. Cohen was the director of engineering IT for BBN Corporation, where she was involved with the initial development of the Internet, working on some of the hottest networking and web technology protocols in their infancy.

As I mentioned my previous post on the recently concluded OpenStack Summit held in San Diego October 2012, OpenStack needs to be taken seriously by anyone who is interested in building a public or private cloud.  For the more technically inclined the latest Folsom release has several new modules and features of interest to the enterprise and cloud service provider alike:

Quantum – Software Defined Networking (SDN), the latest darling of the cloud world is moving forward quickly with some really valuable features including L2 to L3 tunneling and a network API.  Expect to see lots of new development here.  On a side note, there was a great panel on the future of SDN moderated by Ken Pepple from Cloud Technology Partners, with people from Midokura, Big Switch and HP Cloud Services talking about their vision of the future for SDN.

Cinder – Now that Cinder has been spun out as its own named project, new features include intelligent location of Virtual Machine image storage, real snapshots, live migration, and more capability for large scale simultaneous VM initiations.

Keystone – New features includes support for role-based identify management (RBAC), PKI functionality, tools to add integration capability with enterprise grade account management systems such as Active Directory and other LDAP based systems.  Rumors of Kerberos support coming were floating around.

Documentation and security are both finally being taken seriously, with a half day documentation track on Monday morning and a full day of security related sessions on Thursday.  There was recognition among the lead techs that the developers who are creating OpenStack are not the users.  This resulted in much discussion regarding new features to make it easier to do deployments, upgrades (finally!) and manage it by operations folks.  There even was some discussion of IaaS/PaaS integration and how the VM’s work with the platform, a long overdue recognition that the IaaS and PaaS layers of the cloud stack are intimately related.

On the subject of operations, the tools to manage OpenStack are still weak, but they are no longer non-existent.  Ceilometer and heat among others look promising, but the better operations management tools are still mostly part of separate distributions such as Cloudscaling, Nebula, StackOps and others rather than being part of the product core.  While I do see the reasoning behind this thinking, at the very least we really need to have some standardized installation tools.  There is little benefit for every instance and distribution to reinvent the wheel.

One final note, one thing that was quite noticeable was that the Grizzly design sessions had a very different vibe from previous summits.  While some were packed with lots of discussions about new features, more often the Technical Project Lead and a few others ran the show with little input from the other participants in the room.  As the number contributors grow, the community is going to need to work hard to avoid having the immediacy of the Summit be diluted.

That doesn’t mean there isn’t plenty of meat already, but as can be seen by this tiny sampling of the current hot projects, there is still much work to be done.  The good news is the OpenStack community is ready willing and capable of delivering the goods in the coming months and years.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

You would think that cloud technology would have standardized long time ago.  While network standards that shape the Internet have been widely accepted throughout the industry, cloud standards have had a much slower adoption path.  Sadly the current state of cloud standards is, after 15 years, still far less mature than it ought to be.  As cloud infrastructure technology matures, increased interest by the enterprise and emerging vendors is driving a renewed effort to create viable standards that will benefit everyone.

For companies looking for integration and the capability to build hybrid clouds, various standards and proprietary and open APIs have been proposed to provide interoperability up and down the three layers of the cloud stack.  The first and so far only, cloud-oriented standard that has been ratified is the Open Virtualization Format (OVF), which was approved in September 2010 after three years of processing by the DMTF.  OVF’s open packaging and distribution format offers some platform independence by allowing migration between some platforms, but it does not provide all the tools needed for full cloud interoperability.

I think we can all agree that everyone benefits from technology standards in the long-term – the operative word is long-term.  You would think that everyone would agree that cloud infrastructure technology standards should be given high priority, but creating compelling proprietary systems and discouraging standards gives early adopter companies competitive advantage in the short-term.  Think about Amazon and VMware’s vast technology and market leads in the cloud services and enterprise infrastructure systems respectively.  They have little motivation to support any standards that have the potential to undercut their monopoly market positions.

Users of cloud are asking when will cloud computing standards mature enough so that more companies will feel comfortable implementing cloud architectures and using cloud services without feeling locked in.  Ironically, while the commercial cloud offerings have been growing, built on the very standards that created the Internet itself, Amazon and others have been reluctant to publish their architectures.  Application Programming Interfaces (API), which hide the underlying architecture, are all well and good, but they do not guaranty true interoperability.  Downstream vendors quickly find that they need to build API interfaces for all the different services they need to support; adding significantly to the development and maintenance costs.  To address this and transparently transfer workloads among the different vendor based on predefined business rules, there needs to be much more comprehensive standards.

One obvious question to ask is if there is an opportunity for the commercial cloud systems to become standards.  After all, there have been precedents where formerly proprietary formats, such as VMDK (Virtual Machine Disk) which was developed by VMware, have become de facto standards simply by being widely adopted by the industry.  One could even argue that platforms such as Amazon’s AWS are already standards.  However, as many companies have found to their chagrin, while it has a vast variety of services, easy to use tools, and a significant technological head start on the competition, it is more of a cloud world roach motel.  Lots of companies have found it easy to get applications running quickly, but changing providers or taking the applications back in-house as requirements change is fraught with unexpected perils.  Amazon’s backing of Eucalyptus does not address that problem directly, but it does offer a viable option for companies that want to build what Amazon euphemistically calls, on-premise services.

In conclusion, the good news is that the cloud industry is finally reaching consensus that the time to build cloud interoperability standards is long overdue.   The biggest need remains for interoperability standards to allow virtual machines to be migrated between clouds transparently and for more robust hybrid cloud solutions.  For the moment companies that want to use multiple platforms or a mix of public and private options are stuck with complex architectures and emerging orchestration tools such as enStratus and Rightscale to bridge the gap.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

We are commonly used to terabytes of data; 3TB hard drives are now usually available for under $200 and 4TB drives are starting to ship at premium prices.  It is not unusual for a company to have at least half a petabyte of data floating around on their storage systems these days and a petabyte of total data if you count all those forgotten data bases and buried servers.   Working in the storage industry I cannot tell you how many times clients would underestimate their actually storage data set by 50% or more.  SAN/NAS solutions, well understood technology that have been around for a while, are robust systems that reliably support storage pools of a petabyte or more.  However, as enterprises’ appetite for ever increasing amounts of data – so called big data – grows there is a need for new architectures that take a different approach to managing massive amounts of data (20 petabytes or more) at lower cost.  That is where object stores have the advantage over traditional storage approaches because they have the capability to store data very efficiently on commodity hardware, scale horizontally to essentially infinite size and seamlessly handle any type of data.

As enterprise data sets grow to tens of petabytes – i.e. beyond the scale of even the largest SAN/NAS solutions available today, there are some very attractive cloud systems that address the need for those ever expanding pools of storage.  It might be worthwhile to take a minute to understand how cloud storage works for very large amounts of data.  First introduced in 1993, object stores, unlike traditional file systems that maintain some type of hierarchical organization using the file and folder analogy, take a different approach.  Each file is treated as an object – hence the term object store – and the objects are placed in the store using a distributed data base model.  Having no central “brain” or master point of control provides greater scalability, redundancy and permanence.  It is not a file system or real-time data storage system, but rather a long-term storage system for a more permanent type of static data that can be retrieved, leveraged, and then updated if necessary. The details vary of course, but the ability to find objects from anywhere in the store using a distributed retrieval mechanism is what allows the stores to handle multiple petabytes of data.  It is ideal for write once, read many types of data pools.  Primary examples of data that best fit this type of storage model are virtual machine images, photo storage, email storage and backup archiving.

The advantage of moving from a SAN storage solution to a cloud solution for very large amounts of data makes sense for many use cases.  Some of the advantages include:

• Widely deployed proven technology with hundreds of petabyte data storage in production today
• Most cost efficient solution for the scale – Substantially lower per gigabyte per month storage costs
• Reduced  data center floor space utilization
• Enhanced flexibility to meet fluctuating storage demands
• Potential for delivering faster throughput to applications and a better end-user experience
• Highly scalable object storage
• Capable of creating seamless storage pools across multiple back-end systems
• Ability to scale horizontally instead of vertically
• The horizontal architecture scales well beyond the 20 Petabytes maximum that traditional storage architectures allow
• Uses interchangeable commodity hardware
• Simplified operations

The average cost of commercial fully managed cloud storage is running $.11-.15/GB/month.  That might be a bit high for companies that have massive data storage needs, but an organization that has the wherewithal to build it in-house can bring the costs down substantially, easily to under $.05/GB/month.  Remember, for every 10 petabytes of data, every additional $0.01/GB/month of savings represents $1.2M/year.  For one such model, check out Amar Kapadia’s blog on cost projections for building an Openstack Swift store, Can OpenStack Swift Hit Amazon S3 like Cost Points?

In the end, if you have more than 10 petabytes of data, it might be worth checking out cloud object storage to take advantage of its ability to cost effectively and transparently scale to hundreds of petabytes.   With the right data set, a company can achieve significant savings and support planned growth.  In addition, object storage offers a more flexible architecture for future growth, and improved control over operational and capital costs.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

People have been managing data centers for 30-40 years now, so that should mean that there are a good set of standard best practices for building highly available resilient components.   That is true for the old style data center, but the old best practices are expensive and do not scale well for cloud architectures.  Duplicating hardware to protect against failure works well when you have hundreds of components but the costs are linear so it does not scale.  Unlike traditional IT operations, over-design to protect against obsolescence is not desirable when scaling to thousands of nodes.  For example, spending an extra $6000/rack for 10GB switches might seem to be a sensible way to protect against hardware obsolescence if you have 10 racks, but that extra cost is much harder to justify when you are provisioning a 100 racks and it has turned into an extra $6 million!

The principal of ‘replacement management’ takes on great importance when managing the thousands of physical devices required for a cloud deployment.  The advantage of the cloud is that you do not need to build expensive high availability redundant systems because an assumption that components will fail is built into the architecture.  By leveraging the huge pools of cloud resources, the level of redundancy can be considerably reduced.  If a component fails, the system will continue to work until someone replaces it.  Since commodity low price devices typically have a high rate of failure, the whole architecture needs to be based on “availability” and “partial failure”.

In a cloud environment, it makes much more sense to just replace a component than worry about what caused the failure and trying to troubleshoot it.  The most common components to fail are disks, since they are mechanical moving parts.  A typical disk failure rate in a cloud data center is about 10-15%.  However, fans, power supplies and memory will also fail less frequently.  For example, the OpenStack Swift architecture assumes that disks, systems and entire zones can and will disappear (fail) at any time.  Yet, there are only three copies of every file, and no additional redundancy in the hardware.

This approach to failure at scale can be very cost effective, but it takes different mindset from traditional operations.  Every cloud operations engineer for cloud should learn what is in the service, where the critical parts are located, and how to replace a failed component, then incorporate the knowledge into standard operations processes.  Automated tools need to be written to help identify the location of failed disks and other components so they can quickly be isolated from the environment and replaced.  To maintain a high level of robustness without sacrificing cost efficiency, the system needs to be designed to replicate data on the application/software level, not disk or network level.

In conclusion, the biggest paradigm shift is that development and operations groups need to work together to optimize the systems and drive down costs.  Tests and metrics need to be created to determine the optimum systems configurations.  By understanding how changes in the components affect the systems as a whole, it will allow you to flexibly configure the systems to meet the application requirements as they change.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

In our work, clients often ask for a cloud reference architecture.  They see it as a holy grail that will provide clear vision and help define their cloud strategies.  In response to a recent client inquiry about a cloud hardware reference architecture that could be used to define a joint venture with another company, I realized that defining a cloud reference architecture as the same hardware platforms was the wrong way to look at the question.  While I do agree a reference architecture is certainly a good starting point, it not something I would build an entire corporate cloud strategy around.

Reference architectures are supposed to be by their nature somewhat theoretical.  Like the OSI model and the cloud stack model, they are artificial constructs or frameworks that can be used to create real systems.  By themselves they tend to be so generic and broad that building an enterprise cloud based on a reference architecture without considerable work defining business objectives and system requirements would only be an exercise in frustration.  For example, an enterprise cloud designed to be used for internal use as a more flexible development platform is a far different animal than an enterprise cloud planned to be used to support massively scalable customer facing applications.

After looking at a few available cloud reference models, a reference architecture means very different things depending on the agenda of the creator.  A useful way to compare the perspectives of the various cloud reference architectures is to map them to the older and simpler Cloud Service Stack Model: IaaS, PaaS and SaaS.  For example: both the NIST and its closely related IBM architecture are relatively generic and high level, but they both have an operational focus which closely matches the IaaS layer.  The recently published RackSpace Private Cloud Reference Architecture while specific to OpenStack, also primarily has an operational/IaaS bias.  Microsoft has defined the cloud not surprisingly more from a development platform/PaaS view, but it also has an IaaS flavored version based on Hyper-V.  The HP and VMWare versions are more appropriate for companies building end to end applications, SaaS or otherwise.

In the end, a good cloud reference architecture should be robust enough to allow it to be used from a variety of different perspectives: business, operations, development and consumer.  However, any company that is foolish enough to try to use a cloud reference architecture to create its strategy without the application of a rigorous amount of common sense and basic good business practices, is likely to be disappointed with the results.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Moving companies’ IT services into the cloud the right way, the first time!