As I mentioned my previous post on the recently concluded OpenStack Summit held in San Diego October 2012, OpenStack needs to be taken seriously by anyone who is interested in building a public or private cloud.  For the more technically inclined the latest Folsom release has several new modules and features of interest to the enterprise and cloud service provider alike:

Quantum – Software Defined Networking (SDN), the latest darling of the cloud world is moving forward quickly with some really valuable features including L2 to L3 tunneling and a network API.  Expect to see lots of new development here.  On a side note, there was a great panel on the future of SDN moderated by Ken Pepple from Cloud Technology Partners, with people from Midokura, Big Switch and HP Cloud Services talking about their vision of the future for SDN.

Cinder – Now that Cinder has been spun out as its own named project, new features include intelligent location of Virtual Machine image storage, real snapshots, live migration, and more capability for large scale simultaneous VM initiations.

Keystone – New features includes support for role-based identify management (RBAC), PKI functionality, tools to add integration capability with enterprise grade account management systems such as Active Directory and other LDAP based systems.  Rumors of Kerberos support coming were floating around.

Documentation and security are both finally being taken seriously, with a half day documentation track on Monday morning and a full day of security related sessions on Thursday.  There was recognition among the lead techs that the developers who are creating OpenStack are not the users.  This resulted in much discussion regarding new features to make it easier to do deployments, upgrades (finally!) and manage it by operations folks.  There even was some discussion of IaaS/PaaS integration and how the VM’s work with the platform, a long overdue recognition that the IaaS and PaaS layers of the cloud stack are intimately related.

On the subject of operations, the tools to manage OpenStack are still weak, but they are no longer non-existent.  Ceilometer and heat among others look promising, but the better operations management tools are still mostly part of separate distributions such as Cloudscaling, Nebula, StackOps and others rather than being part of the product core.  While I do see the reasoning behind this thinking, at the very least we really need to have some standardized installation tools.  There is little benefit for every instance and distribution to reinvent the wheel.

One final note, one thing that was quite noticeable was that the Grizzly design sessions had a very different vibe from previous summits.  While some were packed with lots of discussions about new features, more often the Technical Project Lead and a few others ran the show with little input from the other participants in the room.  As the number contributors grow, the community is going to need to work hard to avoid having the immediacy of the Summit be diluted.

That doesn’t mean there isn’t plenty of meat already, but as can be seen by this tiny sampling of the current hot projects, there is still much work to be done.  The good news is the OpenStack community is ready willing and capable of delivering the goods in the coming months and years.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

The traditional storage market has settled into several distinct streams.  On the high end is the EMC approach which is to build very complex storage systems with multiple redundant hardware plus many configuration options and features.  This very expensive approach works for enterprises that really need fast reliable storage and have the budget to justify the investment.  At the low end of the spectrum, there are lots of options for small storage pools — half a petabyte or so – which are generally sold through channels.  The prices for these systems have remained relatively stable at the $1 million for a petabyte range.

For companies that want massive amounts (15 petabytes or more) of cheap storage or a horizontally scalable cloud type architecture, the options have been limited to a few vendors or for the technologically adventurous, a roll your own system.  All this is happening despite the fact that there are now three hard disk manufacturers left, a de facto monopoly, and most users treat hard drives as interchangeable commodities with excessively high failure rates that require massive amounts of duplication.  Any vendor that can demonstrate a system that reduces the need for three data copies to maintain integrity by improving the failure rate from the current 10-15% a year, would deliver a big win for everyone.

As the cloud storage business grows the traditional vertically scalable storage is less and less viable.  There are several companies that are already working on building horizontally scalable storage, Scality and CloudBytes are two that come to mind.  There are also several Open Source projects like Openstack Swift that are taking on the large data store architecture problem as well.  Not all the cloud storage vendors are using a pooled block of storage approach.  ScaleIO turns all the disks in the servers into a pool of storage.  This radically different approach is dependent on the right use case and could have network bottleneck issues. The newer horizontally scaled cloudy architectures, such as Mezeo and Openstack Swift have finally opened up the possibility of building attractively priced massive data pools.

The biggest innovation of late is the incorporation of SSD technology.  Amazon is already offering an all SSD cloud storage service and a number of new storage vendors, such as SolidFire are building SSD only storage systems from the ground up.  SolidFire isn’t the first vendor with an SSD offering, Nimbus Data began selling all-SSD systems more than a year ago. EMC jumped in with a Symmetrix with all SSDs this summer, and others are expected to follow shortly. Texas Memory Systems, Avere, Violin Memory and Alacritech, have all-SSD systems designed to speed SAN and NAS performance for internal storage solutions.  Most storage vendors now give customers the option of installing some SSD alongside hard drives in their systems. In the long run, this is just an interim step as SSD prices will continue to fall over the next few years.  The future is a move to all SSD units coming to a data center near you.

It is good to see innovation coming back in the storage industry after years of slack.  Expect to see more hardware innovation as SSD technology becomes the standard for fast reliable cloud storage and new systems take advantage of more reliable hard disk hardware and continued rise in data densities.  My crystal ball says that we might see a repeat of the massive changes that occurred in the 1990’s as smaller disks rapidly swept away all the incumbent vendors who were focused on their established large customers.  Watch your back EMC…

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

We are commonly used to terabytes of data; 3TB hard drives are now usually available for under $200 and 4TB drives are starting to ship at premium prices.  It is not unusual for a company to have at least half a petabyte of data floating around on their storage systems these days and a petabyte of total data if you count all those forgotten data bases and buried servers.   Working in the storage industry I cannot tell you how many times clients would underestimate their actually storage data set by 50% or more.  SAN/NAS solutions, well understood technology that have been around for a while, are robust systems that reliably support storage pools of a petabyte or more.  However, as enterprises’ appetite for ever increasing amounts of data – so called big data – grows there is a need for new architectures that take a different approach to managing massive amounts of data (20 petabytes or more) at lower cost.  That is where object stores have the advantage over traditional storage approaches because they have the capability to store data very efficiently on commodity hardware, scale horizontally to essentially infinite size and seamlessly handle any type of data.

As enterprise data sets grow to tens of petabytes – i.e. beyond the scale of even the largest SAN/NAS solutions available today, there are some very attractive cloud systems that address the need for those ever expanding pools of storage.  It might be worthwhile to take a minute to understand how cloud storage works for very large amounts of data.  First introduced in 1993, object stores, unlike traditional file systems that maintain some type of hierarchical organization using the file and folder analogy, take a different approach.  Each file is treated as an object – hence the term object store – and the objects are placed in the store using a distributed data base model.  Having no central “brain” or master point of control provides greater scalability, redundancy and permanence.  It is not a file system or real-time data storage system, but rather a long-term storage system for a more permanent type of static data that can be retrieved, leveraged, and then updated if necessary. The details vary of course, but the ability to find objects from anywhere in the store using a distributed retrieval mechanism is what allows the stores to handle multiple petabytes of data.  It is ideal for write once, read many types of data pools.  Primary examples of data that best fit this type of storage model are virtual machine images, photo storage, email storage and backup archiving.

The advantage of moving from a SAN storage solution to a cloud solution for very large amounts of data makes sense for many use cases.  Some of the advantages include:

• Widely deployed proven technology with hundreds of petabyte data storage in production today
• Most cost efficient solution for the scale – Substantially lower per gigabyte per month storage costs
• Reduced  data center floor space utilization
• Enhanced flexibility to meet fluctuating storage demands
• Potential for delivering faster throughput to applications and a better end-user experience
• Highly scalable object storage
• Capable of creating seamless storage pools across multiple back-end systems
• Ability to scale horizontally instead of vertically
• The horizontal architecture scales well beyond the 20 Petabytes maximum that traditional storage architectures allow
• Uses interchangeable commodity hardware
• Simplified operations

The average cost of commercial fully managed cloud storage is running $.11-.15/GB/month.  That might be a bit high for companies that have massive data storage needs, but an organization that has the wherewithal to build it in-house can bring the costs down substantially, easily to under $.05/GB/month.  Remember, for every 10 petabytes of data, every additional $0.01/GB/month of savings represents $1.2M/year.  For one such model, check out Amar Kapadia’s blog on cost projections for building an Openstack Swift store, Can OpenStack Swift Hit Amazon S3 like Cost Points?

In the end, if you have more than 10 petabytes of data, it might be worth checking out cloud object storage to take advantage of its ability to cost effectively and transparently scale to hundreds of petabytes.   With the right data set, a company can achieve significant savings and support planned growth.  In addition, object storage offers a more flexible architecture for future growth, and improved control over operational and capital costs.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions