We are commonly used to terabytes of data; 3TB hard drives are now usually available for under $200 and 4TB drives are starting to ship at premium prices.  It is not unusual for a company to have at least half a petabyte of data floating around on their storage systems these days and a petabyte of total data if you count all those forgotten data bases and buried servers.   Working in the storage industry I cannot tell you how many times clients would underestimate their actually storage data set by 50% or more.  SAN/NAS solutions, well understood technology that have been around for a while, are robust systems that reliably support storage pools of a petabyte or more.  However, as enterprises’ appetite for ever increasing amounts of data – so called big data – grows there is a need for new architectures that take a different approach to managing massive amounts of data (20 petabytes or more) at lower cost.  That is where object stores have the advantage over traditional storage approaches because they have the capability to store data very efficiently on commodity hardware, scale horizontally to essentially infinite size and seamlessly handle any type of data.

As enterprise data sets grow to tens of petabytes – i.e. beyond the scale of even the largest SAN/NAS solutions available today, there are some very attractive cloud systems that address the need for those ever expanding pools of storage.  It might be worthwhile to take a minute to understand how cloud storage works for very large amounts of data.  First introduced in 1993, object stores, unlike traditional file systems that maintain some type of hierarchical organization using the file and folder analogy, take a different approach.  Each file is treated as an object – hence the term object store – and the objects are placed in the store using a distributed data base model.  Having no central “brain” or master point of control provides greater scalability, redundancy and permanence.  It is not a file system or real-time data storage system, but rather a long-term storage system for a more permanent type of static data that can be retrieved, leveraged, and then updated if necessary. The details vary of course, but the ability to find objects from anywhere in the store using a distributed retrieval mechanism is what allows the stores to handle multiple petabytes of data.  It is ideal for write once, read many types of data pools.  Primary examples of data that best fit this type of storage model are virtual machine images, photo storage, email storage and backup archiving.

The advantage of moving from a SAN storage solution to a cloud solution for very large amounts of data makes sense for many use cases.  Some of the advantages include:

• Widely deployed proven technology with hundreds of petabyte data storage in production today
• Most cost efficient solution for the scale – Substantially lower per gigabyte per month storage costs
• Reduced  data center floor space utilization
• Enhanced flexibility to meet fluctuating storage demands
• Potential for delivering faster throughput to applications and a better end-user experience
• Highly scalable object storage
• Capable of creating seamless storage pools across multiple back-end systems
• Ability to scale horizontally instead of vertically
• The horizontal architecture scales well beyond the 20 Petabytes maximum that traditional storage architectures allow
• Uses interchangeable commodity hardware
• Simplified operations

The average cost of commercial fully managed cloud storage is running $.11-.15/GB/month.  That might be a bit high for companies that have massive data storage needs, but an organization that has the wherewithal to build it in-house can bring the costs down substantially, easily to under $.05/GB/month.  Remember, for every 10 petabytes of data, every additional $0.01/GB/month of savings represents $1.2M/year.  For one such model, check out Amar Kapadia’s blog on cost projections for building an Openstack Swift store, Can OpenStack Swift Hit Amazon S3 like Cost Points?

In the end, if you have more than 10 petabytes of data, it might be worth checking out cloud object storage to take advantage of its ability to cost effectively and transparently scale to hundreds of petabytes.   With the right data set, a company can achieve significant savings and support planned growth.  In addition, object storage offers a more flexible architecture for future growth, and improved control over operational and capital costs.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions

Question:  Just when we were getting comfortable with storing our confidential data in the cloud, now I hear about the FBI shutting this company down.  Do I need to reexamine my cloud security policies?

A few years ago, I wrote a scary story about Widgitco a factitious company that found out the hard way that reading the fine print on cloud service contracts is important.  Widgetco’s problems stemmed from a lack of clarity about who actually owned the data in the cloud.  When Widgitco discovered their customer list had fallen into the hands of their main competitor they had little or no legal recourse because this issue had never been properly addressed by any of the parties in the fiasco, including Widgitco, its SaaS providers or the actual owners of the data center hardware.

Now the courts are addressing a dramatic and all too real example of the question of who owns data in the cloud and even more importantly, who is responsible for the files.  The recent legal problems of Megaupload, a filesharing service, not untypical of many others in the cloud storage and file sharing market, highlight these issues.   In a nutshell it boils down to the differing perspectives on the legal nature and purpose of file sharing.  Clearly there is lots of legitimate file sharing going on.  Dropbox, MediaFire, SugerSync, and even Megaupload, despite the government legal actions, all have a huge number of users who are using it to share personal files for both business and personal reasons.   According to MediaFire, employees at 86 percent of the Fortune 500 use its services.  They are not providing information on the nature of those files and what they are being used for, but I think we can safely assume at least some of the files are being legitimately used by collaborative teams to produce real work.  One can argue that using Dropbox and its ilk in the corporate setting does pose a serious risk for exposing sensitive corporate data.  I do agree with this sentiment, but I am also realistic about the reasons company employees are turning to these services in the first place.  Like the adoption of other consumer driven innovations like mobile devices and IM, it is often simply because the available internal corporate file sharing tools leave something to be desired.  How many of you have used a file sharing service as a team collaborative tool simply because it was easy to use and met the objective for expediency?

So what is the real issue?  The problem stems from a clash between the interests of the media content delivery companies such as Sony, Warner Brothers and others who are worried that these sites are primarily being used to share pirated copies of movies, e-books or music, and the far more common and benign private file sharing.  While one could argue that they should not be worried about the pirating in the first place, (Read Charlie Stross’s interesting take on that issue :  What Amazon’s ebook strategy means) that is a discussion that will be left to another time.  For the moment it appears that the government is siding with the large media conglomerates at the expense of everyone else.

Part of the reason that there is renewed government interest in prosecuting these services is a change in the technology and service delivery model.  In many ways these cloud file sharing systems are using a combination of the unenforceable peer to peer sharing strategy of BitTorrent, where the files are literally scattered all over the globe on millions of private computers, and the centralized server model of the late lamented Napster.  The government case boils down to the fact that the files are located in an identifiable set of machines that constitute a cloud file repository, therefore they can be subject to seizure.   I suspect the government case rests more on the relative ease of access to the servers than anything more valid.  A similar but more legally defensible case against the poor Boston University graduate student, Joel Tenebaum who got caught with pirated music garnered far more sympathy for the student than its intended anti-piracy warning message.

Where does that leave the legitimate business user?  The bottom line is that the on-going battle among pirates, business and the legal system will continue to work slowly through the courts, while the enabling technology to beat the system will remain far ahead of the law.  In the meantime, I think you can safely continue to use cloud file sharing services; just make sure they are business friendly and meet proper regulatory requirements for data security.

About the Author

Beth Cohen, Cloud Technology Partners, Inc.  Transforming Businesses with Cloud Solutions