Dedupe has helped by effectively compressing backups into a smaller physical space, and server virtualization has simplified the process by encapsulating a server’s data into a single file. Snapshot and cloning technologies have reduced the impact of taking backups in real time and restoring them, and the cloud has provided nearly unlimited capacity and a ready-made off-site target.

Most backups today are done with these innovations, which improve performance and reduce costs. But most still involve a separate backup application moving data to separate storage in coordination with the primary storage infrastructure to get that process accomplished without too much disruption. But what if the backup application itself could be eliminated?

There are “converged storage” systems available that combine primary storage and data protection into the same systems, using onboard SSD and SATA drives. They use a log-based file/block architecture that maximizes data reduction and provides intelligence to keep data on the appropriate storage tier (flash or SATA), producing the performance required by primary storage. With integrated snapshot technology and extended effective capacity to store several months of snapshots, these converged storage systems provide internal data protection that can replace the backup process — and the requirement to implement a separate backup system. Replication to a second unit is available for local redundancy or a remote DR system. 

This approach represents another alternative for companies looking for a way to simplify storage infrastructures. Instead of just improving the backup process or incrementally reducing its cost, converged storage systems can eliminate dedicated backup altogether. The result can be a storage system that protects itself, reducing the time and resources required to do so.

Follow me on Twitter: EricSSwiss

The thought of primary deduplication certainly came up early on in the adoption cycle of the technology, but there were plenty of “high-value targets” for the dedupe vendors to go after in backup. When it was first introduced to backup customers, they were promised effective data reduction in the double digits — in the high double digits for some data sets — and by and large they got it. While dedupe has certainly not been adopted by everyone (current estimates hover around one-third for market penetration), dedupe vendors seem to be ready to move on.

Now with non-backup data, data reduction numbers are firmly in the single digits. While this wouldn’t have sold may dedupe appliances for Data Domain 10 years ago, it seems to be enough for this second wave of vendors in the primary storage space. But some have questioned the ability of dedupe to perform as a real-time data optimization technology, as it would be in this application. In order for dedupe to become a common technology in primary storage, users would have to be confident it would have no effect on data integrity and no appreciable impact on performance.

It turns out this isn’t as much of a stretch as many think, once they look at how dedupe works under the covers. Essentially, dedupe uses technologies similar to those developed for file systems and snapshots (or clones). It leverages extents, metadata management and references to existing data blocks that primary storage users have relied on for years.

As a VAR, you’ve probably seen primary deduplication included in some of your current vendors’ products, or at least you’ve been exposed to it in the news.  Dell recently purchased Ocarina Networks, which developed a primary deduplication technology. BlueArc and Xiotech have announced partnerships with Permabit to incorporate dedupe into their offerings. Based on the number of companies putting dedupe into their primary storage solutions, one would conclude that it’s not just for backups anymore.

Follow me on Twitter: EricSSwiss