Basically, dedupe requires (surprise) duplicate data blocks in the same data set in order to generate the opportunity to reference a block multiple times. The applications that suit this process are (surprise, again) backup, virtual machine images, traditional office documents and email, to name a few. These applications involve multiple instances of the same data associated with redundant operations or multiple people doing similar work.

Compression, thankfully, works well on data types that don’t do well with dedupe. It works on all data, although with varying effectiveness, and is a good solution for data sets that contain essentially unique data blocks. Media content, like audio and video files, are a natural fit, as are scientific and sensor-based data sets. These files can get very large, so even a small percentage of effective compression can make a difference. But what about running compression and dedupe together?

When dedupe first came out it was assumed that it wouldn’t work on compressed data. But this wasn’t accurate; combining compression and dedupe is actually a good practice, especially in primary storage systems, where data sets include a range of file types and software applications. In fact, compression and dedupe together can produce a higher rate of data reduction than the sum of both when run separately.

Primary storage systems are coming to market with deduplication embedded in their controllers, and dedupe technologies are evolving to meet this new use case. Like compression before it, primary storage dedupe carries lower overall reduction rates than backup. But the combination of dedupe and compression for primary storage can produce better results.

Follow me on Twitter: EricSSwiss