I totally agree with you that inband compression is scary – it affects performance of writes and reads, and does not give you enough choice about what to compress or when. Fortunately, in-band compression is not the only option these days. Files can be compressed in the background, after they are saved. There’s no reason you have to compress a whole disk, share, or directory either. Modern tools can use policies to decide what to compress (eg, all files with the extension .mp3, or all files that have not been modified for 60 days) and how aggressively to compress it (compress for maximum space saving or compress for fastest decompress time?).

Dedupe is just one form of compression, and by no means is it the most effective for online storage (the files on your hard disk or NAS share). Dedupe is good for backups, because repetitive backups create duplicates. You won’t find as many dupes in your online set of files as you will in 30 or 365 days worth of backups. Different compression techniques are called for if you are trying to reduce the size of an online data set.

Further, almost every file that is driving today’s storage growth is already compressed. Microsoft Office 2007 compresses every file on save, and PDF, JPEG (as you point out), all video formats, and most other common file types already include some form of compression done by the application itself when it saves its files. Typically, that compression is some variant of a common generic algorithm – Lempel-Ziv for example. So if the native format of a file is compressed, it’s hard to say that additional lossless compression would alter its legal validity.

For compliance purposes, compression has to be bit-for-bit lossless, and that can be verified by taking and storing cryptographic checksums before compression. That way, you can always compare your decompressed file with that original checksum to see if it’s bit for bit the same. That makes sense if you’re talking about corporate memorandums; it might not be as important if you’re talking about your JPEGs of the family vacation last summer.

While in some cases it’s become the archive your backup shouldn’t *be* your archive. If someone goes down the route of invalidating compression due to the fact it modify’s the data that makes everything written to a tape drive with onboard compression enabled (or encryption enabled) suspect. The same thing if you’re using backup software to encrypt or compress data or using backup software at all since that will inject it’s own meta-data into the mix as the backup is written out in the backup app format.

Ultimately all of these de-dup technologies (Except encryption but including thin provisioning) are *capacity optimisation* technologies. There is something different to chose depending on your needs.

I have a MS launch T-shirt somewhere which says “We came, we saw, we DoubleSpaced”