Backup and recovery solutions that are comprehensive and include an effective dedupe functionality meet these requirements. In the data protection space there are a number of options for capturing a backup, from application-specific tools to products that are designed for a specific platform, like VMware. Obviously, point solutions are something to avoid, as they typically result in more management overhead, and IT managers turn to full-featured backup applications that can handle the vast majority of an organization’s data sources for this reason.

‘Progressive deduplication’

Deduplication has become a must-have feature in data backup, as a way to reduce the impact that backups can have on an IT infrastructure, especially storage. While the original dedupe solutions were implemented as target-side appliances, the shift is being made to source-side products. Doing the dedupe process on the client server reduces data upstream from the backup server and compounds the benefits of deduplication throughout the infrastructure. In addition to where the dedupe process is run, how it works is also important. Arkeia’s “progressive deduplication”[1]  is a source-side solution that uses specific block sizes for each file type, a process the company developed through extensive empirical research. This technology also handles data insertions to large files, something common in VMware environments, to produce a new level of dedupe performance.

Mobile devices

On the issue of data security, mobile devices (PDAs, smart phones and tablets) are the focal point as IT managers try to get their arms around a flood of new potential vulnerability points. At Symantec Vision last month, the topic of data protection and security around mobile devices was discussed by a panel from the industry. Several points were made that bear repeating. First, the “traditional” security method of storing a list of known threat profiles on each device is not possible with many mobile device types, since they don’t have the capacity that computers do.

Another issue is how to handle personal devices, like smart phones, that employees bring to work and expect to use with company data. While organizations can certainly restrict the use of these devices, there’s a growing concern about the need to allow this practice in order to attract and retain the best people. Personal mobile devices have become a part of everyday life, and people expect to use them at work. For example, who’s going to carry two cell phones? Part of the solution rests in education, a practice the government is certainly familiar with. Employees need a better awareness of the vulnerability their mobile device use can represent, simple things like the amount of data they hold and how easy they are to lose and steal. Also, taking a more holistic approach to security was advocated, like data loss prevention (DLP) methods that take a “content and context” approach to securing data itself, instead of simply making devices harder to penetrate.

Follow me on Twitter: EricSSwiss

Although another dedupe tool was available at about the same time, it was installed on the client server. The storage appliance implementation was the overwhelming favorite as a technology, since it didn’t require replacement of the backup software or tax client servers with the dedupe processing overhead.

Deduplication algorithms differ somewhat, but all use some method of examining each block of data, assigning it a unique identifier (a “hash key”) and comparing it with an index (a “hash table”) of all previous blocks. In this way, duplicate blocks can be represented by a reference (or pointer) to the original block, with the space savings coming from the act of recording the reference for each duplicate block instead the entire block. This hash table can grow very large and in some cases become the gating factor over how large the dedupe device itself can grow, since it must usually reside in memory. Currently, different dedupe tools use different block sizes, different ways to index the hash keys, etc., but still perform these same classify and look-up steps on data blocks that enter the dedupe engine.  

The deduplication algorithm is either applied “inline,” meaning it’s run as data enters the dedupe engine, or “post-process,” which caches data first, then runs the deduplication algorithm as a second step. Which method is better probably depends on the environment, and the issue is certainly debated. But in general, inline can require less storage space since it dedupes all data first, and post-process can require less backup window time, as it writes all data to disk without waiting to dedupe it. But where the dedupe process occurs is probably more important than how it’s done.

Some dedupe tools perform this process on the client server (source-side or client-side) and others on the storage array itself (target-side). Source-side dedupe reduces data volumes before they’re sent over the network, so it can reduce real backup window times and bandwidth requirements but puts the CPU load of running the deduplication algorithm on the client server. It also doesn’t typically include as much data in the dedupe comparison as target-side dedupe, so its compression numbers aren’t usually as good. Target-side dedupe technology is easier to implement (there’s one appliance as opposed to loading software on each client) and typically produces better compression. But it also loads the network with the entire backup data set and doesn’t reduce backup window time appreciably.

As dedupe technology has evolved, several variants have appeared and the technology has found its way into most storage device categories. Backup is still the primary application, with more successful source-side products available, but dedupe is also being put into primary storage arrays as an optimization technology, usually for lower tiers. Primary storage dedupe ratios are much lower than they are for backup data (typically single-digit percentages), due to the lack of redundancy in primary data, but it’s still worth the effort.

Deduplication isn’t the “gee whiz” technology that will get you appointments like it used to. It’s almost a checkbox item for backup storage products and not the differentiator it used to be. But as a VAR, it’s advisable to keep up with dedupe technology enough to understand which product to introduce into each application. Given the breadth of solutions available, more than a single dedupe product should be on the line card. Understanding how each works can also help you win deals against other dedupe technologies.

In general, the more storage processes that occur after dedupe, the better, as they will enjoy a smaller data set. This would indicate that, everything else being equal, source-side dedupe is preferable to target-side and any dedupe is a good idea before off-site replication. That said, everything usually isn’t equal and the dedupe technologies can be close enough in performance that there will be multiple appropriate solutions. This puts dedupe into the familiar “choosing a vendor” decision matrix that every VAR deals with on a regular basis.

Follow me on Twitter: EricSSwiss.

That leads to a question for Data Domain resellers, both those with an existing EMC relationship and those without one: How’ve you fared with the new corporate structure at EMC? Have you seen or do you expect any changes coming down the line? And does a product like GreenBytes’ have appeal as an alternative to Data Domain? Email me at stroy@techtarget.com.

To be fair, backup has “evolved” into an application that moves an entire organization’s information assets from a large number of random access devices to a smaller number of serial devices (in the case of tape), over a network in real time. A big job indeed, which is the point.

As a wise old storage engineer used to tell me, “The only way to improve backup is to do less of it.” So let’s improve your customers’ backups. The concept of archiving certainly isn’t new (see HSM, ILM, DLM, etc.), but it’s recently become a viable alternate landing place for primary data, and it’s actually a better one for long-term retention.

Enterprise archive systems from companies like EMC, Nexsan and Permabit, for example, provide a Tier 1-class destination for data that’s not changing or is not accessed often. They’re built with a high-availability and redundancy architecture that users now expect, but also add data reliability and continuous verification features needed for long-term viability. There’s also WORM and encryption available to meet security requirements.

As a VAR, your job is to solve problems for your customers. True solutions to these problems are seldom “point solutions.” Let’s face it, if this stuff was easy, your customers wouldn’t need you. Implementing an enterprise archive will provide real relief for your customers’ backup issues, relief they can’t get with deduplication. And, it will set them up to enjoy other real benefits (through their VAR) as they get more of their enterprise’s data under control for the long term.

After all, as a VAR, solving your customers’ problems is the way to the promised land — becoming their “trusted storage advisor.” Attacking your customers’ issues leads to that messy process of problem definition, proof of concept, system design, product evaluation, implementation, testing — you know, integration and professional services engagements. It’s the “value” in “value-added reseller.” 

Back to dedupe …

So, dedupe does generate POs, but does it really solve any storage problems for the customer? Well, it doesn’t reduce primary storage requirements. And, as mentioned in recent articles on Storage Switzerland, it doesn’t reduce network requirements or administration requirements. It also doesn’t usually reduce backup windows (for target-side dedupe, anyway).

What dedupe provided when it first appeared several years ago was a much simpler solution for disk backup than virtual tape libraries (VTLs). It didn’t really solve the fundamental backup problem then, and it still doesn’t now. 

Dedupe does make backup more efficient by reducing the amount of redundant data that has to be physically written in a backup. This enables you to postpone the purchase of additional backup-to-disk hardware. (I know dedupe appliances can also provide a relatively simple remote DR solution, but let’s pretend we’re only interested in local backup.) But it doesn’t really reduce the number of files in primary storage that need to be handled by the backup system, by its associated networks and servers or by the backup administrator.

For most organizations, the way to do that is with an archive. When files are moved off to an archive, they’re out of the backup stream altogether and the amount of data handled, processed and backed up is reduced. This also reduces per-terabyte storage acquisition costs and all the other costs associated with a larger primary data set. This is the kind of real problem solving that will keep your customers for the long term — and generates orders along the way.

Now, we’re not talking about your grandfather’s HSM here, but enterprise archive. More on this next time.

Eric Slack is a senior analyst for Storage Switzerland. He can be reached at eslack@storage-switzerland.com.