The DX6000G SCN is an appliance based on the Dell PowerEdge R410 server that connects to its DX6000 object storage nodes. Dell director of DX product marketing Brandon Canaday said the compression appliance can reduce data by 90%, depending on file types. Although Ocarina technology can dedupe or compress files, the object storage appliance will only use compression. It has two modes — Fast Compression mode is optimized for performance and Best Compression mode is optimized for capacity reduction. Customers can choose one or both modes.

Canady said customers can set policies to use fast compression when data is first brought onto the storage system and then switch to the best compression after a pre-configured time period. The appliance uses different compression algorithms depending on file type.

“It’s like applying tiered intelligent compression,” Canady said. “Because we maintain metadata with the file inside of the storage device, we can employ algorithmic policies as part of the lifecycle management of content.”

List price for the DX6000G SCN will begin at about $25,000, depending on the amount of data ingested. The appliance will become generally available next week.

Dell plans to incorporate Ocarina’s compression and deduplication across its storage systems, with more reduction products expected early next year. Canady said the performance and compression modes will likely show up in all of the data reduction appliances.

“Each implementation is likely to be slightly different, but we see value in having a performance approach and a capacity approach,” he said.

But Frank Slootman, president of EMC’s data backup and recovery division and Data Domain’s CEO until EMC acquired the company last year, says there is no dedupe in Nimble’s storage. Slootman saw my story on SearchStorage about Nimble, and sent an email claiming “there is no dedupe in

Nimble whatsoever. Read their white paper, or just ask them. We did. They do have local compression.”

I did ask Nimble CEO Varun Mehta when I spoke to him before their launch. He said his storage systems use inline compression for primary data and dedupe for backups. And according to Nimble’s press release on its product launch (emphasis added):

The CS-Series is based on the company’s patent-pending architecture, Cache Accelerated Sequential Layout (CASL™), which enables fast inline data compression, intelligent data optimization leveraging flash memory and high-capacity disk, instant deduped backups, and WAN efficient replication – all in a single device. CASL allows organizations to reduce their capital expenditures for storage and backup by at least 60 percent, while eliminating the need for separate, disk-based backup.

And a data sheet on the Nimble web site states:

Nimble slashes IT costs by converging compressed primary storage, deduped backup storage, and disaster recovery into one solution.

Slootman is correct about the whitepaper, though. A paper called “A New Approach to Storage and Backup” on the Nimble site does not say it uses deduplication. It claims “Nimble Storage CASL provides in-line compression on all data” and in a section on its backup technology says “CASL enables instant, application-consistent backups on the same array with very efficient (up to 20x) backup capacity optimization.”

Capacity optimization could be dedupe or compression. But nowhere in the 15-page whitepaper does Nimble claim to dedupe backup data.

While Nimble execs said in press interviews that they dedupe, they had a different message at a blogger TechField Day in Seattle where the startup officially launched Thursday. Nimble presenters did not mention deduplication at the blogger event.

I asked Nimble for clarification about its mixed marketing, and its VP of marketing Dan Leary replied via email:

“Sorry if there was any confusion regarding deduplication. Nimble does not deduplicate in the Data Domain sense, where all duplicate blocks are eliminated using a content-based signature. Our snapshot-based block sharing eliminates duplicate blocks across backups like deduplication systems. Nimble compresses, but does not deduplicate, within a primary storage volume. However, we offer better space savings compared with any secondary storage. Secondary storage systems require a baseline copy of the original data to get started. Because converged storage doesn’t require a baseline full backup, Nimble provides even better capacity optimization than secondary storage. Look for an upcoming blog from our CTO who will cover this topic in more detail.”

If Nimble can shrink data enough to make backups and replication for DR more efficient without taking much of a performance while compressing, it may not make much of a difference how it’s doing it. Nimble beta tester Dave Conde, IT director of eMeter, says he’s found performance outstanding and he’s getting a reduction in data although he hasn’t measured the actual rate.

But if Nimble is deduping, EMC execs probably want to know just how close the startup’s dedupe technology is to the dedupe it paid $2.1 billion for when it acquired Data Domain.

In a follow-up email, Slootman attributed Nimble’s mixed message to “a disconnect with marketing. They probably mean like NetApp that their snapshots use block differentials. They should not be using the term [deduplication] so indiscriminately.”

While those numbers may look staggering on a page, they probably don’t shock anybody charged with managing data storage. The real shocking – and frightening – number is that IDC says the amount of IT staff to manage all this data will only grow by a factor of 1.4 by 2020. If IDC is correct, than the dreaded “do more with less” mantra will become a long-term way of life.

So how will this all change the way we manage data? Chuck Hollis, global marketing CTO of EMC – which sponsored the IDC study – says the data growth will push a lot more of it to the cloud this year. Hollis says the IT staffs at large enterprises that he talks to are ready to set up private clouds to manage data.

“For tech guys, this is the year of putting your cloud strategy together,” Hollis says. “We’re way beyond the ‘What is the cloud?’ discussion, and it’s a very mature discussion with the IT guys I talk to.

“The larger enterprises say, ‘We’re big, we can do this ourselves. We can build a private cloud behind the firewall and get comfortable with it.’ They’re saying, ‘We pay the same price for this stuff – the processors, server, storage – there’s no reason I can’t do what Amazon does.’”

Hollis says as long as organizations feel they can control their data in the cloud, they’re willing to move it there.

“The cloud works when enterprise guys can be in control,” he said. “Ask them to give up control, and it’s not that attractive a proposition for them. You can’t outsource responsibility and accountability. In financial services, a trillion dollars a day floats around the global economy over the cloud. Most days we’re OK with that. Clouds, schmouds, it doesn’t matter as long as enterprise guys feel they’re in control.”

Other emerging methods of managing growth aren’t quite as mature, Hollis says. That includes data deduplication for primary data. While EMC is now the leader in backup dedupe, Hollis says the success of primary deduplication “has a lot to do with processors being fast enough to do it without impacting performance. If you have a SAP application with 10,000 demanding users, maybe it [deduplication]’s a false savings. The concern is, at what cost? The technology gets better year over year, but some are of the opinion this is just a temporary fix, you’re just buying yourself some time. A lot of information is not compressible, like JPEGs. You can’t compress something that’s already compressed.”

Flash solid state storage is another area where EMC has been out front, but it’s another technology where the greatest benefits are still down the road. “If you take what processors have done in the last 10 years as far as density, price and performance, then start with flash in 2010 and forecast it out in 10 years, it could actually get cheaper than disk,” Hollis said. “That would be an interesting world.”

Prior to its acquisition by Exar, Hifn began sampling Express DR 250 and 255 cards to OEMs, but they hadn’t made their way into any announced third-party products. At this spring’s SNW, Hifn launched its own product based on the DR 255.

It was unclear why the chip boards, which perform processor-intensive data reduction and encryption in silicon, hadn’t caught on with OEMs. Maybe Hifn’s announcement today of its new DR 1600 series may tacitly answer that question with new features such as high availability and boosted performance.

The DR 1600 line consists of six new models offering different levels of performance and combinations of compression, encryption, and dedupe. The Express DR 1600, 1610 and 1620 perform LZS compression and encryption only, at speeds of up to 300 MBps, 900 MBps, and 1800 MBps, respectively. The Express DR 1605, 1615, and 1625 run at the same three levels of throughput, but offer compression, encryption and hardware-based hashing for data deduplication (hash comparisons must still be performed by an OEM in software).

Hifn has also developed new software to go with the cards for this release, which includes a new API to standardize and ease integration of the cards into storage products to make it quicker for OEMs to take them to market. The 1600 series includes new high availability software for failover between cards, or to “pass through” traffic. That means if one card fails, the other can still perform compression, encryption, and dedupe in software.

According to Zack Mihalis, director of product marketing for Hifn, the new cards are sampling to OEMs and will become generally available at the end of July. Mihalis claimed that several large OEMs are considering the cards, potentially for primary storage dedupe. EMC, NetApp and Quantum are traditionally among Hifn’s OEMs, but Mihalis declined to disclose if any of them are sampling the DR 1600 cards.

Still, some industry analysts see this as the first step toward primary storage data reduction products becoming as ubiquitous as those for backup workloads. “Hifn has some very major OEMs as clients,” said IDC analyst Benjamin Woo. “This release is very timely – in this downturn we need to be more efficient with how we deal with data.”

However, Taneja Group analyst Jeff Boles pointed out that there’s still plenty of engineering work to be done to produce primary storage dedupe products, even with some of it already completed by Hifn. “Keep in mind that Hifn is hashing at 1,800 megabytes per second, but that’s not the speed of writing out to disk,” he said. “It’s still up to someone to make maximum use of this on disk, with caching, etc. Can you use this to service a random workload? That may be an engineering feat in itself.”

I wrote to one of the researchers involved and got a response through IBM’s PR spokesperson that:

• The IBM-developed algorithm is based on VMware API available in Virtual Infrastructure version 3, not the VMWare LinkedClone API. Specifically, the algorithm uses VMware Infrastructure SDK 2.5.0 as documented at https://secure.techtarget.com/exchweb/bin/redir.asp?URL=http://www.vmware.com/support/developer/vc-sdk/ and file system level access on ESX servers.
• We developed the algorithm based on the API that was publicly available and supported at the time that we began development efforts
• VMware can provide detail on the differences between the APIs in Virtual Infrastructure version 3 and VMware LinkedClone API

So far no response from VMware.

Regardless of what API was or was not used, what I am trying to get at is the functional difference between these two products, if any. If there is one, it’s important for users to know about. If there isn’t one, it speaks to the growing convergence between VMware’s virtual infrastructure and storage vendors’ value-add software.

the bottom line right now seems to be that IBM’s product is for existing IBM customers, since it requires professional services through IGS. There are some shops that need the IBM label before they buy, and so VSO could at least be a fit for them.

Appreciate weigh-ins from IBM, VDI, and / or VMware experts.

The next day, at a keynote, VMware officials demonstrated a new concept they’re rolling out in the next version of VMware Infrastructure called LinkedClones. LinkedClones create a “golden image” of virtual desktop files, as well as incremental changes;. The golden images, VMware demonstrated, can be updated with patches that automatically proliferate to all virtual machines based on the image to simplify rollouts and updates.  

From the briefing I’d had with IBM the day before and this keynote demonstration, these products seemed similar. Since the VMware demo last Wednesday, I’ve been trying to assess what might be different about them. VMware officials I spoke with Wednesday and Thursday said they didn’t know enough about IBM’s product to comment on its differentiation and IBM spokespeople were unavailable.  IBM’s press release about VSO had stated that it was “based on an algorithm developed by IBM Research,” but VMware said LinkedClones  wasn’t based on anything from IBM.

Today I spoke with VMware director of enterprise desktop Jerry Chen, who told IBM’s VSO is based on the LinkedClones API. I asked about the algorithm developed by IBM. “There are things our partners can do to further optimize LinkedClone,” Chen said. “For example, there are different settings for the number of LinkedClones each master virtual machine can copy.”

Chen said he wasn’t familiar enough with the IBM product to say what VSO adds on top of LinkedClones.  Meanwhile, IBM has been coy on this one. Since last week, I’ve put in numerous requests for comment by phone and email, including a fresh round of requests today after speaking with Chen. So far, no comments have been forthcoming.

So it was puzzling to see the announcement of the Ocarina Compression Prize, a $1 million fund that will be doled out in $10,000 increments for each submission that advances the current best scoring compressor by at least 3%. Isn’t the idea supposed to be that Ocarina has the most compression expertise in the market?

“A lot of our compression work is already based on prior art,” CEO Murli Thirumale told me. The idea, he explained, is to make this contest a “category builder,” raising interest in the subject of primary storage compression. “A lot of compression work is focused around transmission of files, rather than reducing them for storage. We want to encourage the best minds in compression to address it in that context.”

So I guess it doesn’t matter how many cool algorithms you can bring to the table if there isn’t really a market yet. “As there’s more widespread adoption [for products], clearly [vendors] with a leadership stance will benefit more,” Thirumale said.

“Good compression has a history of coming from independent researchers, open source or anywhere that can foster easy standardization and non-proprietary code,” Taneja Group analyst Jeff Boles says. “So this seems like a pretty good approach to me. Interesting stunt to boot.”  

The initial prize fund will include awards for three categories: JPEG 2000 recompression, h.264 video recompression and an industry file mix for engineering CAD file types. Maybe Riverbed, Silver Peak and Autodesk will jump in on that last one.