Although not the first product to offer data reduction in primary storage, IBM raised the bar by doing compression inline (real-time) and without performance impact. Other solutions in the open systems storage area primarily compress data and sometimes dedupe it as a post-processing task after the data has been written.

Competition for storage business is intense, and inline compression of data for primary storage will be a major competitive area because of the economic value it brings customers. If the compression can effectively reduce the amount of data stored, the reduction amount serves as a multiplier to the amount of capacity that was purchased.

IBM claims a 5x capacity improvement, which gives customers five times as much capacity as they pay for. Even if IBM’s compression comes in at 2x, that would still be significant savings despite an additional license fee for the feature.

Doing compression with no performance impact means the compression is transparent to the application and server operating system. The customer gets increased capacity benefits without having to make an accommodation such as installing another driver or version of an application. The effective compression rate will vary with data types, but there has been a long history of compressing data and the types and compression rates are not a new science. Vendors usually publish an expected average and sometimes offer a guarantee associated with the purchase.

Compression of real-time data in the mainframe world goes back to the StorageTek Iceberg (later offered as the IBM Ramac Virtual Array) that compressed mainframe count-key-data in the 1990s. That system compressed data at the channel interface and then stored the compressed information on disk.

The use of the Log Structured File system and the intelligence in the embedded storage software allowed the system to manage the variable amount of compressed data (done on a per-track level), and removed the direct mapping to a physical location. That was an effective compression implementation and demonstrated the effect that compression multiplies the actual capacity.

One of the more significant aspects of compressing data at the interface level was the effect that had on the rest of the system resources. With data that was reduced by something like 5x or 6x, the other resources in the system benefited.

Some examples:

• The cache capacity was effectively multiplied by that same amount, allowing for more data to be resident in cache giving higher hit ratios on reads and greater opportunity for write coalescing.

• The interface to the device had the data transfer bandwidth effectively multiplied for much faster transfer of data from the disk drive buffers.

• The disk devices, while storing more data, also would transfer more data over a period of time to the disk buffers and the controller.

Similar benefits gained by the implementation in the StorageTek system can be achieved in new systems targeted for primary storage in open systems.

In the case of the StorageTek system, the compression was a hardware-intensive implementation on the channel interface card. With IBM’s Storewize V7000 and SVC, the implementation is done in software, capitalizing on the multi-core processors available in the storage systems. Faster processors with more cores in succeeding generations should provide additional improvement. Having compressed data in cache and compressed data transferred on the device level interface and from the device means performance gains there offset time spent in the compression algorithm.

There are other potential areas where transparent compression could be done. Compressing the data in the device such as in the controller for solid state technology is another option.

Customers will benefit from reduction of data actually stored and the inline compression of data that is transparent to operations. The benefits are in the economics and this will be a competitive area for vendors.

There will be a considerable number of claims regarding implementations until this becomes a standard capability across storage systems from a majority of vendors. You can expect a rush to bring competitive solutions to market.

(Randy Kerns is Senior Strategist at Evaluator Group, an IT analyst firm).

IBM ported the Random Access Compression Engine (RACE) technology acquired from Storwize in 2010 into its Storwize V7000 and SAN Volume Controller (SVC) virtual storage arrays. This is IBM’s first integration of the compression technology into SAN arrays.

Until now, IBM used the technology only in its Real-Time Compression Appliances, which were re-branded boxes that Storwize sold before the acquisition. Even the Storwize V7000 launched in late 2010 lacked compression, despite its name.

Now IBM is claiming it can compress active primary data with no performance impact on SVC and Storwize V7000 storage, and says it can reduce primary data accessed via block-based protocols by up to 80%.

It turns out that integrating data reduction into primary storage isn’t easy. Dell bought primary deduplication startup Ocarina around the same time that IBM picked up Storwize, and has yet to port primary dedupe onto its Compellent or EqualLogic SAN arrays. Dell did launch a backup appliance using Ocarina dedupe in January, and may have a primary data dedupe announcement next week at its Storage Forum.

Other IBM enhancements include support for Fibre Channel over Ethernet (FCoE) and non-disruptive volume moves between I/O groups for SVC and Storwize V7000, and four-way clustering for Storwize V7000.

IBM added thin provisioning and Enhanced FlashCopy (allows for more snapshots) for DS3500 and S3700 midrange arrays, and a new web-based UI for the IBM Tivoli Storage Productivity Center (TPC) suite. For tape management, it added IBM Tape System Library Manager (TSLM) software that helps manage multiple libraries, and an IBM Linear Tape File System (LTFS) Storage Manager for customers using LTO-5 tape libraries and IBM’s LTFS Library Edition.

IBM also said it plans to extend its Easy Tier automated tiering software to direct attached server-based solid-state drives (SSDs) so customers can migrate data between disk systems and servers.

Xiotech chief marketing officer Brian Reagan said the vendor had no timetable for embedding primary dedupe into its storage systems. “We’re looking at incorporating Albireo into our next-generation products,” he said, adding that Xiotech does not plan to use it in current shipping products. “It’s an opportunity to enhance products on our roadmap now.”

However, the press release issued by the companies said Xiotech will use dedupe with its Intelligent Storage Element (ISE) storage blades, suggesting that it will be in an ISE upgrade. Xiotech has been pushing its storage’s performance with virtual desktop infrastructures (VDI), and the release said “Xiotech’s leading cost per virtual desktop will be taken to the next level when combined with Albireo’s enormously powerful deduplication of virtual desktops, greatly reducing storage requirements.”

The release also noted that Albireo sits outside the data path, has no impact on data reads, and scales to petabytes.

Reagan said Permabit and Xiotech are headed in the same strategic direction with data reduction, and the move to incorporate reduction technology is inevitable.

“We’ve seen the rise of deduplication and compression in backup and archiving, and it’s a matter of time until it becomes a table stakes feature in primary storage,” he said. Reagan also said using Permabit’s deduplication leaves the door open for Xiotech to add compression as well, although the vendor has no concrete plans for that yet.

Xiotech is the second vendor to publicly announce plans to use Albireo, following NAS vendor BlueArc last month. Permabit CEO Tom Cook says more are coming.

“Until now, NetApp has had a strong advantage as the only vendor with [primary] deduplication,” Cook said. “We’re seeing a clear objective from all storage vendors to have these products in the market in 2011.”

NetApp has offered dedupe for primary data since 2007, but other vendors have been lining up this year to follow that path. Dell acquired primary reduction vendor Ocarina Networks and IBM bought compression vendor Storwize this year. EMC added block compression to its midrange Clariion storage systems, and Hewlett-Packard said it will extend its StoreOnce backup deduplication technology to primary data. Storage systems based on ZFS can also take advantage of the dedupe built into that file system.

BlueArc has used Ocarina Networks’ add-on primary dedupe product for archival data, but picked Albireo for primary data after Dell acquired Ocarina. BlueArc senior director of solutions marketing Ravi Chalaka said the vendor will license Permabit’s software and make it available for all of its NAS systems.

“We chose Permabit because we wanted zero impact on performance,” Chalaka said. “We saw Permabit could maintain high performance while reducing the amount of disk it writes to.”

Chalaka said BlueArc will ship primary dedupe sometime next year as a licensed feature with its NAS but no decision has been made on pricing. He said OEM partner Hitachi Data Systems has yet to commit to using Albireo on the BlueArc systems it re-brands.

“Clearly, primary deduplication is here to stay,” Chalaka said. “Around 70 percent to 80 percent of our customers expect to see it by the end of next year.”

Storage array vendors have been busy developing or buying primary data reduction technology. Besides the Dell-Ocarina deal, IBM acquired primary compression vendor Storwize, EMC launched block compression for primary data on its Clariion and Celerra storage systems and Hewlett-Packard said it would expand its StoreOnce backup dedupe to primary data. NetApp already ships deduplication for primary data on its FAS storage arrays.

Permabit CEO Tom Cook has claimed the dedupe provider will make many OEM announcements in the coming months for Albireo.

IBM expects the deal to close by the end of September. It did not disclose financial terms.
The IBM-Storwize acquisition comes less than two weeks after Dell bought Ocarina Networks, which had been seen as Storwize’s main competitor although the vendors use different methods to shrink data.

The Storwize STN-6000 appliance works with NAS systems, including the IBM N series (rebranded NetApp storage) and Scale Out Network Attached Storage (SONAS).

In a letter emailed to “friends of Storwize” today, Storwize CEO Ed Walsh said IBM will continue selling Storwize’s STN-6000 appliance while expanding the platform. Storwize has been working on adding block storage reduction to go with its traditional file compression, and that apparently will continue under IBM.

“Storwize will continue to sell and deploy its STN-6000 series of products and support CIFS and NFS protocols,” Walsh wrote. “Additionally, the Storwize product will continue to evolve to support additional storage systems and additional protocols. … Under IBM we will continue to deliver capacity optimization without compromise to you, across more storage platforms, and to additional new customers.“

IBM’s press release issued today said it found Storwize attractive because it compresses primary data – files, virtualization images, and databases – and lets customers store up to five times more.
IBM already has backup deduplication technology in its ProtectTier virtual tape library (VTL) software and Tivoli Storage Manager (TSM) application.

IBM claimed Storwize has more than 100 customers including Mobileye, Polycom Israel, Shopzilla, and Sumitomo Mitsui Construction.

For more on this story, check out SearchStorage.com.

Dell last month hinted at an OEM deal with Ocarina when a Dell storage executive was quoted in a press release Ocarina put out about its OEM product. Following the deal today, Dell product manager Brett Roscoe confirmed there was an OEM deal in the works but said Dell wanted more control over the dedupe technology.

Unlike Dell’s acquisition of EqualLogic in 2008, Dell isn’t getting a mature business with Ocarina. This was a pure technology buy, which highlights the importance storage vendors place on dedupe for primary data.

“We believe that deduplication is a key strategic pillar for storage going forward,” Roscoe said. “We started working with Ocarina some time ago, developing solutions around EqualLogic and other storage products. The more we worked with them, the more interesting they became.”

Dell did not disclose the price of the acquisition.

Although Roscoe wouldn’t discuss specific products, sources familiar with the Dell-Ocarina relationship say Dell was already working on integrating Ocarina’s dedupe in three products: EqualLogic iSCSI SANs, a scale-out NAS product it is developing from IP it picked up from Exanet this year, and a disk backup target. Dell currently OEMs dedupliation backup products from EMC Data Domain and software partners CommVault and Symantec.

Roscoe talked about dedupe for EqualLogic and for unstructured data, but wouldn’t get into using Ocarina for backup. “We’re going to look at all our opportunities,” he said. “There’s nothing specific around that now.”

The future of  Ocarina’s current shipping products — appliances aimed at reducing unstructured data — is unclear although Dell plans to sell and support the appliances until it can develop its own branded version and move the technology to other platforms.

Storage vendors are moving to incorporate technology in their storage systems to shrink primary data. NetApp has had dedupe for primary data for three years. Hewlett-Packard last month launched its StoreOnce deduplication for backup and primary data. IBM has been linked in a possible deal for data compression vendor Storwize, and EMC is planning on delievering compression for primary storage on its Clariion and Celerra platforms.

“We knew having IP in the deduplication space was going to be strategic for all Dell storage products going forward,” Roscoe said. “We believe we had to have the ability to build deudpe into our product set.”

Roscoe wouldn’t say how close Dell is to having dedupe in any storage products, but hinted that it’s not far off. “Let’s just say this isn’t a five-year project.”

Permabit and Ocarina each say they have one large OEM primary deduplication deal nailed down and are working with more storage vendors to secure others. Dell, Hewlett-Packard, Hitachi Data Systems, IBM, and LSI are all believed to be on the prowl for the technology and it’s a matter of whether they will forge OEM deals or acquire the technology outright. NetApp and EMC already have primary reduction capabilities.

Ocarina has been the subject of acquisition rumors, and Ocarina director of marketing Mike Davis says an IBM-Storwize deal would raise his company’s value.

“There’s demand for this technology, and we’ve had contact and serious conversations with all the OEMs out there,” Davis said. “We don’t know how serious IBM’s interest is [in Storwize], but Storwize only does a subset of what Ocarina does. So if they’re worth $140 million, Ocarina should be worth even more than that.”