» VIEW ALL POSTS Feb 1 2008 9:54AM GMT

NetApp pulls a fast one on EMC with SPC

Posted by: Beth Pariseau

Storage

I don’t refer to many things related to data storage as humorous, but I have to admit this is a hoot.

Everybody knows by now that EMC hasn’t submitted its products to the Storage Performance Council (SPC) for performance benchmarking, saying it’s a rigged system.

So some in the storage industry feared that hell had frozen over when they saw an SPC benchmark published for Clariion this week (scroll down, it’s in the table). Actually, two SPC-1 benchmarks have been published for the Clariion CX-3 model 40, one with and one without SnapView enabled.

One little twist, however: in the “test sponsor” column next to EMC’s products is the name “Network Appliance Inc.”

Now that. is. hilarious.

Shockingly, the NetApp-submitted benchmark numbers show the Clariion CX-3 40 with lower performance than that of NetApp’s FAS3040. Or, as a NetApp press release put it:

In both cases, the NetApp FAS3040 outshined the EMC CLARiiON CX3-40, delivering 30,985.90 SPC-1 IOPs versus 24,997.48 SPC-1 IOPs (baseline result) and a robust 29,958.60 SPC-1 IOPs versus just 8,997.17 SPC-1 IOPs (baseline result with snapshots enabled). These results further validate NetApp as the high-performance leader for real-world data center deployments featuring value-add data management and data protection functionality.

While I agree NetApp’s move is somewhat ridiculous, if there were a mom refereeing between these squabbling siblings of the storage market, NetApp could accurately say, “But he started it!”

In fact, not only did EMC start it, but it did this exact thing first. This bickering goes back to the hoary days of November 2006. NetApp released the 3000 series and published performance specs that showed its new array performing far better than EMC’s newest Clariions. Although performance testing is generally against its beliefs, EMC couldn’t let that stand, so did its own test on NetApp’s equipment. EMC’s internal tests showed that NetApp’s filers initially perform better than Clariion, but as NetApp systems fill up its WAFL file system causes fragmentation that slows everything down. So EMC conceded NetApp’s original results, but contended the initial results were reflective of how performance on the NetApp system would change over time. I waded into this whole mess back when it happened, if you want to read what analysts had to say, it’s all here.

“Many companies have access to other vendor’s equipment–competitive analysis is nothing new,” argued SPC administrator Walter Baker. “NetApp’s not the only EMC competitor to have run competitive analysis.”

“But they’re the only competitor whose analysis you’re endorsing,” I replied. Baker insisted it’s not an endorsement–that publication on SPC’s Web site among all its other specs merely serves as notification that NetApp’s results have been submitted for approval. He also pointed out that unlike, say, vendor-published white papers about another vendor’s product, there’s a redress process for EMC in this case.

There’s a 60-day review period before the result is officially accepted. Until then it’s submitted for review status. In that period, it can be challenged by any member company, or in this case, EMC, that the testing was not compliant with the SPC-1 spec or did not represent the performance the Clariion should have attained.

Baker said EMC has not challenged yet. “Absolutely not–and they have been notified, because I spoke with them myself,” he said. He added, “as the auditor I feel the result produced by NetApp is representative.” Pressed further, Baker said his basis for that conclusion was ”talking to people who are familiar with EMC equipment.

“I understand what you’re saying,” he admitted. “At first blush it does seem to be a conflict of interest–but it really doesn’t serve Netapp’s purpose if they were to understate or undermine the performance of the EMC equipment, because it would bring about an immediate response from EMC.”

EMC hasn’t yet responded to my e-mail about this, but something tells me they’ll have something to say before the review period is up. And what about this really serves NetApp’s purposes anyway? Have they done anything with this than cast aspersion on the very spec at the core of this latest volley against EMC? If there’s anything to be learned from this from my point of view, it’s to add an extra shake of salt when referring to SPC benchmarks.

And seriously, I would love to see the user considering a Clariion against an FAS3040 for whom this is the tipping point in one direction or another–I would love to see the user on the verge of signing on the dotted line for a Clariion suddenly saying, “But wait! NetApp’s performance testing shows this array doesn’t perform as well as the FAS3040!”

It’s kind of like when MacDonald’s tells you its fries taste better than Burger King’s, when Coke tells you more ~~compensated~~ blindfolded taste testers picked its soda over Pepsi’s in a ~~carefully controlled~~ totally off-the-cuff random taste test, when a Red Sox fan walks up to you with a T-shirt that says “YANKEES SUCK!” All that it really, reliably tells you is what one company thinks of its competitor. And we kind of don’t need a press release about that, especially not when it comes to NetApp and EMC.