> HP comes clean in latest wave of storage benchmarks

> <a href="http://www.theregister.co.uk/2002/12/11/hp_comes_clean_in_latest/" title="http://www.theregister.co.uk/2002/12/11/hp_comes_clean_in_latest/" target="_blank">http://www.theregister.co.uk/2002/12/11/…</a>

> while its an old article. The point about altering for a
> benchmark versus “real world applicability” illustrate the
> problems with SPC.

What “problems”? Name one (please provide details, not
hand-waving).

So 6 years ago HP gets caught doing a benchmark “special” and
I guess your point is the benchmark can’t be legit because HP
gets caught? That’s rather lame as Sun pulled a fast one with
SPEC and cooked the “art” benchmark, SPEC CPU had to toss that
like they tossed “matrix” years before.

The author of that article goes on to point out that LSI was
guilty of the same stunt:

“LSI switched off write cache mirroring when it tested
its FastT arrays earlier this year.”

IBM (FastT) uses write cache on their more recent FastT tests.

Perhaps HP was wise enough to realize that their stunt
certainly didn’t impress anybody and came clean.

I’m sure more than a few get hair-brained ideas about gaming
a benchmark.

Either that portion of the benchmark is tossed (art, matrix in SPEC CPU),
a rule is changed to prevent abuse, the offender is shamed into discontinuing
the practice or the benchmark is replaced altogether (i.e. TPC-E set to replace the
much too simplistic TPC-C for OLTP testing).

Benchmarks are important and will need constant revisioning as systems
get faster and engineers game them.

HP comes clean in latest wave of storage benchmarks
 <a href="http://www.theregister.co.uk/2002/12/11/hp_comes_clean_in_latest/" title="http://www.theregister.co.uk/2002/12/11/hp_comes_clean_in_latest/" target="_blank">http://www.theregister.co.uk/2002/12/11/…</a>

while its an old article. The point about altering for a benchmark versus “real world applicability” illustrate the problems with SPC.

“exactly, so why would EMC play along with that? it’s an unwinnable situation for EMC, as you yourself point out. so how can we call that fair?”

That NetApp posted EMC numbers?
It isn’t a matter of fairness but participation, getting
numbers out there for all to see.
The fact that NetApp posts EMC numbers isn’t a unique idea, you know! Maybe they borrowed it from what goes on occasionally
at SPEC benches.

SPEC CPU benchmarks are well
regarded and help many decide when making a purchasing decision.
Here is an example of where Intel does AMD a favor and benchmarks
their CPU:
 <a href="http://www.spec.org/cpu2006/results/res2007q3/cpu2006-20070723-01527.html" title="http://www.spec.org/cpu2006/results/res2007q3/cpu2006-20070723-01527.html" target="_blank">http://www.spec.org/cpu2006/results/res2…</a>

Why would Intel do that? To show they have better performing kit, so folks can make reasonable assumptions about purchasing
based on performance. Maybe AMD turned around and posted better numbers (I haven’t looked, but I doubt it… they do use Intel
compilers and I’m sure Intel tuned it up for their AMD run
So is SPC a legitimate test? (That’s all that’s left to debate in my opinion). The fact that there are a number of major players participating, the tests are well documented lend credence to the tests.

i think i understand what the tests are. the issue isn’t whether the nuts and bolts of the tests themselves are good in concept, the issue here is the context. sometimes just the appearance of a conflict of interest is enough to invalidate some statements, or at least open them to scrutiny.

i also find something a little wrong with basically saying, “if people who won’t join our club want to refute the negative things our club says about them, they can just join us already.” that seems pretty coercive to me, and not something that adds to SPC’s credibility, either.

//But even if EMC does, NetApp will crow they forced it and are confident their numbers (FS) will be higher anyhow.//

exactly, so why would EMC play along with that? it’s an unwinnable situation for EMC, as you yourself point out. so how can we call that fair? with all the reasons EMC has not to play along with this, it’s clearly not a straightforward or automatically ’self-healing’ process, and i don’t think EMC’s silence on the issue can necessarily be interpreted as acquiescence.

Not necessarily. But if that is all there is and it satisfies
an RFP process - so be it. One would think that EMC has the incentive to do better SPC numbers, but I doubt they do it. But
it does seem rather strange that you see a lot of larger (volume,
revenue) vendors posting numbers (IBM,HP,NetApp).

“this is something that would fly in most environments? i agree that performance numbers are important, but in this case doesn’t the sourcing make the number a little suspect? if not, why bother to include SPC numbers at all? why not just rely on competitive white papers?”

In my earlier post, I suggested you read what SPC-1 and 2 test.
Educating yourself in what the tests are might help you to
validate them as good tests.

“think about the slippery slope aspects of this, too. turnabout’s fair play, right, so HDS could go and take a NetApp filer and run SPC benchmarks against the stuff it OEMs from BlueArc, and then have those numbers be submitted to SPC as fact?”

Absolutely. And then BlueArc could do the right thing, run SPC
and post higher numbers. HDS would then be forced to pull their
numbers and look quite asinine in the process. That’s why NetApp
is taking somewhat of a risk here (minor) in that they risk EMC
posting their own numbers. But even if EMC does, NetApp will
crow they forced it and are confident their numbers (FS) will be
higher anyhow.

“if competitors are allowed to supply what are supposed to be objective measurements, the SPC stats that go into RFPs and RFQs, as you mention, become less meaningful. that doesn’t bother you?”

Heck no. It is a self-healing process and the major reason
you don’t seeing this happening more often (it is a first) is:

1) It isn’t cheap (purchase the equipment, dedicate the engineers, etc.)
2) Most of the major players are doing it, so it isn’t exactly a target rich environment for gaming a competitor.
3) The vendors themselves knowing their own equipment would run
their own SPC tests - post higher numbers and make the competition look silly in posting lower numbers.

and as mentioned above, the NetApp test run
sort of pressures EMC to run the benches
and show how poorly NetApp did with EMC kit.