The Virtualization Room

Jul 1 2008   9:01AM GMT

QLogic and Microsoft taken to task for “benchmarketing” by Chris Wolf

Alex Barrett Alex Barrett Profile: Alex Barrett

Anyone with five minutes of IT experience knows that vendors sometimes publish bogus “benchmarks” that portray their products in the best of all possible lights. Virtualization guru and Burton Group analyst Chris Wolf recently uncovered a particularly spectacular example of this, courtesy of QLogic and Microsoft.

In a release, QLogic Corp., a networking technology provider, said it tested virtual machines running on Windows Server 2008 Hyper-V and attached to a storage area network (SAN) via its 8 Gbps Fibre Channel (FC) host bus adapters, and saw near-native performance of 200,000 I/O operations per second (IOPS).

But, as Wolf discovered, what QLogic failed to mention was that it ran its tests against a very unusual SAN array: the Texas Memory RamSan 325 FC, which uses solid-state storage. Further, the benchmark used block sizes of just 512 bytes, compared with a more real-world block size of 8 K or 16 K.

This left Wolf feeling duped and betrayed:

If I was watching an Olympic event, this would be the moment where after thinking I witnessed an incredible athletic event, I learned that the athlete tested positive for steroids.

Wolf ran this benchmark by a colleague, who calculated that had the same benchmark been performed using “real disks” with latency of 7 milliseconds, it would have limited throughput to a much less impressive 9,142 IOPS. Hardly anything to write home about.

Thanks to Wolf for taking the time to look into this.

1  Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • Kirk Bradley
    I think the point of the RAM SAN was to show that with virtualization and Qlogic you will be able to issue I/O requests at a rate that's close to what you can do without virtualization (near native) and perhaps w/o Qlogic. To that end I'm glad they removed the storage bottleneck (spinning disks) from the test as it's not a storage system test it's an I/O issues/interrupts etc. test.
    0 pointsBadges:

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: