QLogic and Microsoft taken to task for “benchmarketing” by Chris Wolf

Anyone with five minutes of IT experience knows that vendors sometimes publish bogus “benchmarks” that portray their products in the best of all possible lights. Virtualization guru and Burton Group analyst Chris Wolf recently uncovered a particularly spectacular example of this, courtesy of QLogic and Microsoft.

In a release, QLogic Corp., a networking technology provider, said it tested virtual machines running on Windows Server 2008 Hyper-V and attached to a storage area network (SAN) via its 8 Gbps Fibre Channel (FC) host bus adapters, and saw near-native performance of 200,000 I/O operations per second (IOPS).

But, as Wolf discovered, what QLogic failed to mention was that it ran its tests against a very unusual SAN array: the Texas Memory RamSan 325 FC, which uses solid-state storage. Further, the benchmark used block sizes of just 512 bytes, compared with a more real-world block size of 8 K or 16 K.

This left Wolf feeling duped and betrayed:

If I was watching an Olympic event, this would be the moment where after thinking I witnessed an incredible athletic event, I learned that the athlete tested positive for steroids.

Wolf ran this benchmark by a colleague, who calculated that had the same benchmark been performed using “real disks” with latency of 7 milliseconds, it would have limited throughput to a much less impressive 9,142 IOPS. Hardly anything to write home about.

Thanks to Wolf for taking the time to look into this.