server consolidation

VMmark a server vendor leapfrog game

This week I wrote a follow up story on VMware Inc.’s virtualization performance benchmarking tool, VMmark, and found it is mainly used by vendors as a way to market their servers.

Server vendors run the VMmark test under a set of guidelines and submit results to VMware for posting. It is my suspicion that vendors play leapfrog with VMmark by looking at exsiting VMmark results and only submitting their performance results when theirs are as good or better.

For instance, IBM submitted a benchmark for its 16-core System x3850 M2
running VMware ESX v3.5, which trumped the other published as of March 2008. IBM then published a press release to brag about the results, but within a few months, Dell submitted results three PowerEdge systems sporting better virtual machine (VM) performance than IBM, and Hewlett Packard (HP) beat them all out with its ProLiant DL585 G5 server results published August 5.

HP also sent out an email to press this week boasting their top 32-core results, but didn’t mention one minor detail; they are the only vendor with results in the 32-core category so far. Sure, they are number one. They are the only one.

System Administrator Bob Plankers sums this game up nicely in his blog with a post called “Why VMmark Sucks.”  Here is what Plankers had to say:

“Having a standard benchmark to measure virtual machine performance is useful. Customers will swoon over hardware vendors’ published results. Virtualization companies will complain that the benchmark is unfair. Then they’ll all get silent, start rigging the tests, scrape and cheat and skew the numbers so that their machines look the greatest, their hypervisor is the fastest. Along the way it’ll stop being about sheer performance and become performance per dollar. Then CapEx vs. OpEx. Watt per tile. Heat per VM. Who knows, except everybody will be the best at something, according to their own marketing department.”

In addition, the benchmark is a real pain to set up and run, and the ‘free’ VMmark software requires other expensive software to work. According to VMware’s website,
VMmark requires  licenses for the following software packages;

• Microsoft Windows Server 2003 Release 2 Enterprise Edition (32-bit)—thre 32-bit copies per tile (two for virtual machines and one for that tile’s client system), and one 64-bit copy per tile (for the Java server virtual machine)
• Microsoft Exchange Server 2003 Enterprise Edition
• SPECjbb2005 Benchmark
• SPECweb2005 Benchmark

Plankers said he won’t be wasting any time or money running VMmark. “Instead, I’ll be in meetings explaining to folks why we are maxed out at 30 VMs per server when the vendor says they’ll run 50. Or why we chose VMware over Xen, when Xen claims 100 on the same hardware. I’ll have to remember the line from the FAQ that says “that VMmark is neither a capacity planning tool nor a sizing tool.”

Which begs the question: if it isn’t for use in sizing or capacity planning, exactly what is it good for?”

VMware says the benchmark is good for users who are making hardware purchasing decisions.

“The intention [of VMmark] is that customers can look at the results and make decisions based on what they see. It isn’t just about the fastest server; it’s about making system comparisons; between blades and rackmounts or a two-core or four-core system. Someone can see how much more performance they get from upgrading to four core processors, for instance,” said Jennifer Anderson, the senior director of research and development at VMware.

This makes sense, but as Plankers said, users should beware of benchmark manipulation by vendors and know that the results do not reflect the same workloads that users will run in their own data center environments.

Virtual machines per server: A viable metric for hardware selection?

When server vendors introduce new blade servers these days, they often mention virtualization in the same breath, often touting the number of virtual machines (VMs) their hardware can support. But those numbers are hardly the result of scientific method.

For instance, San Diego, Calif.-based Verari Systems recently announced that its VMware ESX 3.5-certified VB1257 for BladeRack 2 XL supports up to twice as many VMs as competitive offerings (16). After speaking with Verari, I asked the competition — Sun Microsystems, Hewlett-Packard and Dell — how many VMs their blades can hypothetically support, and was given some big numbers.

But are these server vendors asking the right question? According to Anne Skamarock, a research director at Focus Consulting, the answer is no. Although vendors boast about the number of VMs their hardware supports, “it really is a silly way to look at it,” she said.

“The number of VMs supported depends on the workload. For CPU-intensive workloads, memory will also be a significant factor in performance,” Skamarock said. ”I have spoken with customers who are running 30 VMs per 8-core system and expect to increase that to 50 VMs per system.”

Skamarock said Virtual Desktop Infrastructure adds another twist. “The rule of thumb is six to eight virtual desktops per core, but again, memory will be a big issue here depending on the OS.”

According to preliminary data from SearchDataCenter.com’s 2008 Purchasing Intentions Survey, 61% of the respondents run less than 10 VMs per server, though 33% run 10 to 25, and a mere 5% run more than 25 VMs on a server.

Vendors make big VM support claims

According to VMware Inc.’s website, server consolidation ratios commonly exceed 10 virtual machines per physical processor; so presumably, a blade server with two CPUs, like Verari’s VB1257, should be able to support at least 20 VMs.

Within HP’s ProLiant blade server line, the ProLiant BL460c/465c and BL680c/BL685c would be a good choice for a virtual server platform, primarily because they offer a large memory footprint, which means more than 16 VMs per blade in both cases, plus more network expansion and storage performance, HP spokesman Eric Krueger said.

“Keep in mind of course the number of VMs always vary – the number could be higher or lower depending on the needs the application/VM — but based on the rule of thumb … the BL460c can support up to 16 VMs and the BL680c up to 32,” Krueger said.

Sun Microsystems Inc. claims its Sun Blade servers pack two and three times that many VMs. The Sun Blade X6250, which has up to eight cores with Intel Xeon processors, 64 GB RAM, 110 Gbps I/O and 800GB of internal storage, supports 36 VMs; the Sun Blade X6450, with two or four dual-core or quad-core Intel Xeon processors and up to 96 GB of memory, can support up to 42 VMs and the Sun Blade X8450 with 16 cores per module and 128 GB Memory, supports up to 48 VMs, according to Sun.

Dell was hesitant to name a number of VMs that its PowerEdge blade servers can support, because the number is dependent on a number of factors, like workload, memory, I/O. A spokesperson did say that “Dell has blades that support up to 66 loaded VMs. This is based on VMware’s VMmark benchmark test,” a spokesperson said. “This is an area where we are doing quite a bit of work, so stay tuned.”

So I’m wondering: Are VM support numbers a consideration when buying server hardware, or is it too subjective? Let us know what you think.

Next Generation Data Center/LinuxWorld 2008: Reporter’s Notebook

I expected this year’s joint LinuxWorld/Next Generation Data Center conference at the Moscone Center in San Francisco Aug. 4-7 to be full of technology vendors, high-level technical sessions, product news and interesting characters.

As you can see (at right), my expectations were exceeded.

This year’s conference is packed, with three to four keynotes each day, a large array of tech vendors and numerous technical sessions, covering storage, security, networking, applications, facility infrastructure, and virtualization.

In the five sessions I attended today, which touched on all of the above, virtualization was a predominant topic of conversation in each.

For instance, Rajiv Ramaswami, the vice president and general manager of Cisco Systems Inc., (at left), gave a keynote this afternoon, “Data Center 3.0: How the Network Is Transforming the Data Center,” and explained that, eventually, everything in the data center will be virtualized, including networks.

In another session I attended on creating an efficient, profitable data center, hosted by the Rocky Mountain Institute, virtualization was listed again and again as a key way to reduce data center power consumption.

Cloud computing (aka distributed computing), which goes hand in hand with virtualization, was also a popular topic in the sessions I attended, including the kickoff keynote, “Stateless Computing: Scaling at Zero Marginal Cost above Capex,” by Jeffrey Birnbaum, the managing director and chief technology architect for Merrill Lynch.

In between sessions, I took a tour of Rackable Systems’ 40-foot containerized data center (at right), Ice Cube, which was one of the most popular attractions on the large show floor.

Ice Cube is packed with up to 22,400 Intel Xeon processing cores in Rackable’s own half-depth servers, has a 36-inch central isle to access servers and uses direct current, or DC, power and self-contained uninterruptible power supply, or UPS, technology.

Ice Cube can be configured with IBM BladeCenter servers as well.

Tomorrow I’ll check out a keynote session by Oracle CIO and Senior Vice President Mark Sunday on delivering business value with next-generation data centers and more sessions on green strategies for data centers, cloud computing and virtualization.