The Virtualization Room

October 10, 2007  2:39 PM

Linux Kernel 2.6.23 = Win for Xen and KVM and a loss for VMware

Kutz Profile: Akutz

How many of you believe that Apple’s 1.1.1 iPhone update accidentally bricked modded iPhones? Personally, I try to air on the side of optimism, but there are certainly many people out there that think Apple intentionally went after those individuals who took it upon themselves to jailbreak and unlock their shiny gadget-of-the-moment.

Here we are again, not even a month later, and the new Linux Kernel, 2.6.23 was released on 2007/10/09. The latest product of the world’s greatest hackers includes a bevy of new features, including increased support for Xen and KVM and two open source virtualization solutions. Users of those products are probably very happy today, eagerly awaiting the adoption of the new kernel by their favorite distribution in order to take advantage of the increased guest support that comes with it.

VMware Server users on the other hand are getting the proverbial shaft. Kernel 2.6.23 has one MAJOR change and one minor change that completely break VMware Server.

For purposes of dramatic effect, I will detail the minor change first. VMware Server inserts a driver module into the kernel called vmnet. It provides magical networking gnomes that help shuffle bits in and out of VMs to the wide world of webs. In one of its source files, driver.c on line 522, the vmnet driver makes a function call to “unregister_chrdev”, a function defined in the Kernel source file “fs/char_dev.c”. Prior to Kernel 2.6.23 the function “unregister_chrdev” returned an integer value; a return value that the vmnet driver keys on in order to determine whether or not to issue a warning. Kernel 2.6.23 changes the function signature of “unregister_chrdev” to return void instead of and integer. This really hoses the vmnet module source file since it expects an integer value to be returned, and thus the vmnet module will not compile when the “” script is run. Luckily there is an easy fix. It seems that the function “unregister_chrdev” has actually returned a value of “0” despite what transpires in the function as far back as 2.6.20, a Kernel that VMware Server runs fine on. Thus the easy fix is to just edit the vmnet driver.c source file and re-run the VMware Server configuration script.

That is the minor problem that the new Kernel creates.

The major problem is a bit more cumbersome, since the fix involves either redacting a change that Linus (Torvalds) has approved for the 2.6.23 Kernel or lying and declaring that the vmmon module is GPL licensed.

But I’m getting ahead of myself. Let’s start at the beginning. A memory structure called mm_struct is defined in a Linux Kernel header file “linux/sched.h”. Prior to 2.6.23 this structure included a field called “dumpable” that would determine how memory was dumped, securely or not. Kernel 2.6.23 removes this field and lets two functions defined in “fs/exec.c” take its place: set_dumpable and get_dumpable. VMware Server uses the dumpable property in its memory management module vmmon: in the file driver.c to be exact. Since the dumpable property is no longer in the 2.6.23 kernel the vmmon module will not compile.

One might think that a quick fix would be to simply edit the vmmon source file to use the new set_dumpable function. In fact, this action will result in a vmmon module that compiles; however, it will not insert into the Kernel, and an error will occur that says the module contains an unknown symbol. A quick check of dmesg reveals that the unknown symbol is indeed set_dumpable. ‘What, what, whattttt,” you say. But the set_dumpable symbol IS in the kernel. That is verifiable by peeking in /proc/kallsyms.

Heh, heh. Hold on to your seats. This is where it gets fun.

The function set_dumpable is exported in 2.6.23 with the new EXPORT_SYMBOL_GPL, meaning that only modules that are GPL licensed can use it. More can be read about this decision on the Kernel mailing list.

VMware Server’s vmmon module cannot use set_dumpable because it is not GPL licensed. There are two solutions to this problem. The first solution is to edit the Kernel source file “fs/exec.c” so that “set_dumpable” is exported with EXPORT_SYMBOL instead of EXPORT_SYMBOL_GPL and compile a custom Kernel. Then, the vmmon module source file “driver.c” still needs to be edited such that the “dumpable” property is no longer used in favor of “set_dumpable”. The second solution is to edit the vmmon module source file the same way as in the first solution, but also using the macro “MODULE_LICENSE” to indicate that the vmmon module is licensed under the GPL.

Neither solution is nice, because the first one involves maintaining a custom Kernel and custom vmmon module, and the second solution involves changing the vmmon module license without permission. A long-term solution is needed where either the Kernel developers change set_dumpable to be exported out from underneath the aegis of the GPL, or VMware could license the vmmon module under the GPL or create some type of GPL-compatible shim module that in turn calls the proprietary code in vmmon.

Perhaps most interesting of all is the timing. The same Kernel that provides extended support for Xen and KVM also breaks VMware Server. Coincidence? Like I said, I try to err on the side of optimism. How about you?

October 9, 2007  3:27 PM

Microsoft support for VMware: wishful thinking?

Alex Barrett Alex Barrett Profile: Alex Barrett

Last week, I wrote a story about Sun’s upcoming xVM virtualization offerings, and in that story, I quoted Sun director of Solaris marketing Dan Roberts as saying that Microsoft does not officially support Windows or its applications running as guests under VMware. “There is no official support,” he unequivocally said.

Not so, countered VMware, pointing to a couple of Microsoft Knowledge Base articles on Microsoft’s support site. For example, one such article says “For Microsoft customers who have a Premier-level support agreement, Microsoft will use commercially reasonable efforts to investigate potential issues with Microsoft software running in conjunction with non-Microsoft hardware virtualization software,” a VMware spokesperson pointed out.

However, the KB article itself also explicity states that customers without Premier level support enjoy no such assurances. For those shops, “Microsoft will require the issue to be reproduced independently from the non-Microsoft hardware virtualization software. Where the issue is confirmed to be unrelated to the non-Microsoft hardware virtualization software, Microsoft will support its software in a manner that is consistent with support provided when that software is not running in conjunction with non-Microsoft hardware virtualization software.”

In other words, Microsoft supports its software running on third-party virtualization software, but not really. More to the point, whereas it may be making “commercially reasonable efforts” at support today, will it always? A “commercially reasonable effort” is a very subjective notion indeed — one which may change dramatically the closer we get to a shipping version of Microsoft’s own virtualization platform, Viridian. Or is that too cynical of me?

Personally, I’m curious to hear how support for Microsoft OSes and apps is playing out in your VMware shop. How has it changed over the months and years? Do you worry that Microsoft will use support as the stick to get people to switch to Viridian? If and when they do, what would it take for you to go along? Feel free to leave a comment, or if you’d rather respond in private, send me an email.

October 8, 2007  1:34 PM

Scaling Up or Scaling Out, Revisited

Joseph Foran Profile: Joe Foran

Some time back, before I was invited on as a blogger for SSV, I was interviewed by the always-fun-to-work-with Adam Trujillo about Virtualization in the Data Center, and, like all good writers, Adam left the best question for last:

“What about hardware decisions — should data center managers be considering scale-up instead of scale-out?”

My response was:

“I personally prefer a scaled-up approach because there is a reduction in ongoing costs, such as power, space, cooling, and physical maintenance. Also, the complexity factor is reduced when there is less hardware to manage. An exception to that would be data centers without existing centralized storage — the initial acquisition becomes more expensive in scale-up operations if a SAN infrastructure is not already in place.”

I’m guilty of being one of those people that says “Durnit, why didn’t I say this or that?” or “Dangit, why didn’t I quantify that a little more?” even well after the fact, making me perhaps my own worst critic. In this case, I really felt I left some stuff unsaid. One item that irks me about that answer is that I should have made more mention of blades. I hate blades in their current incarnation. I think they’re the worst idea in IT – they’re hot, cramped, delicate, with slower components and limited expansion ports – if you name something about a blade, I can find a reason to hate it. That said, I shouldn’t have left them out of my line of thought – a good IT Manager needs to consider uncomfortable things, difficult things, even distasteful things, when looking at something impactful. Or so says the wisdom of Frank Hayes, to whose articles I often find myself nodding to the affirmative while reading. So, here goes.

Blades are hot – they have limited cooling options built-in. That’s often a “value-add” (choke) of specialized rack systems and chassis systems provided by third-party vendors. Here’s a few links to illustrate the point:

A rack of big-honkin’ boxes will make you feel toasty on the parts next to their fans. A rack of blades will cook you medium-well given enough time. To prevent the data equivalent of multiple mini-supernovas you need to install the correct cooling – the correct tonnage of AC, hot and cold rack aisles, proper ventilation, air temperature monitors, system heat monitors, etc. In many data centers, the cost of new construction (or re-construction) may very well exceed even long-term cost savings from server consolidation, and even if you can afford the construction and still come out with positive ROI, that cooling comes at a monthly utility cost – you must increase your power consumption to keep things cool.

That said, this is where virtualization has been proven out over the last decade as a way decrease the number of servers and offload them to blades. That may mean that you can remove enough servers to use your existing heat management systems in a more focussed way and not have to break the bank. Even if it’s a five-to-one ratio of servers removed to virtualization-equipped blades added, you’re coming out ahead. Add in centralized storage systems to connect to the blades and the scales may well tip back in favor of Mr. Heat Miser again, but probably not. Getting a ten-to-one ratio means blades are a winner. This is assuming a large server consolidation via virtualization project. If it’s not a big percentage of your boxes being affected, you’ll be back in the hot seat, quite literally.

Ever need five or more NICs for a virtualization host? I have. If I had blades, I’d be using three blades to get that done, assuming dual nics, and five or more on single-nic blades. That means more blades, more virtualization software licenses I don’t need, more hardware to fail, and more physical boxes when what I want to do is REDUCE the number of physical boxes. Right now server blades are still too young – many vendor’s products have all the components are included on the blade, and not modular enough. PC blade systems have it a little better – some limited peripheral connectivity at the user-site (see this link for one manufacturer’s solution), but still, it’s an entire box in a chassis with all the difficulties of expanding that micro-sized PCs and laptops have.

So, I think it’s safe to say that I still hate traditional blades. But I think they’ll be the saviour of the data center soon, and then I will love them. Why? Because here’s my ideal blade system: a truly modular system that will change everything about blades. The best part, it’s available now from several of the larger vendors. The changes are part of a new design “paradigm” (please note my bias against that word) – the end-result is a blade system where the blades can be NICs or other devices, as needed and plugged into the chassis, connected in either a physical layer with ye olde jumper or a software layer (in the chassis management software, perhaps). Lets say I get a blade and I need to put ESX on it, but I need six NICs because of guest system network i/o requirements… ok, I get another blade with a quad-NIC on it, plug it into the chassis, and configure it – voila, a single computer with five or six NICs in two blade slots, using one license. Or perhaps I need ten USB connectors for some virtualized CAD desktops, which require USB key fobs in order to use the CAD software – I plug in a server blade and a USB blade, configure it, and voila, one server, ten USB ports, one license. Expand that out far enough, and you can have whatever you need in terms of peripherals in a blade chassis. If you go to IBM’s website, you get a whole panopoly of choices – switchblades (that one always give me a chuckle) and NIC blades are readily available for expanding your blade chassis out to do more than just host some servers. HP upstages them a bit and has a great product out now that provides PCI-X and PCI-e ports. This is from their website:

“Provides PCI-X or PCI-e expansion slots for c-Class blade server in an adjacent enclosure bay.

  • Each PCI Expansion Blade can hold one or two PCI-X cards( 3.3V or universal) ; or one or two PCI-e cards(x1, x4, or x8)
  • Installed PCI-X cards must use less than 25 watts per card. Installed PCIe cards must use less than 75 watts per PCIe slot, or a single PCIe card can use up to 150 watts, with a special power connector enabled on the PCI Expansion blade.
  • Supports typical third-party (non-HP) PCI cards, such as SSL or XML accelerator cards, VOIP cards, special purpose telecommunications cards, and some graphic acceleration cards.”

This is interesting – a couple of PCI-e quad-NICs in one of an expansion unit and my NIC requirements are set. Or perhaps a couple of PCI-e USB add-in cards. Or a high-end PCI-X or PCI-e video card. Ok that gets troublesome when you need a lot of them – you can wind up with one blade and a chassis full of expansion slits containing video cards – the cost might not be worth it.

In any case, this dramatically changes my view on scaling up or out. Right now, I still stand for scaling up because blades don’t work in my enviornment – I have heat problems. I have space problems too, which blades could solve, but not with my heat problems. I prefer to buy larger-sized servers with lots of expandability (DL300 and 500 series, PowerEdge 2000 and 6000 series, etc.) and add in NICs as needed rather than buy blades or 1U boxes because I can do more with these larger-sized machines even though they take up more room. I fully expect that to change in the future – at some point I see myself stopping with the scaling up and starting with the scaling out – only I expect the “out” part of that will involve a lot less real estate and more options than currently available.

October 2, 2007  12:27 PM

Call for’s Product of the Year

Ryan Shopp Ryan Shopp Profile: Ryan Shopp is now soliciting nominations for its Products of the Year awards. We invite you to nominate your favorite product or your company’s product by using the form at the entry page. Winning products will be featured in January 2008 on

Instructions: staff and other industry experts will judge the entries. Your product(s) qualify for submission if they have shipped (or have been significantly upgraded) between October 31, 2006 and before November 1, 2007.

If you are submitting more than one product, you must fill out a separate form for each product.

Note: Products entered for Best of VMworld awards are eligible for entry. This is an entirely separate award.

The deadline for all submissions is November 2, 2007.

Products are limited to one category. They must fit into one of the following categories for consideration:

-Data protection (Including backup, replication, HA and FT products)

-Systems management: Monitoring and reporting

-Hardware for virtualization (Including, but not limited to: Servers, storage, I/O components and client devices).

-Virtualization platforms (e.g. VMware ESX, VI3, Microsoft Virtual Server, Virtuozzo, XenEnterprise, etc.)

We’ve identified the following criteria as being most important, and will judge accordingly:

* New or upgraded features and capabilities
* Innovation
* Functionality
* If the product is an upgrade, how the upgrade has affected sales and user adoption, and
* User reviews.

Submit your product for entry today!

Bloggers, feel free to mention this in your own blog to spread the word!

October 1, 2007  8:47 AM

Hiring Questions in the Virtualization Age

Joseph Foran Profile: Joe Foran

The need to hire qualified staff to design, implement and manage virtualized environments is growing, and that means hiring managers are having to shift focus towards this distruptive technology and be ready with good interview questions for their prospective hires.

1. Do you have experience in (VMware/Xen/Virtual Iron/Virtuozzo) implementations?

This is the no-brainer question, and the lead in to the others. If the prospective hire’s answer is no, stop right here, do not proceed past go, do not collect $200. Even a certified candidate may not have any experience, and an inexperienced candidate isn’t one you want for the job, since you probably have staff who would like to learn on the job or be trained, and already have the internal processes and procedure knowledge to edge out the competition from outside.

2. When implementing a virtualization environment, what do you consider the most important feature of the product to ensure overall success of the implementation?

This question is good for sorting out who sees the strategic value of virtualization and who is focussed on the technical aspects. A good answer will cover either failover functions or the ability to reduce costs, and relate how they will benefit the business in technical terms. Neither a techie or a managerial answer is right or wrong, but rather will help you sort the crowd of interviewees into the categories you are looking for.

3 . When you were at WidgetMakers, Inc. you list in your resume that you used VirtualBlahBlahBlah to aid your company in meeting the goal of DoingThisOrThat. Can you share with me what challenges you experienced and how you overcame them?

This is a typical interview question surrounding any product, and it needs to be asked for any product you are hiring somebody to work with.

4. How deep is your understanding of storage systems, and can you share an example of how you used this knowledge in a virtualized environment at WidgetMakers, Inc.?


5. How deep is your understanding of network switching, and can you tell me how you would use virtual switches in a broad virtualization implementation?

Cross-disciplinary skills are crucial for virtualization, particularly around storage. Many larger companies have storage administration teams, server administration teams, network administration teams. Being able to work with these groups doesn’t mean that the candidate can work with the technology, and its important that, if the position is technical, that they can do both.

6. Tell me about how you would configure a virtual environment to best take advantage of its features in a backup and disaster recovery framework?

Being able to understand how to use DR-friendly features like VMware’s vmotion and backup-friendly features like snapshots can make all the difference in candidate selection. It’s important for a candidate to know how to keep the business running, even if they don’t know the business itself yet.

7. Tell me about VirtualizationProductFeature, and what you think makes it valuable or not valuable.

This gets into the technical understanding of the product, a crucial point in both technical and managerial interviews. If a technical candidate blows this one, they need to go home. If a managerial candidate doesn’t provide a business-oriented answer, they need to go home or consider a technical position.

8. BadThing happens. Tell me how you would troubleshoot the situation and get it resolved.

A typical technical question, and one that should always be asked to both technical and managerial candidates. Managerial candidates may get some leeway in technical minutia, but absolutely must speak about their role as the manager and how they would deal with their technical staff to get the problem resolved. This is also a rinse-and-repeat question that should be asked a couple of times, using different BadThings.

There are also consultant-specific questions to ask, if that’s what you’re looking for. Things like:

1. How many VCPs do you have on your staff?

Until the other companies start with their own certs, the VCP is where the game is at.

2. How many virtualization projects has your company undertaken in the last year?

The default no-brainer.

3. Do you eat your own dog food? By that I mean does your company use the product internally as well as support it?

Also a no-brainer

4. What was your company’s most spectacular failure?

Everyone is going to tall you about their company’s great success. Make them squirm a bit and tell you about how they failed, then then ask:

5. What did you do to correct the situation?

This will tell you what kind of consulting firm you are dealing with. If they can be upfront with these two questions, if the failure wasn’t a show-stopper for your environment, and if they dealt with it right, they get high kudos.

Obviously there are many, many more questions to be asked of potential staff, managers, and consultants, so many that I’d like to encourage people to comment about questions you like to ask, would like to be asked, or think would be important – I’m looking forward to some audience participation!

September 28, 2007  11:17 AM

VMware clarifies stance on virtual switch security

Bridget Botelho Profile: Bridget Botelho

Following the launch of the article “VMware dispels virtualization myths (sort of),” VMware emailed me to correct some issues about virtual machine security.

According to VMware, an “incorrect statement” was made by Burton Group Analyst Chris Wolf, who, like all of the engineers at VMware he’s spoke with, he thought to be correct.

In the article, Wolf said, “one significant issue with virtual machine security is with virtual switch isolation. The current all-or-nothing approach to making a virtual switch ‘promiscuous’ in order to connect it to an IDS/IPS is not favorable to security.”

For example, “if you connect an IDS appliance to a virtual switch in promiscuous mode,” Wolf said, “not only can the IDS capture all of the traffic traversing the switch, but every other VM on the same virtual switch in promiscuous mode could capture each other’s traffic as well.”

This statement ruffled some feathers at VMware, and they quickly emailed me and Burton to “educate us” and the VMware community that in fact, VMware allows (and encourages) users to configure only the ports they need to be promiscuous as such. This is not a per vswitch setting, but rather a per portgroup setting. The way to configure a vswitch for IDS/IPS is to create a separate portgroup from those used for normal VMs and configure it for “Promiscuous Allowed,” a VMware spokesperson said.

After testing this out in his own lab, Wolf said it is really an easy solution, because the architecture is already there.

“At the switch level, promiscuous mode is an all or nothing configuration. VMware doesn’t argue this. However, a way around this issue is by configuring a separate port group on a virtual switch just for the IDS and making the port group promiscuous. That allows the IDS to monitor the vswitch traffic and still keep all other traffic isolated,” Wolf learned from VMware.

“So, with the port group feature it isn’t all or nothing, it can be granular,” Wolf said. That said, “Vmware’s own team wasn’t even aware of this,” therefore it’s unlikely many VMware administrators are either, he said.

So the record stands corrected. “The option of making a virtual switch ‘promiscuous’ in order to connect it to an IDS/IPS is not favorable to security and should never be used,” Wolf said. Instead, administrators should create a dedicated port group on the switch for the IDS and only make the IDS port group promiscuous. This would allow the IDS to monitor all unicast traffic on the switch while preventing all other VMs on the virtual switch from seeing each other’s unicast traffic.”

September 13, 2007  11:02 PM

Can virtualization stay sexy in 2.0 era?

Ryan Shopp Ryan Shopp Profile: Ryan Shopp

Developments at VMworld 2007 show that virtualization 2.0 has arrived, says Burton Group analyst Andrew Kutz. But can virtualization stay sexy when it is mainstream?

September 13, 2007  9:30 PM

VMworld’s scale wows VMware CEO Diane Greene

Ryan Shopp Ryan Shopp Profile: Ryan Shopp

VMware CEO Diane Greene says VMworld 2007 wowed her with innovation and enormous vendor and user participation.

September 13, 2007  9:29 PM

Virtualization 2.0 started at VMworld 2007, analyst says

Ryan Shopp Ryan Shopp Profile: Ryan Shopp

Analyst Barb Goldworm explains why VMworld 2007 ushered in a new era in virtualization.

September 13, 2007  9:28 PM

VMworld Awards: Printing for the virtual computing age

Ryan Shopp Ryan Shopp Profile: Ryan Shopp

Good-bye to pesky print drivers, hello to virtual printing. ThinPrint’s VDI-focused printing approach won recognition in the VMworld Awards’ Utilities category.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: