The Virtualization Room


August 19, 2008  8:39 AM

Is a 100% virtualized environment possible?

Eric Siebert Eric Siebert Profile: Eric Siebert

Organizations that have virtualized their environments often virtualize only a portion of their servers, leaving some servers running on standalone physical hardware. Is a 100% virtualized environment possible? Certainly it is, because almost all workloads can be virtualized, but there are some arguments against completely virtualizing your environment.

I recently wrote about an experience I had with a complete data center power failure. The problems resulted from all the DNS servers being virtualized and until the host servers and storage-area network were online no DNS was available, which made it difficult for anything in the environment to function properly. Having a DNS server and Active Directory domain controller running on a physical server would have been a great benefit in that situation.

Additionally, many organizations are leery of having too many servers virtualized because they want to avoid the risk of a single host outage causing many virtual machines to go down at once. This risk can be partially offset by some of the high availability features that are available in many of the virtualization products. In addition, if a virtual environment relies on a single shared storage device and that device has a major failure, it can take down all the virtual machines that reside on that storage. This risk can also be partially offset by having a well architected SAN environment with multiple switches and host bus adapters so multiple paths to the SAN are available.

Another reason that you may not want to virtualize your whole environment is that many software vendors do not fully support running their applications on virtual machines and subsequently may require you to reproduce a problem on a physical system. Because of this it is a good idea to have a few physical servers running applications that may be effected by these policies. For example, if you have multiple Oracle, SQL or Active Directory servers, consider leaving one or two of them on physical hardware.

Finally, you may consider leaving a few physical servers for applications that have non-virtualization friendly licensing and hardware requirements that can be difficult to virtualize (licensing dongles, fax boards, etc.) or for servers that have extremely high I/O requirements.

So is a 100% virtualized environment possible? Yes it is, but is it advisable? In most cases it is not recommended. The cost savings that are typically seen by implementing virtualization will increase the more an environment is virtualized but you may want to stop at around 90% and leave a few physical server for the reasons that were previously mentioned.

August 15, 2008  8:19 AM

News of the week: VMware’s ESX 3.5 bug causes VM failure

Bridget Botelho Bridget Botelho Profile: Bridget Botelho

This week, the biggest news item on SearchServerVirtualization.comVMwarewas the havoc caused by VMware Inc.’s ESX 3.5 update 2 bug, which kept virtual machines (VMs) from booting up and live migrating (VMotion) on Aug. 12.

Users posted their fury on IT forums like ARS Technica’s the Server Room. One user on the forum summed up the situation perfectly. “This was a very big deal, make no excuses for VMware. It certainly had potential to completely disrupt a lot of customers. … At most it should have disabled VMotion and other extras but not starting a VM.”

In the afternoon on Aug. 12, VMware issued an Express Patch on its Knowledgebase site and warned users not to install ESX 3.5 Update2 or ESXi 3.5 Update 2 if it has been downloaded from VMware’s website or elsewhere prior to Aug. 12, 2008.

VMware’s new CEO, Paul Martiz, issued an apology letter the day of the bug explaining the issue.

When the time clock in a server running ESX 3.5 or ESXi 3.5 Update 2 hits 12:00AM on August 12th, 2008, the released code causes the product license to expire. The problem has also occurred with a recent patch to ESX 3.5 or ESXi 3.5 Update 2. When an ESX or ESXi 3.5 server thinks its license has expired, the following can happen:

  • Virtual machines that are powered off cannot be turned on;
  • Virtual machines that have been suspended fail to leave suspend mode; and,
  • Virtual machines cannot be migrated using VMotion.

The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2. This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.

… I am sure you’re wondering how this could happen. We failed in two areas:

  • Not disabling the code in the final release of Update 2; and
  • Not catching it in our quality assurance process.

We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

It remains to be seen whether Maritz’ apology is enough to satisfy frustrated users. A major issue like this may prompt users to try other virtualization products. For instance, the day of the incident, some users were singing praises of Microsoft Hyper-V on technical forums.

Either way, having to deal with this issue after only a month in charge is really initiation by fire for Maritz.

And I imagine that VMware co-founder and ex-CEO Diane Greene, who was ousted by VMware’s board of directors July 8, might feel at least somewhat vindicated.


August 14, 2008  11:33 AM

Powerful scripting options with the VBoxManage command

Rick Vanover Rick Vanover Profile: Rick Vanover

Sun xVM VirtualBox offers a powerful command-line interface (CLI) component, VBoxManage, which can perform most functions within VirtualBox. Having a robust CLI is key to automation and scripting, even in a workstation virtualization product. In my continued coverage of VirtualBox, this blog will highlight some of the parameters of VBoxManage with some examples and areas where this command can be of use.

Modifyvm parameter
Probably one of the more versatile commands for VBoxManage is modifyvm. This parameter can set memory, operating system type, pae settings, monitor quantity, hardware inventory as well as snapshot configuration. Here is a sample command that sets a memory amount, makes the CD-ROM disk the first boot device and disables USB support:

vboxmanage modifyvm XP-TestSystem -memory 512 -boot1 dvd -usb off

The modifyvm parameter also has extended options such as BIOS display time, network interface driver type, host network interface assigned to the VM and enabling or disabling of the clipboard. Overall, modifyvm has over 50 parameters for an individual VM.

Controlvm parameter
From an automation standpoint, the controlvm parameter would be used to start a VM at host system boot. Controlvm can also be used to attach USB or DVD devices. The entry below will disconnect the media of the two virtual Ethernet adapters and reset the power state:

vboxmanage controlvm XP-TestSystem reset setlinkstate1 off
vboxmanage controlvm XP-TestSystem setlinkstate2 off

Note that in the case of an inventory of multiple devices of the same type, a separate entry would be required as in the case of disabling the network interface.

Snapshot parameter
The snapshot parameter can be used to manage all elements of a snapshot. In the case of a frequently used test system, it may be a good idea to automate the change back to the base snapshot. The following command would revert a VM to the existing snapshot:

vboxmanage snapshot XP-TestSystem discardcurrent -state

This command cannot be executed while the VM is running, yet a leading with a power down controlvm parameter can get the system to a state where running the snapshot parameter will do the trick.

Powerful Stuff
This is a very quick sample of what is capable with the VBoxManage command. I don’t know of anything that can be done in the interface that cannot be done with this command. VBoxManage commands also interact the same way across different platforms of xVM VirtualBox. This flexibility offers a compelling solution for an automated deployable solution at the zero-cost of xVM VirtualBox. The online user manual has the entire chapter 8 dedicated to the VBoxManage command, which can be downloaded from the VirtualBox website.


August 14, 2008  11:27 AM

Is VMware’s apology enough?

Eric Siebert Eric Siebert Profile: Eric Siebert

In the aftermath of the infamous bug in the latest release of VMware ESX, VMware CEO Paul Maritz has released a letter that apologizes for the incident and also explains what went wrong and how they are committed to ensure it never happens again.

For customers who were effected by the widespread problem with ESX 3.5 Update 2 released several weeks ago, is VMware’s apology and promise to improve their processes enough? Or is it going to leave some lingering doubt in the minds of some that may inspire them to look at other virtualization products?

The letter provided an explaination of what what happened:

The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2.  This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.

And why it happened:

I am sure you’re wondering how this could happen.  We failed in two areas:

  • Not disabling the code in the final release of Update 2; and
  • Not catching it in our quality assurance process.

And finally what they will do to ensure it never happens again:

We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers.  We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

Despite it all, VMware still has a great enterprise product that is robust and mature and is still the virtualization software of choice for most Fortune 500 companies. This incident still could have easily been prevented by following processes when preparing a beta build to become a final build. In addition, their QA processes which are usually designed to ensure a quality product also failed to detect that the time bomb code was still present and active.

Will VMware learn from this incident? Absolutely. Sometimes it takes a big event like this to inspire changes and improvements in a company that may have been set in its ways and wasn’t paying attention to details.

One area that many users were critical of was VMware’s communication on the matter. They were initially slow to issue public communications and proactively contact customers to let them know about the issue. The thread in the VMware Technology Network (VMTN) forums that was started on this issue became the rallying point for many of the users who were experiencing problems as a result of the bug. VMware employees did provide some updates to the thread which let users know they were aware of the bug but did not provide much other information until much later in the day. Another breakdown was that VMware’s knowledgebase that had information on the bug and is often the first place users go to when experiencing a problem becamse so overwhelmed by the number of requests that it was unavailable for over 6 hours.

VMware delivered the fix for the problem fairly quickly as it was available roughly 24 hours after the problem was first reported. Many users were hoping to get it quicker then that, but VMware needed time to package and test the fix before releasing it. VMware also did provide good communication later in the day with detailed updates and emails that were sent to customers.

So is VMware’s apology enough? In my mind it is. Yes, it was an unfortunate incident that caused many customers a good deal of grief but the end result is that VMware responded quickly and effectively and this incident will serve as a lesson that they won’t soon forget and will help make their products and processes stronger going forward.


August 14, 2008  10:56 AM

New HP unit to offer one-stop desktop virtualization shopping

Ryan Shopp Ryan Shopp Profile: Ryan Shopp

Hewlett-Packard (HP) has expanded to incorporate a new business unit that will focus on desktop virtualization. Client-side virtualization has been a tough sell, primarily because the ROI comes from areas that companies aren’t used to valuing, such as security, technical support and power savings over regular desktops or notebooks. “Generally, these are encompassed in the facilities cost of the buildings,” said Roberto Moctezuma, VP and GM of the new desktop solutions global business unit. “When we present the ROI case to customers they find it very compelling, but it’s not as granular as in the server virtualization world.”

Roberto Moctezuma

Although no major product announcements will be made for another two weeks, HP shows promise in the client-side virtualization space due to its Remote Graphics Software (RGS), a technology that seeks to address one of the obstacles of virtual desktop adoption to date: the lack of graphics capabilities. RGS enables remote users to access high-end graphic software such as CAD.

Mixed virtual environments
HP hopes to position itself as a one-stop shoppinag vendor for all types of virtualization and data center purchases. “We think we have a unique portfolio in terms of breadth from the client-side and access devices, to notebooks to hardware for the data center coupled with differentiated software and manageability offerings.”

Partnerships with all three major virtualization vendors (VMware, Microsoft and Citrix) have allowed HP to preload their thin client devices with any of the above platforms, which enables a plug-and-play like experience for the end-user.

“When you look at the market a couple of years out, people will be using all types of virtualization platforms and technologies. HP’s strategy is to be the best option that brings them all together,” he said.

For VMware users, HP entering the thin-client virtualization space could mean fewer headaches about hardware and software compatibility in the future. “HP is working very closely with VMware to make the client virtualization paradigm simple to use and deploy. We want to provide a leading experience for our customers, both from an end user perspective and an IT perspective,” Moctezuma said. HP can now provide the thin-client, the virtualization software via its partnerships with virtualization software companies, and the blade servers to run the system.

Security still a major concern
There is a need for businesses to have the ability to keep data secure in an increasingly mobile work force, and the ability to add end-users quickly when, for example, opening up a call center in a new country, Moctezuma said.

Security is a primary concern in health care and financial industries. “You get high-CPU consumption users in the financial trading industry that are running four 24-inch monitors off of one thin-client device, accessing a single blade workstation back at the data center,” he said. Moctezuma also said that other high-end users get better hardware utilization because of worldwide remote employees: now, a blade server can be accessed around the clock from different parts of the world via employees’ thin client devices. HP calls this the “follow the sun” model.

 


August 13, 2008  9:24 AM

Sun xVM Virtual Box expands reach with global OEM agreements

Bridget Botelho Bridget Botelho Profile: Bridget Botelho

Santa Clara, California-based Sun Microsystems, Inc. announced a handful of multi-year original equipment manufacturer (OEM) agreements with Avanquest Software, Q-layer and Zenith InfoTech Ltd. to allow them to deliver Sun’s xVM VirtualBox virtualization platform.

Sun xVM VirtualBox software is a component of Sun’s broader xVM virtualization and management software portfolio, which includes Sun xVM Ops Center, Sun xVM Server and the Sun Virtual Desktop Infrastructure (VDI) Software. The xVM VirtualBox software is the free, entry-level offering into the Sun xVM platform.

Sun xVM VirtualBox supports whichever operating system and application stack a user chooses, and has a small enough footprint to be an embedded component in OEM equipment. 

Since its release in January 2007, Sun xVM VirtualBox has surpassed 5 million downloads, and is the first free hypervisor to support all major host operating systems, including Mac OS X, Linux, Windows, Solaris and OpenSolaris.

The 20 megabyte download installs in less than 5 minutes, and has received positive third-party reviews and awards, and is being used by the Texas Advanced Computing Center, or TACC on part of its 4,000-node supercomputer.

La Garenne-Colombes, France-based Software publisher Avanquest Software will produce and publish Sun xVM VirtualBox bundled with OpenSolaris and sell it in retail outlets in the UK, Germany, Italy, Spain and France. Beginning this fall, Avanquest will provide Mac users with a solution to run the Windows operating system through Sun xVM VirtualBox.Mountain View, Calif.-based Q-Layer, a provider of cloud computing through Virtual Private Data Centers (VPDC), is using Sun xVM VirtualBox to deliver virtualization capabilities for its customers.

Bombay, India-based Zenith InfoTech Ltd., a managed services and business continuity software provider, has built its network attached storage appliance for small and medium-sized businesses using Sun xVM VirtualBox.

Sun xVM VirtualBox is available free of charge under a Personal Use License. OEMs have two options for licensing xVM VirtualBox: open source edition under GPLv2 or under a commercial license.


August 12, 2008  11:46 AM

Major ESX bug plagues thousands of VMware customers

Eric Siebert Eric Siebert Profile: Eric Siebert

A bug in the latest versions of both VMware ESX and ESXi (3.5 Update 2) has effected many of VMware’s customers — and VMware is asking its users to wait 36 hours for a patch.

As the date changed to August 12, 2008, customers were finding out that they could no longer start virtual machines on there ESX hosts or vMotion them to other hosts.

A post was made to the VMware Technology Network (VMTN) community about this bug to which many customers responded that they were experiencing the same problem and had spent hours trying to figure out what was wrong. The problem was not immediately obvious to most because the error that was being displayed was that a general system error has occurred, the actual error that could be found by going through the virtual machine log files was that the product had expired. Many users contacted support, who eventually figured out they had a major issue on their hands.

Currently, the only workaround for this is to set the host clock back and to restart virtual machines; however, this workaround is not acceptable for many customers who rely on accurate time for their systems and applications as well as to satisfy compliance regulations. Virtual machines that are already running are not effected by this bug unless they are rebooted or powered off and back on.

The bug appears to have been code that was left in the beta version of ESX to stop working on a specific date after the beta had ended. This is commonly done by software vendors and is known as “time bombing”: software stops working past a certain date and users are forced to use the latest gold version instead of continuing to use the beta version.

VMware has published a knowledgebase article on this issue and promises to release a fix within 36 hours. For most customers this is not enough, having to wait 36 hours is much too long for a problem of this magnitude. They are looking for an immediate fix to the problem so they can apply it to their effected hosts. Additionally there is concern about how the fix will be delivered, presumably it will be released as a new build of ESX which will require ESX hosts to be offline as it is installed and they are re-booted.

Many customers posting to the VMTN thread have expressed anger and frustration at VMware for this. To make matters worse and further frustrate users, VMware’s knowledgebase went offline shortly after the document was published presumably because it could not handle the extraordinary amount of requests.

It is hard to believe a company the size of VMware could allow this to happen. Something like this could not be picked up in beta testing and is not necessarily a bug but negligence on VMware’s part by not removing or disabling this code before it was released as the gold version. Most software companies have strict processes for developing, testing and performing quality assurance before releasing a new build. How something like this could happen is anyone’s guess right now but it appears that either processes do not exist or they were simply not followed.

In the meantime, customers continue to wait for VMware to release a fix for this. Because of the severity and the effect on so many customers there will most likely be some type of fallout at VMware over this. Something needs to be done for VMware to assure customers that they are taking this very seriously and are committed to doing everything possible to ensure that this never happens again. With Hyper-V now a viable alternative, VMware can’t afford major mistakes like this.


August 6, 2008  8:42 PM

New Oracle VM Templates speed Oracle app installation

Bridget Botelho Bridget Botelho Profile: Bridget Botelho

I sat down with Oracle Corp.’s VP of Linux engineering, Wim Coekaerts, during the LinuxWorld/Next Generation Data Center conference in San Francisco today to find out about their new virtualization product, Oracle VM Templates, announced August 6.

Oracle VM Templates are basically a time-saving approach to deploying a fully configured Oracle software stack, since each template provides pre-installed and pre-configured images of Oracle software.

Oracle VM Templates can be downloaded for free to install Oracle products onto servers where Oracle VM resides. Following the download, the servers will have a fully installed and configured software environment (based on Oracle’s Linux product) to play with, Coekaerts explained.

“The VM Template includes all of the patches and anything else that we would want our customers to have, and they don’t have to do any of the install work,” Coekaerts said. “The template is good for test and dev, because it allows people to compare us to other products. They download it and can play around with it, and if they are happy with it, they then pay the normal oracle Licence ad support fees for whichever product they are using the template for.”

The VM templates are not replacing the traditional method of installing Oracle software, but is simply an alternative he said.

The first set of Oracle VM Templates are now available for Oracle Database 11g, Oracle Enterprise Manager, Oracle’s Siebel CRM 8, and Oracle Enterprise Linux.

Coekaerts said Oracle’s goal is to come out with a VM Template for existing Oracle products about once a month, and there will be a library of templates users can use to deploy Oracle software quicker than it would traditionally take. Oracle partners will also use the templates to help users deploy their software, he said.

The templates are essentially a way for Oracle to promote its Xen-based hypervisor, Oracle VM, which supports both Oracle and non-Oracle applications. Oracle VM is also available for free download, and users pay for support.


August 6, 2008  8:08 AM

More on optimizing virtual hosts

Rick Vanover Rick Vanover Profile: Rick Vanover

Eric Siebert’s recent post on optimizing the host environment is a very important concern that may frequently be passed aside in the interest of reducing implementation time for virtual environments. In this blog, I would like to pipe in with a few of my own tips related to the host environment. These strategies are applicable to many virtualization platforms, and will transcend products as virtualization advances.

DNS configuration for the hosts
Having a correct DNS environment is important for all systems, not just virtual environments. Pay particular order to the suffix search order, as the first result for queries should be consistent and timely across hosts. Also, consider host entries for fixed systems, with an entry for the host itself, all other hosts, the management system and any other relevant systems with which the host would need to communicate. A specific issue is VMware’s DRS functionality, which can have issues with incorrect DNS configurations.

Time configuration for the hosts
For platforms that are Windows based and members of an Active Directory domain, this concern is somewhat eased. But for Linux systems, you want to have an automated mechanism in place to manage accurate time across hosts. For ESX and VirtualCenter, Eric again has covered this well over on SearchVMware.com with a tip.

Also decide whether you want guest virtual machines to sync time with the host via the driver software (VMware Tools, Guest Additions, etc.) This will relieve issues that go with multiple time zone support as well as separate issues in time synchronization.

Get environment agent notifications right
For virtualization hosts on the server level, all hardware failure notifications should be configured to the fullest extent possible. This can be device alerts (Dell DRAC/HP iLO), SNMP alerts, agent configurations or even blade server management software. With the scope of the virtual environment, maybe even use multiple notification mechanisms.

Single hypervisor per platform
This is more relevant on desktop environments, but it goes without saying that you should not install two products on a single system. Even though it may be tempting to have the functionality of multiple platforms, it may complicate the host environment. Take VMware Server and Sun xVM VirtualBox as an example, they theoretically could exist on same systems because of the VMware Bridge protocol binding and the VirtualBox explicit host adapters able to have their own configuration. This is one of those just-because-you-can-does-not-mean-you-should scenarios.

Host configuration is an area ripe for configuration procedures and policy enforcement to ensure consistent behavior among host systems. The procedural investment can usually help present the virtualization solution with more credibility as well.


August 4, 2008  8:05 AM

VBox 1.6.4 upgrade fixes folder sharing, VRDP

Rick Vanover Rick Vanover Profile: Rick Vanover

Sun has released VirtualBox 1.6.4, and the upgrade process requires some forward planning. Version 1.6.4 is a collection of fixes to the previous release that mostly revolve around shared folders and the VRDP (VirtualBox Remote Desktop Protocol) implementation. Here is what you need to know if you are upgrading:

During the upgrade installation, you are presented with the familiar message about installing a device that has not passed Windows logo testing. These messages are common across virtualization platforms, as these drivers and devices enable the hypervisor to present the virtual machines. A typical message is shown below:

Windows message

After these messages are accepted, the installation will continue and allow you to access your existing VMs from the previous version that you may have. 

The one unfortunate point of the upgrade process is that any host interfaces created on an existing installation of 1.6.2 or earlier will be removed by the upgrade process. Overall, I think VirtualBox’s networking implementation is a little short of both VMware Workstation and VMware Server’s VMware bridge protocol. Before you embark on the upgrade, I recommend you enumerate any host interfaces that you have created. Then, make a quick script in the following fashion that will recreate them with the same names you already have:

VBoxManage createhostif "VM-Bridge1"
VBoxManage createhostif "VM-Bridge2"
VBoxManage createhostif "VM-Bridge3"

Any VMs with a bridged interface will be configured to an invalid network interface after the upgrade to 1.6.4. I have an earlier blog posting about bridged networking on VirtualBox, and the commands and planning points are unchanged from 1.6.2 to 1.6.4.

The VMs will not need to be upgraded directly, but it would not hurt to get the 1.6.4 version of Guest Additions installed to optimize the corrected functionality between these two versions. Once the new version is installed, the systray icon and the VBoxControl getversion will show the 1.6.4 release.

Version 1.6.4 is still lean, at only 23 MB, it remains a ready to go virtualization platform and is still freely available from the Sun website.


Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: