June, 2008

What makes VMotion so great?

To many of us, some VMware product features and functionalities still seem like magic. Sure, I understand what the products do, but considering that my education and background is in writing and editing (read: not computer science), how they work still remains a mystery.

At least a few pieces of the VMware Infrastructure feature set were explained to me by Alliance Technologies solutions architect and Central Iowa Virtualization User Group (CIVUG) member Sean Clark. In a recent conversation, Clark explained how VMotion works, what Distributed Resource Scheduler (DRS) is and how the competition stacks up. Check out this audiocast if you’re still using VMware Server or just the ESXi hypervisor but are considering moving into the rest of the suite as it more provide more information to help you with your decision.

Isolate ESX hosts in separate clusters for maintenance, upgrades

Performing key maintenance on a VMware Infrastructure 3 (VI3) host OS or can be an involved process, as can the process of simply adding a host. Because of the length of time that a system may be offline or because of what exactly is being performed, having the hosts in an isolated cluster can allow a safe environment for virtually any task. An isolated cluster will not apply to the same DRS and high availability rules that may apply to the live cluster. Further, you can reboot the host as needed without needing to wonder if there will be any effect to the live workload.

With the isolated cluster, the following tasks can be safely performed:

-Version upgrades (ESX 3.0x to 3.5 Update 1)
-Hardware maintenance
-Adding a new host to an existing cluster
-Testing network connectivity to various port groups
-Confirming VMware HA and DRS performance with in an isolated set of rules
-Importing or configuring new storage types

A good practice for the isolated cluster would be to keep it less visible to the live workload and named accordingly within the VMware Infrastructure Client, so a name like “TestingCluster” or “ZZUpgradeCluster” can distinguish the collection to be different from that of the live workload.

The figure below shows a cluster with a name and position that contains one host in maintenance mode for any task better suited in an isolated environment:

It is important to note that this cluster would still require licensing as it would be in live workload cluster. More information on creating a VI3 cluster can be found in the VI3 online library.

Choosing a block size when creating VMFS datastores

When you create a VMFS datastore on your VMware ESX servers many administrators select the default 1MB block size without knowing when or why to change it. The block size determines the minimum amount of disk space that any file will take up on VMFS datastores. So an 18KB log file will actually take up 1MB of disk space (1 block) and a 1.3MB file will take up 2MB of disk space (2 blocks). But the block size also determines the maximum size that any file can be, if you select a 1MB block size on your data store the maximum file size is limited to 256GB. So when you create a VM you cannot assign it a single virtual disk greater then 256GB. There is also no way to change the block size after you set it without deleting the datastore and re-creating it, which will wipe out any data on the datastore.

Because of this you should choose your block size carefully when creating VMFS datastores. The VMFS datastores mainly contain larger virtual disk files so increasing the block size will not use all that much more disk space over the default 1MB size. You have the following choices when creating a datastore:

• 1MB block size – 256GB maximum file size
• 2MB block size – 512GB maximum file size
• 4MB block size – 1024GB maximum file size
• 8MB block size – 2048GB maximum file size

Besides having smaller files use slightly more disk space on your datastore there are no other downsides to using larger block sizes. There is no noticeable I/O performance difference by using a larger block size. When you create your datastore, make sure you choose your block size carefully. 1MB should be fine if you have a smaller datastore (less than 500GB) and never plan on using virtual disks greater then 256GB. If you have a medium (500GB – 1TB) datastore and there is a chance that you may need a VM with a larger disk then go with a 2MB or 4MB block size. For larger datastores (1TB – 2TB) go with a 4MB or 8MB block size. In most cases you will not be creating virtual disks equal to the maximum size of your datastore (2TB) so you will usually not need a 8MB block size.

So remember, choose carefully, once you create your datastore there is no changing it later.

64-bit guest operating system issues avoided with BIOS configuration

On more than one occasion, I have built a new ESX host system and had immediate migration issues with 64-bit guest operating systems. This is due to an easy-to-forget BIOS default that disables 64-bit processing. To save you the hassle of trying to diagnose why the migration may fail, here is a specific configuration that can prevent this issue.

The symptom is that the virtual machine migration of a 64-bit guest VM fails with the “CPU on host is incompatible with CPU feature requirements of virtual machine” message due to the lack of 64-bit support on the BIOS. To resolve the configuration issue, check the server’s processor configuration to ensure that proper support is enabled. On Dell PowerEdge series servers, the option to enable the functionality so that ESX permits the 64-bit migrations correctly is shown below:

This option can be accessed by pressing F2 during the boot process to gain entry into the BIOS. Go to the CPU Information section to enable the 64-bit support functionality option. Check your server documentation for other platforms.

Once this setting is enabled, you can migrate the 64-bit virtual machines without issue. This situation makes advanced testing on BIOS updates on your ESX servers a very good idea as the migration functionality may be impacted. More information on this issue and related processor topics related to this are discussed on a VMware Communities thread.

Storage and network planning takeaways from Iowa virtualization user group meeting

Sean Clark is a VMware Certified Professional (VCP) and a member of the Des Moines, Iowa-based Central Iowa Virtualization Users’ Group (CIVUG). The CIVUG emerged from the Central Iowa Linux Users’ Group as a way for virtualization users to learn more without being tied to one vendor. That is, the CIVUG discusses Hyper-V and Citrix Systems’ Xen in addition to VMware.

Clark has cut his teeth on VMware and other virtualization platforms as a solutions architect at Alliance Technologies, where he works with small and medium—sized businesses and some enterprise businesses to develop strategies to best implement virtualization, be it storage, server or application virtualization. Clark posted notes on a presentation he did at a recent meeting on Red Hat Network File System (NFS) configuration being used with VMotion, High Availability (HA) and Distributed Resource Scheduler (DRS).

NFS vs. Fibre Channel and iSCSI
The NFS configuration how-to Clark followed was posted by Mike Laverick on RTFM Education and was chosen for the demo not only because of it’s easy configuration, but also because implementing Red Hat NFS is a good low-cost alternative to shelling out big cash for more expensive devices for virtual machines (VM) storage. “Most people think that you need a Fibre Channel SAN [storage area network] or an iSCSI SAN; you’ll spend at least 20 or 30 grand on that device. But all you really need is a reasonably new server with some pretty fast discs, and you can basically have a SAN for the cost of your hardware.”

Clark also says that despite what some benchmarking stats say, performance won’t suffer on the cheaper NFS. “If you do your homework, you’ll find that in many benchmarks when you compare the performance of NFS versus iSCSI, they’re almost nearly identical. You can spin your benchmarking tasks so that one will come out on top every time, but for the majority of uses out there, and especially for test and dev, NFS is just as good as iSCSI.”

The storage system lines blur when you scale out to enterprise-level requirements because the needs go beyond the hardware and software backing your VM storage. Clark says that it’s really about having enough discs. “It doesn’t matter if you have NFS, Fibre Channel or iSCSI if you don’t have enough discs to meet the I/O demand that the number of virtual machines can put on a SAN. Most of the time, it becomes almost a religious decision — whatever you’re most comfortable with.”

HA snafu highlights proper network configuration
After walking through the NFS configuration, Clark went on to demo VMotion, HA and DRS, but ran into some problems with the HA portion. At his home in Pella, Iowa, Clark has a small test lab of two 1U servers running two ESX VMs, which he brought to Des Moines for his presentation.

With little time to reconfigure the servers (Clark was asked only a few days prior to the meeting to give the presentation), Clark decided against bringing his PC that he had used as his NFS server in favor of a Red Hat VM running on his laptop, which served as the basis for his presentation. Although both the ESX servers were able to see the Red Hat VM through a little Linksys switch, the demonstration came to a standstill during the HA portion.

The problem? Because the conference room that the VUG meets in is an island network, there is no default gateway address available as there was at Clark’s lab in Pella. An available, ping-able gateway address is required for HA to work. This is because when HA is set up, each ESX server establishes and communicates through a heartbeat and that’s how it determines whether each server is awake or not. If that heartbeat ceases, then HA makes a decision to restart the VMs that are running on a different ESX server. Sometimes the network becomes unavailable, but the other ESX server continues to run.

The isolation address was the pain point in Clark’s HA demo. The default gateway that used to exist in Pella didn’t exist and HA failed. As Clark explains, “It’s just an additional IP address that an ESX server can gain to say, ‘OK, I lost my buddy, but I need to make sure that the network is still up.’ So as long as it can reach that other IP address, then it can assume that the other host is actually truly dead and that it needs to restart its failed VMs.”

Proper planning and redundancy is key to successful deployments
Clark says that he almost never has problems because he is careful to use VMware-supported hardware and plans out each deployment carefully. But not all IT departments are so careful. “I ran into a customer the other day that didn’t want to invest in redundant networking; they had a four-host cluster and all their networking was going through one physical switch. They had HA set up, and when that switch went down, ever single VM that was on one of their ESX servers powered down.”

Clark says that this can happen as a result of not properly planning your configuration. “One of the configuration options for HA [concerns] what to do if an ESX host becomes isolated. Because it lost its network, there wasn’t a second physical switch for redundancy. Each host, even though it was still running, became isolated according to the HA configuration. It was probably a default setting at the time when they set up the HA cluster, and it powered down all the VMs.” Clark says that with proper preparation, the ESX hosts may not have appeared isolated to HA and this organization may have been able to save the headache of restarting all its VMs and troubleshooting what caused the power-down.

Ensure VMware Infrastructure Client is current

When VMware issued VirtualCenter 2.5 Update 1 and ESX 3.5 Update 1 in April of this year, I felt that this release was mature enough to upgrade my environment. One observation I have made is related to the VMware Infrastructure Client (VIC) versioning. The VMware Infrastructure 3 release notes mention many configurations related to supported environments of VirtualCenter and ESX, but do not distinguish between the base release of the VIC for VirtualCenter 2.5 and 2.5 Update 1.

The VIC for VirtualCenter 2.5 (base release) has a build number of 64192 whereas the VIC for VirtualCenter 2.5 update 1 has a build number of 84767. Unlike the upgrade from VirtualCenter 2.0 to VirtualCenter 2.5, connections into VirtualCenter are not prompted to download and install the new version of the VIC. Therefore, you can have VIC installations with the older version interacting with the newer VirtualCenter server. I recommend that you enforce the version currency of the VIC on all systems connecting to a VirtualCenter 2.5 Update 1 to be at build 84767 so that commands match the capabilities of VirtualCenter. For most situations, this is not going to be an issue, but should you use VirtualCenter plug-ins, advanced features, or have a large amount of VIC traffic there may be some anomalies in the version mismatch.

The figure below shows two information screens from two installations of the VIC. Both of these installations are interfacing to the same VirtualCenter server, but the top one is at the current version:

During the VirtualCenter upgrade process, the VIC is updated locally on the server. Further, unlike the VIC for VirtualCenter 2.0 the VIC for VirtualCenter 2.5 and 2.5 Update 1 have the same binary update path, so you will not have to manage two shortcuts for access.

Upgrading VirtualCenter does not update certificate store

When logging into the VMware Infrastructure Client (VIC) or using the web interface to the VirtualCenter server, you are presented with a certificate message. Best practice or not, I usually accept the certificate and instruct the software to not ask me again about this topic. I have just completed end-to-end testing of VirtualCenter 2.5 update 1 and ESX 3.5 update 1, however, and noticed something about the certificate store for VirtualCenter.

The VirtualCenter certificate store is valid for two years from the date of initial installation. Since this instance of VirtualCenter was installed, I have upgraded to version 2.5 base release and, most recently, to the version 2.5 update 1. But neither installation updated the certificate. Below are the details of my test certificate:

The good news is that you now know about this issue. The bad news is that you better correct it before the two year anniversary of your installation of VirtualCenter as it is required to process logins. VMware has a comprehensive PDF that outlines the certificate procedures for VirtualCenter and the ESX hosts. The ESX hosts, however, have a much longer lifespan for the local certificate, around 20 years, and do not exhibit this behavior. The VMware server certificate documentation is available for download from the VMware website.

Changing the VMware Server 2.0 default permissions

I have been using VMware Server 2.0 (beta 2) on both Windows and Linux platforms for a while now. For Windows systems that are a member of an Active Directory domain, there are inherited permissions that may be assigned from Group Policy. If you want to change that, here are a couple of pointers in changing the security model.

To start looking at the permissions of the server installation, click the server on the left side of the browser view and then click the permissions tab at the top. The default permission is the local Administrators of the Windows system will be in the VMware Server Administrators role as shown below:

Before you make any modifications to the security model, add your desired configuration. That way, you can protect yourself from orphaning your administrative access to the server. Click the New Permission link in the command section, and add the user from either the local accounts of the Windows system or a user from the Active Directory domain or security group from the domain. The figure below shows the addition of the RWVVMwareAdmins group to the role of Administrator within the VMware Server 2.0 web interface:

Once that is added, the new configuration should be tested to ensure the proper access is available. Once the new access is verified, it would be safe to then remove the previous default access (if needed). If you get stuck, you can save off the .VMDK files and reinstall the product if needed.

While the web interface for VMware Server 2.0 takes some getting used to when compared to the thick client for versions 1.0x, features continue to be added to the free virtualization product that can be suited to test and development or live environments.

VMware adds ODM relationships with Tyan, ASUS, Gigabyte, Inventec

At Computex Taipei 2008on June 3, VMware, Inc. announced new relationships with original design manufacturers ASUS, Gigabyte, Inventec and Tyan, adding to VMware’s current relationship with Supermicro announced in February 2008.

The relationships lets channel partners and system builders make virtualization available to a broader range of customers by certifying various one-, two- and four-socket servers and blade servers for the VMware virtualization platform.

The vendors are certifying their server systems for the VMware ESX and ESXi hypervisor, with availability expected by the end of the third quarter of 2008.