Rick Vanover

Strategies for determining the number of hosts per cluster in VI3

In discussions with other administrators, determining the number of hosts per cluster gets a wide range of response. Although there are a number of factors, the unfortunate simple answer for every installation is “it depends.” Let’s tour of some of the reasoning behind various hosts per cluster configuration.

Similar generations of hardware

Basing cluster configurations on similar hardware is a common logical type of cluster separation. This is because VI3 does not currently support dissimilar migrations to different processor systems with VMotion technology. Another common cluster configuration is to separate clusters following the internal proof of concept, a cluster separation that makes the business and internal IT totally comfortable with VMware virtualization. This generally has the ‘pilot’ equipment in a cluster because it was possibly taking on a less important role due to new resources being added.

HA and DRS configurations and standby capacity

Depending on the granularity of the VMware HA and DRS configuration, more or less hosts may be required when considering separation and failures permitted values. These generally make the difference between one or two more hosts in clusters with less than ten hosts.

Many administrators implement VI3 as part of the disaster recovery (DR) plan, and there may be a cluster with excess host capacity planned for use in a DR situation. This configuration will either mirror the primary cluster’s host capacity or be at a percentage needed to sustain the DR workload.

Separating development, test, QA, and live workloads

Depending on the requirements to define the process for internal systems to go from the conceptual stages to a live workload, different clusters may hold these stages well. Separating the workloads between clusters is a natural protection from a resource perspective and can make the process more likely to be enforced. On the same token, the various clusters should be configured respective to their roles. For example, the development cluster should not have access to the live network segment or live storage system.

Internal and external-facing systems

Over on SearchServerVirtualization.com, I posted a blog about putting external-facing VMs on the same hosts that hold internal workloads. Within VI3, these workloads would be better suited on a separate cluster. This can lower the risk of a hypervisor vulnerability affecting an internal workload.

Cost allocations

Some clusters may be configured and populated exclusively by funding situations. Various large customer projects, consulting entities, or departmental chargeback may dictate how the clusters are configured. This granular approach may effectively create a large amount of underutilized capacity, but may be unavoidable when it comes to the financial traits of an organization.

More powerful hosts or more hosts?

This is a tough one to call as there are benefits either way to having a large number of very capable systems or a smaller number of lesser expensive systems. Both systems are comparatively large systems, but in terms of an ESX host the smaller system may be a dual socket, dual core system with 32 GB of RAM and the larger system may be a quad socket, quad core system with 128 GB of RAM. Both are good ESX host candidates, and there are many upfront cost factors with each configuration.

What is your strategy on cluster configuration?

As you can see, there are many approaches to this topic. VI3 is quite adaptive to most configurations, but upfront planning remains important with this important configuration topic. Share your comments below on your cluster configuration.

VMware Server 2.0 virtual machine version compatibility note

It goes without saying that working with a beta incurs a small amount of risk. With VMware Server 2.0 beta 2, I came across a particular situation that caused an issue with a virtual machine (VM). My CentOS Linux system had been running VMware Server 2.0 beta 1 since its initial release. On a separate Windows system, I had been running VMware Server 2.0 beta 2. To archive the VM, I had copied it from the beta 2 system to the beta 1 system. After the VM was copied over, I could add it to the virtual machine inventory, modify the hardware inventory to get the processor and RAM configuration correct for the destination system, however I could not power it on.

I had seemingly forgotten that the Linux system was at the beta 1 release. After downloading beta 2, a quick run of the vmware-install.pl file uninstalled beta 1 and then installed beta 2. The virtual machine, however, was still unusable due to the backward version modifications. A quick scan of the release notes for beta 2 did not make specific mention of this situation, however it makes sense that the older build (# 63231) would not be able to power on and use the newer build (#84186). Once the Cent OS Linux system was upgraded to beta 2 and had the VM copied back over, it was able to power up without modification.

The gotcha: if I would have performed an initial move of the VM instead of a copy, it would have been destroyed!

VMware products generally perform well in the category of moving VMs between versions that are on different host operating systems, as well as importing VMs to newer products. But, exercise a little bit of caution in the beta use of VMware Server 2.

Isolate ESX hosts in separate clusters for maintenance, upgrades

Performing key maintenance on a VMware Infrastructure 3 (VI3) host OS or can be an involved process, as can the process of simply adding a host. Because of the length of time that a system may be offline or because of what exactly is being performed, having the hosts in an isolated cluster can allow a safe environment for virtually any task. An isolated cluster will not apply to the same DRS and high availability rules that may apply to the live cluster. Further, you can reboot the host as needed without needing to wonder if there will be any effect to the live workload.

With the isolated cluster, the following tasks can be safely performed:

-Version upgrades (ESX 3.0x to 3.5 Update 1)
-Hardware maintenance
-Adding a new host to an existing cluster
-Testing network connectivity to various port groups
-Confirming VMware HA and DRS performance with in an isolated set of rules
-Importing or configuring new storage types

A good practice for the isolated cluster would be to keep it less visible to the live workload and named accordingly within the VMware Infrastructure Client, so a name like “TestingCluster” or “ZZUpgradeCluster” can distinguish the collection to be different from that of the live workload.

The figure below shows a cluster with a name and position that contains one host in maintenance mode for any task better suited in an isolated environment:

It is important to note that this cluster would still require licensing as it would be in live workload cluster. More information on creating a VI3 cluster can be found in the VI3 online library.

64-bit guest operating system issues avoided with BIOS configuration

On more than one occasion, I have built a new ESX host system and had immediate migration issues with 64-bit guest operating systems. This is due to an easy-to-forget BIOS default that disables 64-bit processing. To save you the hassle of trying to diagnose why the migration may fail, here is a specific configuration that can prevent this issue.

The symptom is that the virtual machine migration of a 64-bit guest VM fails with the “CPU on host is incompatible with CPU feature requirements of virtual machine” message due to the lack of 64-bit support on the BIOS. To resolve the configuration issue, check the server’s processor configuration to ensure that proper support is enabled. On Dell PowerEdge series servers, the option to enable the functionality so that ESX permits the 64-bit migrations correctly is shown below:

This option can be accessed by pressing F2 during the boot process to gain entry into the BIOS. Go to the CPU Information section to enable the 64-bit support functionality option. Check your server documentation for other platforms.

Once this setting is enabled, you can migrate the 64-bit virtual machines without issue. This situation makes advanced testing on BIOS updates on your ESX servers a very good idea as the migration functionality may be impacted. More information on this issue and related processor topics related to this are discussed on a VMware Communities thread.

Ensure VMware Infrastructure Client is current

When VMware issued VirtualCenter 2.5 Update 1 and ESX 3.5 Update 1 in April of this year, I felt that this release was mature enough to upgrade my environment. One observation I have made is related to the VMware Infrastructure Client (VIC) versioning. The VMware Infrastructure 3 release notes mention many configurations related to supported environments of VirtualCenter and ESX, but do not distinguish between the base release of the VIC for VirtualCenter 2.5 and 2.5 Update 1.

The VIC for VirtualCenter 2.5 (base release) has a build number of 64192 whereas the VIC for VirtualCenter 2.5 update 1 has a build number of 84767. Unlike the upgrade from VirtualCenter 2.0 to VirtualCenter 2.5, connections into VirtualCenter are not prompted to download and install the new version of the VIC. Therefore, you can have VIC installations with the older version interacting with the newer VirtualCenter server. I recommend that you enforce the version currency of the VIC on all systems connecting to a VirtualCenter 2.5 Update 1 to be at build 84767 so that commands match the capabilities of VirtualCenter. For most situations, this is not going to be an issue, but should you use VirtualCenter plug-ins, advanced features, or have a large amount of VIC traffic there may be some anomalies in the version mismatch.

The figure below shows two information screens from two installations of the VIC. Both of these installations are interfacing to the same VirtualCenter server, but the top one is at the current version:

During the VirtualCenter upgrade process, the VIC is updated locally on the server. Further, unlike the VIC for VirtualCenter 2.0 the VIC for VirtualCenter 2.5 and 2.5 Update 1 have the same binary update path, so you will not have to manage two shortcuts for access.

Upgrading VirtualCenter does not update certificate store

When logging into the VMware Infrastructure Client (VIC) or using the web interface to the VirtualCenter server, you are presented with a certificate message. Best practice or not, I usually accept the certificate and instruct the software to not ask me again about this topic. I have just completed end-to-end testing of VirtualCenter 2.5 update 1 and ESX 3.5 update 1, however, and noticed something about the certificate store for VirtualCenter.

The VirtualCenter certificate store is valid for two years from the date of initial installation. Since this instance of VirtualCenter was installed, I have upgraded to version 2.5 base release and, most recently, to the version 2.5 update 1. But neither installation updated the certificate. Below are the details of my test certificate:

The good news is that you now know about this issue. The bad news is that you better correct it before the two year anniversary of your installation of VirtualCenter as it is required to process logins. VMware has a comprehensive PDF that outlines the certificate procedures for VirtualCenter and the ESX hosts. The ESX hosts, however, have a much longer lifespan for the local certificate, around 20 years, and do not exhibit this behavior. The VMware server certificate documentation is available for download from the VMware website.

Changing the VMware Server 2.0 default permissions

I have been using VMware Server 2.0 (beta 2) on both Windows and Linux platforms for a while now. For Windows systems that are a member of an Active Directory domain, there are inherited permissions that may be assigned from Group Policy. If you want to change that, here are a couple of pointers in changing the security model.

To start looking at the permissions of the server installation, click the server on the left side of the browser view and then click the permissions tab at the top. The default permission is the local Administrators of the Windows system will be in the VMware Server Administrators role as shown below:

Before you make any modifications to the security model, add your desired configuration. That way, you can protect yourself from orphaning your administrative access to the server. Click the New Permission link in the command section, and add the user from either the local accounts of the Windows system or a user from the Active Directory domain or security group from the domain. The figure below shows the addition of the RWVVMwareAdmins group to the role of Administrator within the VMware Server 2.0 web interface:

Once that is added, the new configuration should be tested to ensure the proper access is available. Once the new access is verified, it would be safe to then remove the previous default access (if needed). If you get stuck, you can save off the .VMDK files and reinstall the product if needed.

While the web interface for VMware Server 2.0 takes some getting used to when compared to the thick client for versions 1.0x, features continue to be added to the free virtualization product that can be suited to test and development or live environments.

Adding a remote datastore to VMware Server 2.0 beta

I have been evaluating the VMware Server 2.0 beta on Windows and CentOS Linux systems since the release in November of 2007. The second beta of Server 2.0 was released in March, and both versions support adding remote datastores for guest virtual machines. Adding datastores can be a great way to store your lesser priority virtual machines without immense storage requirements on your VMware server system.

I have used the 1.0x version of VMware Server for years in testing and development systems. With version 2.0, I have started to use remote datastores for storing test systems. Datastores are storage locales in VMware Server 2.0. You can add NFS datastores for Linux-based installations of VMware Server. The Microsoft networking server message block (SMB) is available for Windows versions.

From a performance and configuration perspective, a non-local datastore is not ideal for live production systems. For situations like mine where a large number of infrequently used virtual machines are used from VMware server, a remote slower-speed disk suits the need very well. This storage configuration may also be good for archival of certain virtual machines, such as a project build that has gone into support mode.

In my example, I used a network attached storage (NAS) device, which can provide some of the cheapest storage available. NAS default configurations usually provide Windows file and print as a native file-serving option. Adding datastores from the “add datastore” command in the web interface after the initial installation is straightforward:

Once this is done, a virtual machine can be added to the local inventory. When virtual machines run from a remote datastore, the disk I/O is executed on the remote system. Adding a remote datastore also puts the CPU, network and memory functions on the host locally. There is definitely a performance hit from this configuration. But for archival purposes, you can free up more capable storage for your most frequently used virtual machines.

Administrators should also note that the virtual machines must be updated before being visible in 2.0 from a remote datastore before you make a remote datastore for all of your existing servers running VMware Server 1.0x from your VMware Server 2.0 beta system. Once updated to virtual machine version 6.0, the 1.0x systems cannot use the guest virtual machines without being upgraded to VMware Server 2.0.

The following section illustrates how to add a virtual machine to the inventory after upgrading:

This can save time by possibly eliminating the need to copy large virtual machine files back and forth over your network.

Use VI3 maps to visualize storage distribution

You may not have used the maps feature within the VMware Infrastructure Client because large environments become difficult to decipher. The maps view has become an important part of visualizing certain elements of the overall configuration of an ESX environment. One of the more useful mapping views that can help to illustrate relationships is the virtual machine (VM) to datastore map. This shows how many virtual machines are contained in each datastore. It will also show the virtual machines that have connections in multiple datastores.

To use the maps view to show the correlation between VMs and datastores, select the maps button from the main toolbar and then deselect all options except “VM to Datastore”. Then select the datastores, hosts, clusters or resource pools you wish to have represented in the map. It is best to construct your diagram based on the storage configuration. So, if your storage is per datacenter, diagram from the datacenter level down. Once you click the “Apply Relationships” button, a map is drawn based on all of the datastores. You can zoom out of the map by pressing the “-“ key or [ctrl] and scroll on a wheeled mouse. Below is a sample map of a datacenter and the VMs connected to each datastore:

In this example, there are three datastores without a VM assigned within the map. This is the local datastore on the ESX servers in this environment. As a result, it can be quickly determined if a VM is on the local disk. Depending on the environment, locally stored VMs may be prohibited if you use VMotion or certain VMware DRS configurations. This visibility presents an opportunity to see the variance from your standards. For example, lets say I build an environment of Windows Server 2003 VMs with 32 GB of storage assigned. In this example the shared storage resources are in increments of 320 GB, making it so that 9 VMs comfortably fit in each logical unit number which then becomes the datastore through the fiber channel interface, while leaving room for snapshots or .ISO files. If I see a datastore with a very large number of VMs attached, I can easily assume that they are very small. Or, in the situation where a VM is connected to multiple datastores, I can look into why that’s the case if it isn’t the norm. The most common example of a VM attached to multiple datastores is a CD-ROM image mapping to an .ISO image.

Why you should upgrade to VI3 Update 1

VMware Infrastructure 3 Update 1, made available on April 10 2008, introduces some core updates to ESX Server 3.5, VirtualCenter 2.5, and the VMware Infrastructure Management Installer. The biggest reason to upgrade, however, is the inclusion of Storage VMotion.

Among the core features now available with ESX 3.5 Update 1 are the addition of the Intel 82598 10 GB Ethernet controller, support of Jumbo Frames and NetQueue, additional Microsoft Clustering Services (MSCS) support, additional backup product and management agent support, additional guest operating systems, and additional server models.

I’ve been working with ESX 3.5 Update 1 for a few weeks, and the installation and behavior are indistinguishable from both ESX 3.5’s base release and ESX 3.02, with the exception of context sensitive tasks or options.

When I test upgrades, I make a point to test the upgrade in an environment with dissimilar ESX host server releases. For example, most of my hosts are ESX 3.02. When I upgrade the first one to ESX 3.5, I want to make sure that nothing goes wrong. I want to know that I’ll be able to sustain a mixed environment with all functionality. When I migrate running virtual machines through host-based VMotion to the ESX 3.5 host, and the reverse, I want to make sure to the best of my ability that nothing will fail. I also want to ensure that all of the VMware DRS and VMware High Availability rules are still enforceable with the mixed-host inventory.

Outlining a functionality matrix and the verification of the behavior is key to having no surprises during a live upgrade. Testing the update to VirtualCenter is a little more difficult but I am setting up a test environment soon to ensure that everything functions as expected in my environment. Overall, the fixes and new features make ESX 3.5 Update 1 an attractive upgrade for systems that are not there already.