So recently he published a “live blog” that chronicles his attempt to find a frustrating — but highly amusing (at least to me) problem: a VM that had been hot-migrated, but failed, so it was powered-off. But vCenter and other parts of the virtual infrastructure were stubbornly sure that the VM was, in fact, still on. Of course, although the VM was listed as powered-on, the options to control it were grayed out, so the poor Mike couldn’t power it off a second time.

Not to be defeated by the rogue, defiant VM that decided to vacation somewhere in vSphere land without his permission, Mike researched, found some possibly suitable solutions (and many not suitable at all), and finally, a lightbulb hit. Mike didn’t want to sacrifice the other VMs relying on their Mama VM (host VM). So why not send the VMs off to grandma’s (read: hot migrate them to another host) for a little vacation while Mama VM gets a reboot? Problem solved, albeit Mike had to bend his own rule of not “pulling the plug” on the host VM to fix a problem.

The full post is worth a read, especially if you’ve experienced your own virtual frustrations. Just grab a cup of coffee, since it takes a bit of time to read in its entirety.

Readers: If you’ve got your own VM troubleshooting story, email it to me. If I get enough submissions, I’ll record it in a podcast that you can later listen to and giggle along with.

CBT is a new feature in vSphere that can keep track of the blocks of a virtual disk that have changed since a certain point in time. This is extremely useful for backup and replication applications that can use this information to greatly improve incremental backup and replication times. Without CBT these applications have to figure out changed blocks on their own so being able to get this information for free using the vStorage advanced programming interfaces is extremely valuable to them.

CBT is not really part of the vStorage APIs but is a new feature of the VMkernel that is built into the storage stack. The CBT feature can be accessed by third-party applications as part of the vStorage APIs for Data Protection. Applications can use the API to query the VMkernel to return the blocks of data that have changed on a virtual disk since the last backup operation. You can use CBT on any type of virtual disk, thick or thin and on any datastore type except for physical mode Raw Device Mappings. This includes both NFS and iSCSI datastores.

Besides requiring vSphere, a prerequisite for using CBT is that a virtual machine must be using version 7 virtual hardware. While this is the default in vSphere (you can still choose the old version 4 hardware that was used in VMware Infrastructure 3, if you upgraded a host from ESX 3 to ESX 4 you must also upgrade the virtual hardware of the VMs to version 7 to use this feature.

The CBT feature is disabled by default; the reason for this is there is a very small bit of overhead that occurs when using it. However this overhead is a small price to pay for the great efficiencies that happen by enabling it. This feature is not global and can be enabled on only select VMs that you want to take advantage of this functionality. It can be enabled either through the vSphere client or by using the SDK. To enable it in the vSphere client you need to add a configuration parameter to each VM using the below steps:

1. Power off the VM. This is necessary to add a configuration parameter, edit the settings of the VM. Select the Options tab, then under Advanced, General click the Configuration Parameters button.

2. Next, click the Add Row button. You first need to add a general parameter for the VM to enable the feature and then add additional ones for each virtual disk that you wish to enable it on. For the general parameter enter “ctkEnabled” for the name and “true” for the value.

3. For each virtual disk you need to add “scsi#:#.ctkEnabled” for the name where the # signs should be replaced by the controller/disk number for each disk. Usually this is 0:0 for the first disk, then 0:1 for the second disk, etc. Also use “true” for the value for this parameter.

4. Click OK when you are done to save it.

An alternate method to enable CBT is using the SDK. Many backup applications that support CBT will automatically do this for you. For details on using the SDK method see this VMware tech note which describes how to use VirtualMachineConfigSpec and ReconfigVM_Task methods to accomplish this programmatically. Typically you will not want to enable this unless you have a specific application that can utilize this feature such as VMware Data Recovery or Veeam Backup & Replication.

Once enabled a VM must go through what is called a stun/unstun cycle for it to take effect. This cycle happens during certain VM operations including power on/off, suspend/resume, create/delete snapshot. During this cycle a VM’s disk are reopened which allows a change tracking filter to be inserted into the storage stack for that VM. You might wonder where CBT stores the information about changed blocks for a virtual disk, it does this in a special “-ctk.vmdk” file that is created in each VM’s home directory for each virtual disk that it is enabled on.

This size of this file is fixed and does not grow beyond its initial size unless you increase the size of a virtual disk. The size of this file will vary based on the size of a virtual disk which is approximately .5MB for every 10  GB of virtual disk size. Inside this file the state of each block is stored for tracking purposes using sequence numbers that can tell applications if a block has changed or not. One of these files will exist for each virtual disk that CBT is enabled on.

CBT is a great feature that really improves the efficiency and speed of virtual machines’ backup, restore and replication operations in vSphere. Several backup applications have already taken advantage of this new feature and are reporting greatly improved incremental backup times and being able to achieve near continuous data protection because of it.

Special thanks to John Troyer and Jon Bock from VMware and Anton Gostev from Veeam for taking the time to help me better understand the vStorage APIs and CBT.

This release is significant in the fact that is the first update to vSphere which should appease all those who were waiting to upgrade to vSphere for fear of bugs and issues that can be present in the first major release of any software application. There are at least 50 patches that have been issued since vSphere was released that have been rolled into this update, which should lead to increased stability and less headaches for administrators.

This release includes updates to ESX, ESXi, vCenter Server as well as to VMware Data Recovery which has been updated from version 1.0.2 to 1.1. Enhancements to ESX(i) and vCenter Server include the following:

• VMware View 4.0 support - This release adds support for VMware View 4.0, a solution built specifically for delivering desktops as a managed service from the protocol to the platform.

• Windows 7 and Windows 2008 R2 support - This release adds support for 32-bit and 64-bit versions of Windows 7 as well as 64-bit Windows 2008 R2 as guest OS platforms. In addition, the vSphere Client is now supported and can be installed on a Windows 7 platform.

• Enhanced Clustering Support for Microsoft Windows - Microsoft Cluster Server (MSCS) for Windows 2000 and 2003 and Windows Server 2008 Failover Clustering is now supported on a VMware High Availability (HA) and Dynamic Resource Scheduler (DRS) cluster in a limited configuration. HA and DRS functionality can be effectively disabled for individual MSCS virtual machines as opposed to disabling HA and DRS on the entire ESX/ESXi host.

• Enhanced VMware Paravirtualized SCSI Support - Support for boot disk devices attached to a Paravirtualized SCSI ( PVSCSI) adapter has been added for Windows 2003 and 2008 guest operating systems. Floppy disk images are also available containing the driver for use during the Windows installation by selecting F6 to install additional drivers during setup. Floppy images can be found in the /vmimages/floppies/ folder.

• Improved vNetwork Distributed Switch Performance - Several performance and usability issues have been resolved resulting in the following:  Improved performance when making configuration changes to a vNetwork Distributed Switch (vDS) instance when the ESX/ESXi host is under a heavy load. Improved performance when adding or removing an ESX/ESXi host to or from a vDS instance.

• Increase in vCPU per Core Limit - The limit on vCPUs per core has been increased from 20 to 25. This change raises the supported limit only. It does not include any additional performance optimizations. Raising the limit allows users more flexibility to configure systems based on specific workloads and to get the most advantage from increasingly faster processors. The achievable number of vCPUs per core depends on the workload and specifics of the hardware.

• Enablement of Intel Xeon Processor 3400 Series – Support for the Xeon processor 3400 series has been added.

VMware Data Recovery has the following enhancements, most notably better support for file level restores of Windows VMs. Previously in VDR 1.0, file level restore was considered experimental and was done using a command line utility. In VDR 1.1 there is a new Windows file-level restore client to make restores much easier using a GUI interface instead. You can read more about the new FLR capabilities in the VDR 1.1 Administration Guide.

• File Level Restore Functionality is Officially Supported – File Level Restore (FLR) provides a way to access individual files within restore points for Windows virtual machines. In previous versions of Data Recovery, FLR was provided as an experimental feature. File Level Restore feature is now officially supported.

• Integrity Check Stability and Performance Improved – The integrity check process is faster and more stable. Note that integrity checks are computationally intensive processes and can take significant periods of time. The exact amount of time integrity checks take varies based on of the size of the deduplication store. Even with these enhancements, integrity checks that take several hours are not unexpected.

• Integrity Checks Provides Improved Progress Information – When an integrity check is running, a progress indicator is displayed. This progress indicator has been improved, although it does not provide the optimal level of detail.

• Enhanced CIFS Shares Support

As usual read the release notes before upgrading and make sure you follow the standard upgrade order of: vCenter Server>Hosts>Virtual Machines.

ESXi 4.0 Update 1 Release Notes

ESX 4.0 Update 1 Release Notes

vCenter Server Update 1 Release Notes

VMware Data Recovery 1.1 Release Notes

Improved Storage Resources Control

As the scope of storage resources have increased significantly with large deployments of virtualization environments, so has the need for greater automation and control of these resources. In the vSphere release, vCenter has been enhanced with several new storage specific capabilities to help the virtual administrator manage these environments with a higher degree of control. These enhancements provide administrators with proactive alerts and alarms to address issues before they interrupt the availability of applications running on those resources. vCenter allows setting permissions and quota limits on datastores, as well as per VM.

His question was to explain the meaning of the following line: “vCenter allows setting permissions and quota limits on datastores, as well as per VM.” In particular he wanted to know about setting quota limits on data stores and VMs. While I know a lot about the new permissions in vSphere, particularly for data stores, the part about quota limits confused me also because I have never heard of quota limits in vSphere. I did some research, checked all the vSphere documentation and couldn’t find anything about them. So I contacted VMware to get some clarification and I received the following response from the white paper author:

The quota limit applies to the storage given to a VM.  There is not really an equivalent for a data store.  One can set alarms to notify/alert one when a certain percent-full or overcommit is hit. But I am not aware of a means to stop allocations or placement of VMs when a certain percent overcommit is hit.  That is a common request that I do not believe is there now.

So basically “quota limits” simply  means the amount of disk space assigned to a VM. I guess technically it’s a quota but once you assign the space to the VM you can’t control how much of it that it can use. If you create a 20 GB virtual disk for a VM the guest operating system is going to see all 20 GB of it regardless of whether it is a thick disk or thin disk. If you do overcommit your data stores by using thin disks you need to carefully monitor datastore free space using alarms and reporting. Currently in vSphere there is no way to set any type of quotas for snapshots, virtual disks or datastores.

So while alarms and permissions are greatly improved in vSphere there are really no quota limits that you can use. Perhaps in a future release VMware will put some type of quota controls in place. Thanks to John Troyer and Paul Manning from VMware for the clarification on this.

After doing some research I found out they were part of the new vApps feature in vSphere. I’ve heard a little about vApps but never looked at them in depth, so I thought I would take the time to research them and write about them.

We’ll come back to IP Pools in a bit. First we’ll cover what a vApp is and how they work in vSphere. VMware’s definition of a vApp is below:

A logical entity comprising one or more virtual machines, which uses the industry standard Open Virtualization Format to specify and encapsulate all components of a multi-tier application as well as the operational policies and service levels associated with it.

A vApp is basically a resource container for multiple virtual machines that work together as part of a multi-tier application.

An example of a multi-tier application is a typical Web-based application where you might have three tiers: Web, application and database; which are often run on three separate servers. For example, you may have Microsoft IIS running on one server (tier 1), IBM WebSphere running on another server (tier 2) and a Microsoft SQL Server running on a third server (tier 3).

The three applications on each server all work together and are mostly dependent on each other for the application to function properly. If one part of the tier became unavailable, the application will typically quit working as it relies on all the tiers for the application to work.

Virtualization can introduce some challenges with multi-tier applications. For example, if one tier is performing poorly due to resource constraints on a host, then the whole application will suffer as a result. Another challenge comes when powering on a host server, as often times one tier relies on another tier to be started first or the application will fail.

VMware introduced vApps as a method to deal with these problems by providing methods for setting power-on options, IP address allocation and resource allocation, and provide application-level customization for all the virtual machines in the vApp. When you configure a vApp in vSphere you specify properties for it, including CPU and memory resources, IP allocation, application information, and start order, as shown below.

Once you are done configuring a vApp, you can add virtual machines (VMs) to it by dragging them using the vSphere client. You can also create resource pools inside of them and nest vApps inside of vApps. If you edit the settings of a VM, select the Options tab, and then select vApp Options, you can enable the vApp functionality for the VM and set individual vApp options for the VM. Once you have created a vApp you can easily export it in Open Virtualization Format (OVF) format, as well as deploy new vApps from one that are already created. To use vApps you must have a Distributed Resource Scheduler-enabled cluster; all of the meta-data information for a vApp is stored in the vCenter Server database.

So now that you know what a vApp is, back to what IP Pools are. IP Pools, as you might have guessed, are pools of IP addresses that you can associate with vSwitch port groups. They essentially act as Dynamic Host Configuration Protocol (DHCP) servers to assign IP addresses from the pool to a VM, so essentially the vCenter Server is acting as a DHCP server. When you configure an IP Pool you specify a range of either IPv4 or IPv6 addresses, DNS and proxy settings, and finally select which vSwitch port groups that the pool will be available to.

To configure an IP address range you need to do it in the following format with the starting IP address, a pound sign and the number of IP addresses, like this:

172.20.20.155#10, 172.20.20.175#5

So the above range would make the following IP addresses available in the IP Pool:

172.20.20.155 - 172.20.20.164 and 172.20.20.175 - 172.20.20.179

Once you have an IP Pool configured, you can assign it to a vApp by editing its properties and changing the IP Allocation Policy from Fixed to Transient. For more information on configuring and using vApps and IP Pools check out the resources below.

• VMware vApp Developer Blog
• vApps and advanced VM templates in vSphere 4 (VMworld 2009 attendees or subscribers only)
• vSphere Basic System Administration (Chapter 10)

PowerCLI has a lot of functionality built in. As of this writing, there are 165 cmdlets that let you do a wide range of tasks such as starting virtual machines or creating virtual switches. Cmdlets are great because they are high-level, task-based, and their usage is mostly consistent across all domains, whether you are talking about virtualization or managing your mail servers.

Here’s a PowerShell command which will display all of the various types of objects which you can manipulate with PowerCLI. As you can see from the output, the scope is pretty broad:

PS > Get-VICommand | sort noun | group noun | Format-Wide name

CDDrive Cluster

CustomField Datacenter

Datastore DrsRule

FloppyDrive Folder

HardDisk Inventory

Log LogType

NetworkAdapter OSCustomizationSpec

ResourcePool ScsiLun

ScsiLunPath Snapshot

Stat StatInterval

StatType Task

Template Tools

VICredentialStoreItem VIEvent

View VIObjectByVIView

VirtualPortGroup VirtualSwitch

VIServer VIToolkitConfiguration

VIToolkitVersion VM

VMGuest VMHost

VMHostAccount VMHostAdvancedConfiguration

VMHostDiagnosticPartition VMHostFirewallDefaultPolicy

VMHostFirewallException VMHostFirmware

VMHostModule VMHostNetwork

VMHostNetworkAdapter VMHostNtpServer

VMHostProfile VMHostProfileCompliance

VMHostService VMHostSnmp

VMHostStartPolicy VMHostStorage

VMHostSysLogServer VMResourceConfiguration

VMScript VMStartPolicy

Even so, the vSphere application programming interface (API) is so large that there are still an astonishing number of things which are not handled by the existing set of cmdlets. Sure, VMware continues to work on this toolkit and to expand its capabilities — the rumor is that over 200 cmdlets will ship with version 4.1 — but the team that works on PowerShell is limited by time and resources, so it’s doubtful that they’ll reach parity with the API anytime soon.

Luckily, there is a release valve — a hidden door, if you like, that will let you go behind the scenes to go where no VMware Infrastructure (oops, meant to say vSphere) admin has gone before. The name of that hidden door is the Get-View cmdlet. “Get” in the PowerShell world just means to retrieve an object from somewhere. But what is it really that you are getting? You are getting an unfiltered view of the “managed objects” which make up the heart of the API. Once you have that view, you can do anything you want which the underlying API supports.

Here is a quick example of the sort of cool things you can see with Get-View:

PS > $h = Get-VMHost | Get-View

PS > $h.Summary.Config.Product

Name : VMware ESX

FullName : VMware ESX 4.0.0 build-164009

Vendor : VMware, Inc.

Version : 4.0.0

Build : 164009

LocaleVersion : INTL

LocaleBuild : 000

OsType : vmnix-x86

ProductLineId : esx

ApiType : HostAgent

ApiVersion : 4.0

InstanceUuid :

LicenseProductName : VMware ESX Server

LicenseProductVersion : 4.0

DynamicType :

DynamicProperty :

And now you know how to write a script to make sure your host servers are all on the same patch level!

Perhaps you have also noticed as well that budget constraints are causing us to rethink how we get training on the latest technologies. It is not always possible to take a week off work to take a class each time a new OS version is released, and additional weeks to receive instruction on all the individual components. Now it is expected that we somehow become proficient in dozens of applications from a score of vendors and implement the latest technologies while continuing to support legacy systems and contribute to the IT community.

This spring VMware announced the GA release of vSphere. Hey, something new and exciting to learn! This will be great – I can take a class, spend my evenings getting familiar with the product, test it in the lab, and come up with a business case for it with my employer. However, scheduling a week for a systematic training class in the immediate future seems less likely than discovering the remains of Atlantis in my backyard.

To this end, I needed to rethink my training on the latest VMware offering. As my mind drifted back to my first real IT training (IPX/SPX was king of the workgroup at the time) it occured to me to try some self-paced training once again. Certainly things have changed since the days of 10 MB networks and hot new technologies like PointCast, and hopefully advances in training have kept up.

Enter TrainSignal. Having benefitted from David Davis’ insightful tech articles in the past, seeing his name as the instructor on the vSphere DVD-based program instantly gave me a positive feeling about heading down this distantly familiar route of ordering up a self-paced video training course.

Within the hour, my VMware vSphere Training was on its way. A few days later, a smallish-looking box arrived. Upon opening it, I was a bit surprised – the entire program was housed in a nice and tidy box with fold-out DVD holders. Compared to the size of my first IT training program (20+ VHS tapes, plastic bags full of floppy disks, plus several books measured more in pounds than pages), this seemed so… small. Could it really be enough?

Not to be put off, I popped the first disk in my PC and was presented with a clean-looking interface and instructions on how to copy various lessons to my iPod Touch. Hey, now this is what I was looking for! Being able to watch, listen and learn while burning calories on the elliptical machine and stationary bike at the gym during lunch is much more in line with the realities of my life (both in IT and personally) as compared to countless hours spent in the easy chair with a pink highlighter and a cup of tea.

Granted, VMware has not relaxed its requirement to take an official class to become VCP-certified, so I will still need to find the time to enroll in an official VMware class at some point in the near future. However, I believe that the TrainSignal program will certainly help with becoming more familiar with the intricacies of the product, as well as prepare me to get the most from the class when the time comes to take it. As I learn more about practical ways (the good, bad and ugly) to use vSphere in the enterprise, I look forward to sharing them with you on SearchVMware.com.

Yes, times have changed since the 90′s. Yet the lessons we have learned on how to get by with less and using our time to the fullest will always be valid, especially as we work to keep our skills and knowledge current with VMware’s continually changing products and technologies.

Cheers!

Mak King

In this post that the most frequent frustration with VMFS is rooted in difficulty working with large files. I’m not going to disagree that working with large files is no fun, but will add that it is not fun in Windows either. Sure, Hyper-V allows migration of .VHD files to USB disks easily. I find that working with virtual disk files (.VMDK or .VHD) is cumbersome regardless of platform and avoid the practice in favor of other mechanisms. There are networking-based solutions to exchange virtual machines (VMs) and if a number of VMs need to be transported, a more robust storage solution or transient host may make more sense.

Truth be told, the use of USB and optical media is possible with VMFS – but not exposed in the current implementation of VMFS released to ESX and ESXi hosts. Will it make its way to the mainstream product? I’m not sure. Then the next question will be for other filesystems such as ext3, ext4, reiserfs, xfs, jfs or zfs. Every feature added onto purpose-built code introduces more complexity to the thin footprint, and is complicates the updating and vulnerability situation.

Long live VMFS! Have a comment? Chime in below.

My guess is that VMware does not see creating a Linux version of the VI Client as a priority as the majority of their customers have predominantly Windows environments. But while a Linux client may be a nice-to-have for most customers, it is a must-have for those customers that have predominantly Linux environments. To accomplish this, though, VMware would have to rewrite its client using another programming language because the current client uses the Microsoft .NET framework, which is not natively compatible with Linux systems. By not providing client support for more OSes than Windows VMware is potentially losing customers that do not run Windows and have no desire to run Windows just to use a management client for their virtualization environment. If VMware wishes to further its journey into cloud computing (which is typically predominantly Linux), it may want to rethink this strategy and shift its development to a multi-platform client.

VMware could ditch the software client completely and use a Web-based client instead. VMware currently has a Web-based user interface (UI) for accessing vCenter Server and ESX hosts but it is limited to only managing virtual machines and not vCenter or host servers. Additionally, ESXi — which, according to VMware, will eventually replace ESX — does not support the Web-based UI as it does not have a Web application server installed. Using a Web-based UI is an attractive option as it is OS-independent and anyone with a supported browser could access it regardless of the OS they are running.

So what will VMware do? It appears that VMware is firmly committed to using the vSphere client as evidenced by the many plug-ins VMware is developing. It’s unlikely that VMware will port the vSphere client to other operating systems like Linux or Macintosh and it is also unlikely that VMware will enhance the Web UI to provide the functionality that the vSphere client provides. Having a multi-platform or OS-independent client would benefit all customers; even predominantly Windows shops have some Linux or Mac systems. VMware may surprise us at VMworld with a client support announcement like it did last year, but users will want to see an actual product this time.

This only affects ESX hosts (VMware vCenter Server has this enabled by default; ESXi does not have a Web access user interface (UI) to manage the host and virtual machines).

There is a tech note that describes the process for enabling this feature in ESX 4.0, but before you go ahead and do this you should ask yourself if you really need this feature enabled on all your hosts. VMware disabled this feature for a reason — Web based access methods are inherently insecure and are subject to numerous vulnerabilities that could potentially compromise your VMs and hosts.

The Web admin UI is very limited as you can only administer guest VMs and not host servers. Leaving it disabled removes a potential attack vector for your ESX hosts and makes them more secure. This is also true of other configuration settings that are disabled by default in ESX such as root access using SSH.

The first thing many administrators do is enable this because it is easier than setting up another user account and using su or sudo. So resist the urge to enable web access and utilize the vSphere client instead. If you must use web access for a specific reason only enable it on the hosts that need it. If you are using a vCenter Server use the web access on vCenter instead and use the roles and permissions built into it for additional security. By leaving vSphere web access disabled you are helping to make your ESX hosts and your whole virtual environment more secure.