Why is this? When a snapshot is created, the original disk becomes read-only, and a separate delta file is created that contains all the disk changes that are made thereafter. The delta file does not contain an ongoing history or transaction log of all the changes to data on the disk, it simply updates disk blocks as they are changed. If a particular block is changed it is written to the delta file, but if that same block is changed again later on the existing block is simply updated with the new data and a new block is not written to the delta file.

For example, if you took a snapshot of a VM with a 10 GB virtual disk, that snapshot could never grow larger than 10 GB, although it might grow slightly larger if every single disk block was changed because of the extra overhead space included in the snapshot disk file. The initial snapshot starts out small (16 MB) and grows in 16 MB increments up to the maximum size of the original virtual disk as changes are made to it.

In most cases the snapshot will not grow as large as the original disk, because typically operating system and application files are not changed once they are installed and therefore those disk blocks are not changed. If you performed a disk defragment inside the operating system, however, this could quickly and easily grow the size of the snapshot as files are being moved around on the disk which results in them being rewritten in a new location and, subsequently, the disk blocks are updated accordingly.

Now this only applies to a single snapshot. It is possible for the combined disk space total of multiple snapshots to exceed the size of the original disk file. The reason for this is that previous snapshots become read-only when new ones are created. If a particular disk block was updated from a previous snapshot, it would be written as a new block in later snapshots. That same disk block could then exist in multiple snapshots, which could make the combined total of the snapshots greater then the original disk file.

Even though copies of a single disk block can exist in multiple snapshots when it comes time to delete the snapshots, only the latest disk block is written back to the original disk and all of the others are discarded. Likewise, if you revert to a particular snapshot, then the existing disk block is discarded if it has been updated since the snapshot that you are reverting to, and the disk block from that snapshot is used instead.

This may all sound a bit confusing but the moral of this story is that a single snapshot can never exceed the size of the original disk file. For more information on snapshots be sure and check out the three-part series that I wrote about them:

How VMware snapshots work (Pt. 1)
Deleting virtual machine snapshots without wasting disk space (Pt. 2)
Troubleshooting VMware snapshots (Pt. 3)

While this was a good time to change the name for VDI to View with the release of the new version (View 3) it was expected that the vCenter name change would not take place until the release of VI4. Subsequently this may lead to confusion for a while as the vCenter name is effective immediately for the current version of the product (VirtualCenter 2.5 Update 3). VMware has updated their website accordingly for all the product download and documentation pages but the name has not been changed in the current documentation or product. Whether they apply the name change inside the application and documentation in the next release of vCenter (presumably 2.5 Update 4) or wait until VI4 is released is not currently known. Additionally this only applies to the current versions which presumably means any version of vCenter 2.5, version 2.0.x is still referred to as VirtualCenter.

All of the product name changes affected by this are listed below:

VMware VirtualCenter → VMware vCenter Server
VMware Lifecycle Manager → VMware vCenter Lifecycle Manager
VMware Converter → VMware vCenter Converter
VMware Lab Manager → VMware vCenter Lab Manager
VMware Stage Manager → VMware vCenter Stage Manager
VMware Update Manager → VMware vCenter Update Manager
VMware Site Recovery Manager → VMware vCenter Site Recovery Manager
VirtualCenter Foundation → vCenter Server Foundation
VMFS → VMware vStorage VMFS
VMware Virtual Desktop Infrastructure → VMware View
Virtual Desktop Manager (VDM) → VMware View Manager
VMware Administrator Interface → VMware View Administrator
VDM Agent → VMware View Manager Agent
VDM Web Access → VMware View Portal
VDM Client for Windows → VMware View Client for Windows
VDM Client for Linux → VMware View Client for Linux

If you look in these directories the contents of them are exactly the same as the directories that are named the same as your VMFS volumes. So what are these directories? Let’s explain what happens when you create a VMFS volume to find out.

When creating VMFS volumes you are prompted to name them. This name is not what the ESX host uses to reference the volume; it is purely a friendly name to make it easier for the user to identify the volume. The ESX host actually uses a unique identifier called a Universal Unique ID (UUID) to reference the volume. The name you specify when you create a VMFS volume is a user-defined device name which is a symbolic link to the UUID of the VMFS volume. This is done to solve the problem of changing the device name, when you change the volume name you are only changing the user-defined device name and not the UUID of the volume. So when you look in your /vmfs/volumes directory you will see both a UUID, ie. 4404e8b4-bcfd52fc-1e4b-0017a4a91076 and the symbolic link, i.e. ServerA-Local. Changing to the symbolic link name by using the cd command or clicking on it in WinSCP simply takes you to the UUID directory. You can see the relationship between symbolic links and UUID’s by using the ls –l command inside the service console as shown below.

Additionally you can see the UUID of a volume in the VMware Infrastructure Client by selecting a volume in the Configuration, Storage section and then looking in the Details pane at the Location field. It’s definitely a lot easier to remember the volumes friendly name then it’s long UUID which is why symbolic links are used with ESX.

P2V works by imaging the physical host within the DMZ and transferring that image to the administrative/management network attached to the service console (management appliance) of the VMware ESX(i) host. This in essence crosses security zones and could connect the hostile DMZ to the ‘in need of protection’ virtualization management network. Access to this network from the DMZ could be disastrous.

One solution is to perform the P2V migration in stages.

1. Create the DMZ virtual network within your virtual infrastructure.
2. Get your security team to bless a laptop/workstation for work within the DMZ. Ensure this laptop/workstation has enough removable storage to contain the resultant VM or VMs of the physical servers you wish to convert.Use your  P2V tool to convert the VM and store it on the removable media.
3. Disconnect the removable media and bring it to your secure administrative network.
4. Connect the removable media to a workstation within the administrative network. Ensure this connection is read-only for the moment if possible.
5. Virus Scan the removable media, but note a VMDK can give false positives; you are really looking for anything that may be hidden from view.
6. Use VMware Converter to import the VM or VMs into the virtual infrastructure ensuring they are connected to the proper virtual network.
7. Power on the VM with the network disconnected and fix any issues that are caused by the P2V migration, such as the need to remove hardware agents, and fix anything that needs to be fixed.
8. Reboot the VM with the network connected

The P2V migration is now complete and isolated from the network. The key to this is to only power on the VM once you are within a safe environment and to check for viruses and worms that may live within your DMZ.

1. Create a new shortcut on the desktop of the PC that you want to setup single sign-on for VirtualCenter.

2. In the Create Shortcut Wizard, click Browse and navigate to the location of the VpxClient.exe program and click OK. (By default it is located in C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\)

3. After the full path is in the Location field append
-passthroughAuth -s <VirtualCenter Server hostname> to the end of the line, where <VirtualCenter Server hostname> is the hostname or IP Address of the VirtualCenter instance you want to connect to.

4. Click Next and give a name for the shortcut and then click Finish. Once you double-click on the newly created shortcut you will be logged into VirtualCenter using your currently logged-in Windows credentials.

If you choose to use this convenient feature, make sure you take precautions to prevent someone from accessing an un-locked workstation and connecting to VirtualCenter using your credentials.

If you log into shared workstations make sure you log out when done. If you do not all someone has to do is open a browser and access the default page on any ESX or VirtualCenter server to download and install the VMware nfrastructure Client on that workstation and they can log in to VirtualCenter as you and access anything you have access to.

Anytime you leave your workstation make sure you lock it, you can make a shortcut for doing this in one mouse click or use the Windows and L keystroke combination. Also make sure you have your workstation set to automatically lock at a minimum of 15 minutes of idle time as protection in case you forgot to lock it when you walk away.

For more information on this feature including hot to change the default Security Support Provider Interface (SSPI) that is used see VMware KB article #1006611.

VMware’s Knowledge Base

First there is VMware’s online knowledge base. It continues to amaze me about the volume and quality of information provided in VMware’s knowledge base. Typically many users access vendor knowledge bases as a tool to try and resolve problems in their environment as many known problems are documented there. VMware’s knowledgebase, however, has much more than that and also contains how-to articles, troubleshooting tips, sample configurations, best practices and so on. Here’s just a sample of some of the great content published there in the last 10 days:

• Configuring the speed and duplex of an ESX Server host network adapter
• Cisco Discovery Protocol (CDP) network information via command line and VirtualCenter on an ESX host
• Sample Configuration – ESX connecting to physical switch via VLAN access mode. External Switch VLAN Tagging (EST Mode)
• Troubleshooting SCSI Reservation failures on Virtual Infrastructure 3 and 3.5
• Advanced Configuration options for VMware High Availability
• Advisory for advanced VMkernel parameter NFS.LockDisabled
• Sample Configuration – Network Load Balancing (NLB) Multicast mode over routed subnet – Cisco Switch Static ARP Configuration
• VLAN Configuration on Virtual Switch, Physical Switch, and Virtual Machines – ESX 3.X
• Diagnosing an ESX Server that is Disconnected or Not Responding in VirtualCenter

They even have a knowledge base article on how to search the knowledge base. I encourage you to frequently check out the knowledge base as new documents are added daily. You can also subscribe to a weekly digest by editing your VMware account preferences and setting up your email subscriptions.

VI:OPS

Next there is a new website VMware launched called VI:OPS that contains many documented proven practices from both VMware itself and customers. The information is organized into several different zones: Strategy, Applications, Security, Management and Availability. Already there are dozens of great documents there that provide some fabulous information.

Another great resource is the fabulous technical and white papers that VMware publishes. There are over 200 of them available on a wide variety of subject matters.

There is a wealth of information available on the VMworld website from the all the great sessions each year. Only attendees can access the current year’s sessions but anyone can register for a free account and access previous years.

VMTN

You may have used the VMTN community forums for posting technical questions but did you know there is a whole separate documents section that users can create technical documents to share their information and tips with other users. There are hundreds of documents already created there with some great tips from both other users and VMware employees.

In addition to all these great VMware resources be sure and check out all the great tips and information on both SearchServerVirtualization.com and SearchVMware.com.

IHA provides healthcare services to 750,000-pooohlus residents of British Columbia through a network of 183 hospitals and offices across the southeastern portion of the province.

IHA found itself adding an astounding 10 physical servers each week to keep pace with business demands over the past few years and the space requirements and costs made that pace unsustainable. IHA brought in VMware to gain control over its IT environment by reversing the physical server sprawl and providing a more efficient way to manage critical applications and data stores, VMware reported.

Kris Jmaeff, information system security specialist, IHA, stated in the release, “We wanted to get handle on our hardware requirements and, just as importantly, we wanted an application environment that could scale reliably…Over the long run, it should deliver millions in cost savings by slashing server procurement dramatically.”

By substituting VMware virtual machines (VM) for physical servers, IHA will avoid purchasing another 200 physical servers. The health authority is now running about 250 VMs in two datacenters that are fully redundant for disaster recovery, and all the VMs are managed centrally via VMware VirtualCenter.

About 95% of the virtualized applications are Windows-based, including Microsoft Exchange, SharePoint and SQL Server. They also virtualize Oracle databases and various custom applications for billing, scheduling and patient care.

The side effect of IHA virtualizing is that the organization is greener; the amount of power required to run and cool IHA’s data centers has been cut by nearly 85% using virtualization. Not only does this reduce carbon emissions by millions of tons, it has generated an annual power savings of over $70,000 for IHA, according to VMware.

Jmaeff said in the statement that IHA looked at other virtualization platforms, including Microsoft Hyper-V before choosing VMware. “VMware[s] platform could provide simplified and centralized management for all our VMs as well as the high availability, automation, and performance that we needed. And our decision has paid off with massive savings and big advances in IT resiliency. Now that we’ve abstracted the applications from the commodity hardware, we can relocate a VM in seconds if a box breaks. Users aren’t impacted. That’s invaluable in a healthcare environment.”

It’s obvious that VMware publicizes use cases like these, from hospitals where sensitive citizen data is kept and life saving technology is used, to dispell fears about virtualization performance and security. I think by now, in 2008, doubts about the performance of virtualization have been put to rest, though it seems security concerns still exist.

• Permissions Can Be Configured for Individual Virtual Machines and Resource Pools in VI Client – Starting with this release, when the VMware Infrastructure Client is connected directly to a host running ESX Server 3.5 or higher, the Permissions tab is available for individual virtual machines and resource pools.

The first notable fix is for displaying a login password in clear-text. You may not think this is too big of a deal, but if someone were to be standing near you when this happened they could see your password (or at least the one you entered) displayed in clear-text.

• VMware VirtualCenter password might display in error window – This release resolves an issue where a user’s password can be displayed in clear text on the login screen. When logging into VirtualCenter Server 2.0 with Virtual Infrastructure Client 2.5, the user password might be displayed in a dialog box on the VI Client in clear text if the login fails. The dialog box alerting the user to the failed login might be hidden under other windows.

The next fix updates the JRE used on the VirtualCenter server to the latest version which is 5.0 Update 16 (1.5.0_16) which fixes some security issues.

• WebAccess component JRE updated to version 1.5.0_16 – The currently installed version of JRE depends on your patch deployment history. For more information about security issues fixed in version 1.5.0_16 and in earlier versions, see the JRE release notes at http://java.sun.com/j2se/1.5.0/ReleaseNotes.html.

An update to the Flex License Manager server is included. This update is not installed automatically when upgrading existing installations and must be installed separately.

• FLEX license server upgrade – This release of VirtualCenter upgrades the FLEX license server to version 10.8.6 in order to resolve known issues with previous releases of the license server and to provide enhance debugging capabilities. The new FLEX license-server is packaged with the VMware Infrastructure Management Installer. Fresh installations of the license server will use the new version but the license server will not be automatically upgraded when using the VMware Infrastructure Management Installer to upgrade an existing installation. To upgrade an existing license server installation, use the standalone installer (VMware-licenseserver.exe) that can be found in the /vpx folder of the installer directory structure.

This one caused some issues with HA because of a network compliance check that was introduced in Update 2. A new HA advanced setting has been added to bypass this check.

• HA network compliance check – During the configuration of HA in VirtualCenter 2.5 Update 2, the Task & Events tabs might display the following error message and recommendation: HA agent on in cluster in has an error Incompatible HA Network: Consider using the Advanced Cluster Settings das.allowNetwork to control network usage. Starting with VirtualCenter 2.5 Update 2, HA has an enhanced network compliance check to increase cluster reliability. This enhanced network compliance check helps to ensure correct cluster-wide heartbeat network paths. VirtualCenter 2.5 Update 3 allows you to bypass this check to prevent HA configuration problems. To bypass the check, add das.bypassNetworkVerification=yes to the HA advanced settings.

In addition to the fixes and updates to the Update Manager and Converter, plug-ins have been released. If you upgrade to Update 3 you must also update these plug-ins or they will no longer work.

• Plug-ins Updates – Previous versions of VMware Update Manager are not compatible with VirtualCenter 2.5 Update 3 and must be upgraded to VMware Update Manager 1.0 Update 3 during the installation process. This release of the VMware Infrastructure 3 software suite also includes the following: • An update to VMware Converter Enterprise. For more information, see the VMware Converter Enterprise Update 3 for VirtualCenter 2.5 Release Notes. • An update to VMware Update Manager. For more information, see the VMware Update Manager 1.0 Update 3 for VirtualCenter 2.5 Release Notes.

This one has caused a few people who upgraded to Update 2 some grief — thankfully VMware has quickly addressed it.

• In HA-DRS cluster, the enter maintenance mode task stalls and VMs do not migrate – In previous releases of VirtualCenter, virtual machines might not be automatically migrated off a host entering maintenance mode if there is not enough failover capacity in an HA-DRS cluster. The enter maintenance mode task stalls at 2% indefinitely and does not complete even if HA admission control was disabled. In this release, the issue has been resolved by allowing the evacuations if HA admission control is disabled. Note that admission control is enabled by default, see Implications of enabling or disabling VMware HA strict admission control when using DRS and VMware DPM (KB 1007006) for more information. Note that the task might also stall if the virtual machines cannot be evacuated for other reasons. And finally a minor one with not being able to delete HA advanced settings.  

• Once HA advanced settings are created, they cannot be deleted – In previous releases advanced settings created on the Advanced Options page for an HA cluster cannot be deleted. Attempts to delete the advanced setting would result in an object reference error dialog box being displayed. This release resolves the issue and advanced settings can be deleted normally.

As always read the release notes before upgrading. This release may not have all the cool new features of the previous release but bug fixes, while not glamorous, are a necessary evil to ensure a stable product.

Certificate management is not one of my areas of expertise. With that, I’m posting this series of blog posts with the hopes that it will help other admins complete this mundane task seamlessly. I am currently faced with upgrading the certificate configuration for my VirtualCenter 2.5 Update 2 system. The VirtualCenter default certificate is made up of three files: rui.cert, rui.key and rui.pfx. All are located in the C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL folder for default installations. Now these certificates are SSL certificates, or web certificates that manage the communication between elements of VI3. This includes ESX hosts, VMware Infrastructure Client connections, the database and VirtualCenter Server connections.

In the earlier post, I mentioned VMware’s PDF as a good starting point for the certificate renewal process. The PDF explains a lot of different things, but leaves a few key areas out about how to fix the immediate problem. Luckily, I came across a very handy blog that simplifies things and gave me easy steps to follow. Leo Raikhman’s Ramblings blog gives direct guidance for those of us who are a little fuzzy with certificates. Leo points out in three separate blogs a basic way to address the problem, a 1-2-3 approach, and some in-depth explanation for an occasional zero-length pfx file causing generation issues.

I have successfully followed Leo’s materials for a seamless upgrade in an isolated test environment with no workload. Soon, I will repeat the drill on the live environment and let you know how it goes in another blog post.

Speaking of virtualization case studies, I wish other software companies were as forthcoming about their customers. Most software companies send out product press releases that are riddled with marketing speak and not enough meat, with no customers to back up claims like “number one provider” of this or “the world’s best” that. Even Microsoft Corp. wouldn’t give me a Hyper-V customer to chat with after their virtualization product was released on the commercial market this summer. To be fair, however, there are virtualization companies that do offer up customers like Massachusetts-based Virtual Iron. But I digress.

Since I have a few VMware case studies from September cramming my Inbox, here they are in summary.

Accountants Inc.
On September 30, VMware announced that Accountants Inc., an accounting and finance department staffing agency, deployed VMware’s virtualization and management suite VMware Infrastructure 3 to lower its overhead, meet its disaster-recovery requirements, and make its data center more flexible.

Accountants Inc. had been using a single physical server per application, and as the company approached a hardware refresh cycle, it found that virtualization would be more cost-effective, VMware reported.

Donald Wong, manager of IT operations and development at Accountants, Inc., stated that if the company continued deploying physical machines in a ‘one server to one application’ way, it would have cost to much. “Now, we’ve standardized on VMware and we have a strict virtualization-first policy. As a result, we can spend less time managing our IT infrastructure and more time focusing on bigger picture things, like growing the business,” Wong said in the VMware statement.

The company was also able to reduce its data center footprint by consolidating about 50 physical servers onto 10 VMware ESX Servers, which led to added savings in power and cooling costs.

Interfaith Medical Center
On September 23, VMware announced that Interfaith Medical Center (IMC) of Brooklyn, New York is using VMware Infrastructure 3 to create a virtualized environment for the Microsoft Windows-based applications.

The hospital began investigating virtualization when its server hardware requirements reached unsustainable levels; IMC faced power, space and budget constraints and needed a solution, VMware reported. After considering offerings from VMware, Microsoft and Citrix, IMC selected VMware Infrastructure and is now running nearly all of its critical applications in VMware virtual machines. The hospital also implemented a VMware-first policy for all new applications, VMware reported.

IMC has reportedly achieved consolidation ratios as high as 17:1 on its physical hosts and has virtualized approximately 95% of its Microsoft Windows-based applications, including IMC’s core Meditech health information system used for patient care and billing, BlackBerry Enterprise Server, Microsoft SharePoint, Kronos software for timekeeping and scheduling, and Lawson software for finance and accounting.

CITOC
On September 3, VMware announced that Houston-based CITOC, an award-winning provider of managed and hosted IT solutions, standardized on VMware Infrastructure as its application environment for internal systems and client solutions.

CITOC, whose customers including Lockheed Martin, Mary Kay Cosmetics, Sysco Foods and Carriage Services, evaluated a number of virtualization platforms to incorporate into its IT operations, including VMware Infrastructure and Microsoft Hyper-V. CITOC opted to standardize on the VMware platform, mainly because they could run three times as many virtual machines on VMware Infrastructure as it could on Microsoft Hyper-V with identical hardware, VMware reported.

James Garrett, chief operating officer at CITOC, said in VMware’s statement that CITOC guarantees 99.999% application availability for clients, and is trusting VMware to help them stick to that guarantee.

Garrett estimates that CITOC is hosting nearly 5,000 virtual machines for its customers. About 85% of those virtual machines are running Microsoft Windows-based applications. Internally, CITOC is using VMware for applications like Microsoft Dynamics, Microsoft Exchange, Microsoft SharePoint, Microsoft SQL Server, and BlackBerry Enterprise Server.

Garrett reported that one customer was running 343 physical machines when it hired CITOC to streamline its IT operations. Today, the customer is supporting its entire application environment with 16 blade servers (32 including redundancy) running VMware Infrastructure.