The Law of Unintended Consequences explains that in any endeavor, actions will have unanticipated results. These results can fall into the categories of positive, neutral or negative (with the negative usually attracting the biggest spotlight, and the subsequent blame).

Today, for example, I encountered this law, but in the all-too-rare, positive consequence. As I was researching some processors for their Fault Tolerance compatibilities on the VMware website (riveting, I know), I happened across a new software tool that was released just hours previously, SiteSurvey 2.0.  Hey, something new to try out — and it’s free!

Having the irresistible compulsion to try out new things, I immediately downloaded it and installed it on my XP machine.  Starting up the application, it looks for your vCenter (or in this case, Virtual Center) to authenticate to, and a cluster to run against. Having completed the fields, it began cranking away (thus giving me time to contemplate the wisdom of running a new tool without spending sufficient time on the documentation; at least it isn’t a production environment).

The resulting report was far more useful than I could have imagined! Displayed in a browser, the page contained the following information:

• CPU type and speed for each host in the cluster
• BIOS Compatibility
• CPU Stepping
• NIC faster than 1 Gb/S
• ESX Version
• VMotion NIC
• ESX Licensed for FT
• Logging NIC
• Shared storage; volumes and other hosts that share the storage

The above results alone would have been more than sufficient to justify keeping the tool close at hand.

Alas, this is where The Law of Unintended Consquences comes in. It turns out that SiteSurvey 2.0 produces other very useful data. The tool reports the FT compatibility on the following categories for each virtual guest machine within the cluster, displayed in an easy to read matrix, with big red “X”s for items that require attention:

• Storage
• CPU
• Disk
• Snapshots (OK, now this is the icing on the cake)
• OS
• PRDM
• PV
• NPIV
• Drives
• Drivers
• NIC

For FT to work properly, all snapshots must be removed. This tool provides the unintended happy benefit of providing a list of hitherto unknown snapshots that were still resident on the cluster. Using the VIC, I immediately removed the offending snapshots. Don’t get me wrong, I love snapshots; I just don’t like keeping track of them, particularly in ESX 3.5, as well as what they can do to your storage if they are forgotten.

So, if you haven’t tried out the new version of SiteSurvey, I encourage you to jump over to VMware and give it a whirl.  Who knows what may happen?

Cheers,

Mak King

When you delete snapshots using the VMware Infrastructure Client, the taskbar that displays the deletion progress immediately jumps to 95% and stays at 95%  the snapshot is entirely deleted. The taskbar does not show any progress while the delta files are being merged back into the original .vmdk file. Additionally, vCenter Server has a default 15 minute task timeout, so if the snapshot deletion task takes longer then 15 minutes it will timeout in vCenter Server even though it is still running on the host server.

The method in the KnowledgeBase article uses an ESX service console command to perform a directory listing on .vmdk files, so you can see their modified times changing and that the delta.vmdk files disappear after the snapshots are deleted. Here is the command from the knowledgebase article:

watch "ls -lhut --time-style=full-iso *.vmdk"

The way the syntax is currently published, however, is incorrect. Is it missing the double-dash before the time-style option; hopefully VMware will correct this soon.

Let’s go over what the command does. The watch command runs a command repeatedly and displays its output every 2 seconds. The command that watch is running in the previous sequence is the ls command, which lists file information in the directory. The options used with the ls command are –lhut, which are individual options that are all grouped together and are listed below:

• -l – use long listing format which displays more information
• -h – display file sizes in human-readable form (i.e. GB instead of bytes)
• -u – when used with –lt sorts and shows files by access time
• -t – sorts by modification time

Additionally the -–time-style option set to full-iso shows the modified time in longer form with seconds, and the final *.vmdk option tells the ls command to only display .vmdk virtual disk files.

While this command works I modified it a bit to drop the group/owner information (-og instead of –l) from being displayed, so the results fits better on the screen.  I also tweaked it to use the shorter –-fulltime parameter instead.

watch “ls –oghut –-full-time *.vmdk”

To use the command, log in to the VMware service console, go to your VM’s directory where the snapshot files are located, and run the command. A listing of the files will be displayed along with size and modification information as shown below:

Once you see the –delta.vmdk files disappear you will know that the snapshot has been deleted, as shown below:

To stop the command from running just press ‘.’ While you will probably not use this method when deleting smaller snapshots, it is useful for larger ones that can take quite a while to delete.

VMware’s vCenter Server does not provide a centralized mechanism for managing snapshots so subsequently you must resort to other methods if you want to find them. Let’s review the various methods that you can use to find snapshots. If you know how to find them, you’ll be aware of snapshots that are still running and won’t be (as) surprised later when your volumes run out of disk space.

The first method is fairly simple and relies on the find command that is part of the VMware ESX service console. To use it, log in to the service console, switch to your VMFS volume directory and type the following command. The command will find all files that have ‘delta’ in their file names. Deltas indicate snapshot files.

find –iname “*-delta.vmdk”

Or, to find orphaned snapshots that have not been modified in a number of days, type:

find –iname “*-delta.vmdk” –mtime +7 ls

The second method requires one of the free Perl scripts (SnapHunter or SnapAlert) that were written by VMware users. This method requires use of a small Perl script which can be run inside the ESX service console or with the remote command-line utility (Remote CLI) and will search for snapshots and create a report. (You have the option to email the report.) To use these scripts you copy it to the service console or the workstation/appliance running the Remote CLI, modify the script with your vCenter or ESX server name and log in credentials, and then run them.

I also listed a third method that used a VB script to query the vCenter Server’s VPX_SNAPSHOT table for active snapshots. While this method works, it will only show snapshots that vCenter Server is aware of and may not show orphaned snapshots or those taken outside of vCenter Server. Also, since the database design is subject to change with any vCenter Server upgrades, this may break any existing SQL queries. So I’d like to recommend a different method instead which involves using PowerShell and the VMware Infrastructure Toolkit to query for snapshots. You can use simple PowerShell commands to check for snapshots. There are a few sample scripts that you can use. You can find them at the following URLS:

http://www.peetersonline.nl/index.php/vmware/powershell-oneliner-4/
http://communities.vmware.com/docs/DOC-6980

Lastly, there are a few more methods that you can take advantage of by using a reporting or search application. Check out the freely available RV Tools or vKernel’s SearchMyVM. Both enable you to search and report on your virtual environment. These free tools are a good addition to any VMware environment. RV Tools is a .NET application that installs on a workstation and SearchMyVM is a virtual appliance that you can download and import to an ESX host.

Whichever method you choose, you should make sure to periodically look at the snapshots running in your environment. You can automate many of these queries (i.e. using a cron job or Windows task scheduler) so that they run on a scheduled basis. By staying on top of your snapshots, you ensure optimal performance of your hosts and VMs and also avoid disk space problems.