I previously documented these methods with VMware Infrastructure 3 (VI3) and wanted to make sure they all worked with vSphere. The methods below are listed in order of usage preference starting with using normal VM commands and ending with a brute force method.

Method 1 – Using the vmware-cmd service console command (the command-line interface equivalent of using the vSphere Client)

1. Log in to the ESX service console.
2. The vmware-cmd command uses the configuration file name (.vmx) of the VM to specify the VM to perform an operation on. You can type vmware-cmd -l to get a list of all VMs on the host and the path and name of their configuration file. The path uses the Universally Unique Identifier (UUID) or long name of the data store; alternatively, you can use the friendly name instead. If you do not want to type the path when using the vmware-cmd command you can change to the VM’s directory and run the command without the path.
3. You can optionally check the power state of the VM by typing vmware-cmd <VM config file path & name> getstate.
4. To forcibly shutdown a VM type vmware-cmd <VM config file path & name> stop hard.
5. You can check the state again to see if it worked; if it did the state should now be off.

Method 2 – Using the vm-support command to shut down the VM by first finding the virtual machine ID

…and then using the vm-support command to forcibly terminate it. This method does a lot more then shutting down the VM, as it also produces debugging information that you can use to troubleshoot an unresponsive VM.

1. Log in to the ESX Service Console.
2. The vm-support command is a multi-purpose command that is mainly used to troubleshoot host and VM problems. You can use the -X parameter to forcibly shutdown a VM and also produce a file with debugging information. This command will create a .tgz file in the directory that you run it in and cannot be run from a VMFS volume directory (running it in the /tmp directory is recommended). First type vm-support -x to get a list of the virtual machine IDs (VMID) of your running VMs.
3. To forcibly shut down the VM and generate core dumps and log files, type vm-support -X <VMID>. You will receive prompts asking if you want to take a screenshot of the VM. A screenshot can be useful to see if there are any error messages. You will also be prompted to see if you wish to send an NMI and an ABORT to the VM, which can aid in debugging. You must say yes to the ABORT prompt for the VM to be forcibly stopped. Once the process completes, which can take 10-15 minutes, a .tgz file will be created in the directory in which you ran the command that you can also use for troubleshooting purposes. To avoid filling up your file system when the file is created, switch to the /tmp directory when you run the command.
4. You can check the state of the VM again either by using the vmware-cmd command or by typing vm-support -x and you should not see the VMID for that VM listed anymore. Be sure and delete the .tgz file that is created when you are done to avoid filling up your host disk.

Method 3 – Using the kill command by first finding the process identifier (PID) of the VM and then using the kill command.

1. Log in to the ESX service console.
2. The process status (ps) command in Linux shows the currently running processes on a server and the grep command finds the specified text in the output of the ps command. Type ps auxfww | grep <virtualmachinename> to get the process ID (PID) of the VM. You will have two entries returned, one is from the running of the ps command. The longer entry is the running VM process. The longer entry will end in the config file name of the VM and is the one you want to use; the number in the second column of that entry is the PID of the VM.
3. The kill command in Linux sends a signal to terminate a process using its ID number. The ‘-9′ parameter forces the process to quit immediately and cannot be ignored like the more graceful ‘-15′ parameter can sometimes be. Type kill -9 <PID> which will forcibly terminate the process for the specified VM.
4. You can check the state using the vmware-cmd command to see if it worked; if it did, the state should now be off.

All three of these methods work identically on ESX hosts in both VI3 and vSphere. These methods also work for ESXi, but their execution is a bit different. In a future blog post we will cover how to use these methods with ESXi.

In effect, while the VM’s OS was not running it will still in a running state on an ESX host and had a .vswp file already out there that could not be deleted. As a result, when another host tried to start it the .vswp file could not be created because the other host had a lock on it.

To resolve this situation I had to find out the host that still had a running process for the VM and forcibly terminate the process. To do so I had to log in to the service console of each host and run the following command: ps auxfww | grep VM name. This command returns a list of running process that contain the name of the VM.

When you run the ps command with the VM name listed you will always have one result regardless of if the VM is actually running on the host. This is because the command itself shows up in the result list as the VM name is being used in the command when it is run. However, if the VM is actually running on the host you will receive two results instead of one. The second result will be much longer as it contains several lines of text and will contain the path to the .vmx file of the VM. This second result also contains the process ID (pid) of the VM which can be used to forcibly terminate it. The pid of the VM is located in the second column of the results right after the username (typically root). As you can see in the below example, the first result with a pid of 25914 is the command itself and the second result with a pid of 23896 is the running VM.

[root@esx1 root]# ps auxfww | grep win2003-1

root     25914  0.0  0.2  3688  676 pts/0    S    13:17   0:00          \_ grep win2003-1

root     23896 0.0  0.2  2008  864 ?        S<   Feb13   4:12 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -# name=VMware ESX Server;version=3.5.0;licensename=VMware ESX Server;licenseversion=2.0 build-123630; -@ pipe=/tmp/vmhsdaemon-0/vmxd0af4bb011822fc5; /vmfs/volumes/442d541b-cb5a815d-6083-0017a4a9c074/ win2003-1/ win2003-1.vmx

Now that we know the pid of the VM (23896), to forcibly terminate it we type kill -9 23896. You can verify that the VM process has been terminated by running the ps command again. Only one result should be returned. Now that the VM has been stopped you can power it on using the VI Client and you should have no problems this time.

Most of these live CDs use one of the many Linux distributions as there is no cost for using them and they are very compact. There are also some Windows versions that use a special version of Windows called Windows PE. While physical servers usually rely on a physical CD/DVD which must be burned and placed in the server’s CD/DVD-ROM drive, virtual machines can make use of ISO images which are image files that can be directly mounted to a VM’s CD-ROM.

Live CDs have a wide range of functionality. It’s handy to have a good selection of them available. So when would you want to use a live CD?

•When you have a server that has been infected with a virus or spyware. Booting from a live CD will allow you to run scanning utilities to remove the infection without the server being active.
• When you have a corrupt operating system that will not boot or that crashes while booting. Booting from a live CD will allow you to repair corrupted files or configuration files.
• When you’re installing an application that renders a system unusable. Booting from a live CD will allow you to disable or uninstall the application.
• When you’ve forgotten the administrator password to log in to your server. Booting from a live CD can allow you to crack or reset it.

Almost all live CD images are available as an ISO download which is ready to use on your virtual machines. You simply edit the settings for the virtual machine and select the CD/DVD-ROM drive and map it to either your client CD or a host data store where you can select the ISO file. It’s a good idea to have an ISO data store available to all your hosts that contains the ISO images so you can use them when needed.

As mentioned, there are many live CD images that you can download and use, so to get you started I’ve compiled a list of some of the best ones that will make a good addition to your ISO library.

• Ultimate Boot CD – A Swiss Army knife boot CD which has many diagnostic, hard disk and file system tools as well as antivirus and network tools.
• Ultimate Boot CD for Windows – A Windows version of the Ultimate Boot CD with a huge array of tools including many antivirus and antispyware, disk, file, security, password reset, network tools and much more.
• Knoppix – A Linux Live CD that contains many tools and utilities.
• Backtrack – A huge selection of security tools that you can use to scan for vulnerabilities of your VMs.
• Kaspersky Resce CD – A Live CD with anti-virus scanner to clean infected systems.
• Gnome Partition Editor – A hard disk partition utility that lets you do all sorts of things with disk partitions and is particularly useful for resizing existing partitions on your VM after you have increased the size with vmkfstools or the VMware Infrastructure Client.
• Ophcrack – A utility that will crack forgotten Windows passwords.
• System Rescue CD – A multi-purpose rescue CD that includes partition editors, file system utilities, password recovery and much more.

These are just a few of the live CDs available. If you would like to build your own live CD based on Windows PE check out the Bart PE website for more info. You can also check out the live CD list website for a huge list of available live CD’s.

VMware’s vCenter Server does not provide a centralized mechanism for managing snapshots so subsequently you must resort to other methods if you want to find them. Let’s review the various methods that you can use to find snapshots. If you know how to find them, you’ll be aware of snapshots that are still running and won’t be (as) surprised later when your volumes run out of disk space.

The first method is fairly simple and relies on the find command that is part of the VMware ESX service console. To use it, log in to the service console, switch to your VMFS volume directory and type the following command. The command will find all files that have ‘delta’ in their file names. Deltas indicate snapshot files.

find –iname “*-delta.vmdk”

Or, to find orphaned snapshots that have not been modified in a number of days, type:

find –iname “*-delta.vmdk” –mtime +7 ls

The second method requires one of the free Perl scripts (SnapHunter or SnapAlert) that were written by VMware users. This method requires use of a small Perl script which can be run inside the ESX service console or with the remote command-line utility (Remote CLI) and will search for snapshots and create a report. (You have the option to email the report.) To use these scripts you copy it to the service console or the workstation/appliance running the Remote CLI, modify the script with your vCenter or ESX server name and log in credentials, and then run them.

I also listed a third method that used a VB script to query the vCenter Server’s VPX_SNAPSHOT table for active snapshots. While this method works, it will only show snapshots that vCenter Server is aware of and may not show orphaned snapshots or those taken outside of vCenter Server. Also, since the database design is subject to change with any vCenter Server upgrades, this may break any existing SQL queries. So I’d like to recommend a different method instead which involves using PowerShell and the VMware Infrastructure Toolkit to query for snapshots. You can use simple PowerShell commands to check for snapshots. There are a few sample scripts that you can use. You can find them at the following URLS:

http://www.peetersonline.nl/index.php/vmware/powershell-oneliner-4/
http://communities.vmware.com/docs/DOC-6980

Lastly, there are a few more methods that you can take advantage of by using a reporting or search application. Check out the freely available RV Tools or vKernel’s SearchMyVM. Both enable you to search and report on your virtual environment. These free tools are a good addition to any VMware environment. RV Tools is a .NET application that installs on a workstation and SearchMyVM is a virtual appliance that you can download and import to an ESX host.

Whichever method you choose, you should make sure to periodically look at the snapshots running in your environment. You can automate many of these queries (i.e. using a cron job or Windows task scheduler) so that they run on a scheduled basis. By staying on top of your snapshots, you ensure optimal performance of your hosts and VMs and also avoid disk space problems.

First, you need to find alternate storage to place the VM’s on. This could be a workstation, a Windows or Linux server or an NFS/iSCSI storage device with enough free space to temporarily hold the VMs. Once you find your temporary home, there are several methods that you can use to move the VMs.

Method 1: Create a network-based data store

The first method is the simplest and involves creating a network-based data store on the ESX host. You can do this with a network file system (NFS) or by using a software iSCSI initiator. If you have a Windows or Linux server that supports NFS, you can create a NFS share and then map your ESX host to it to use as a data store by creating a new data store. You can also setup one of the open-source iSCSI appliances like OpenFiler to use as a data store for your ESX host. Once you have a new data store you simply shut down the VMs and use the service console command-line tools to move the VM to the new data store. The vmkfstools –i command will allow you to clone the VMs disk to another VMFS datastore. Then, after you copy the VMs disk to the new data store, you simply add the data store back to the newly rebuilt ESX host and use vmkfstools –i to copy the disk back to it and create a new VM, and tell it to use an existing disk.

Method 2: Use VMware Converter

Another method is to use VMware Converter to migrate the VM to a local workstation or network drive. To do this just install Converter on the VM, choose to convert the local server, and for the destination choose “Other Virtual Machine.” You can then select a network path to store the new VM. It’s a good idea to resize the VM so there is less data to transfer; you can always resize it back to its original size when you move it back to the ESX host.

Method 3: Use a secure copy (SCP) file transfer utility

Finally, the last method is to use one of the secury copy (SCP) file transfer utilities to copy the VM’s files to a local workstation or network drive. FastSCP works best for this. It copies data the fastest because it does not encrypt the data being copied. Then you simply copy the data back to the host after it is rebuilt and either re-register the VM or create a new one and tell it to use an existing disk.

Once you copy virtual disks to an ESX host it’s always a good idea to clone them before using them using vmkfstools as it allocates all the disk space at once which results in a disk with less fragmentation. Tools like FastSCP only allocate space as the data is copied which can result in more fragmentation of the virtual disk file. To clone a disk with vmkfstools just type vmkfstools –i . Once the cloning process completes you can remove the existing disk from the VM and add a new one and browse to the newly created virtual disk.

Whichever method you choose, make sure you have a good backup of the VM before starting.