virtual machine

Killing a frozen VM on a vSphere ESX host

Occasionally virtual machines (VMs) get stuck in a zombie state and will not respond to a power-off command using the traditional vSphere client power controls. Rebooting a host will fix this condition — but rebooting is usually not an option. Fortunately, there are a few methods for forcing the VM to shut down without rebooting the host.

I previously documented these methods with VMware Infrastructure 3 (VI3) and wanted to make sure they all worked with vSphere. The methods below are listed in order of usage preference starting with using normal VM commands and ending with a brute force method.

Method 1 - Using the vmware-cmd service console command (the command-line interface equivalent of using the vSphere Client) Continued »

Fixing VMware “no swap file” VM power-on failures

Recently I experienced a VMware HA event in my environment which caused the VMs on the affected hosts to be restarted on other servers. While most of the VMs started OK, there were a few that did not. When I manually tried to start them I received the error “Failed to power on VM - No swap file” and the VM would fail to start. What happened is that several VMs were in a zombie-like state, as they were not shut down gracefully. Even though their statuses were displayed as shutdown in the VMware Infrastructure Client (VI Client), there was still a process running on an ESX host that prevented it from being started.

In effect, while the VM’s OS was not running it will still in a running state on an ESX host and had a .vswp file already out there that could not be deleted. As a result, when another host tried to start it the .vswp file could not be created because the other host had a lock on it.

To resolve this situation I had to find out the host that still had a running process for the VM and forcibly terminate the process. To do so I had to log in to the service console of each host and run the following command: ps auxfww | grep VM name. This command returns a list of running process that contain the name of the VM.

When you run the ps command with the VM name listed you will always have one result regardless of if the VM is actually running on the host. This is because the command itself shows up in the result list as the VM name is being used in the command when it is run. However, if the VM is actually running on the host you will receive two results instead of one. The second result will be much longer as it contains several lines of text and will contain the path to the .vmx file of the VM. This second result also contains the process ID (pid) of the VM which can be used to forcibly terminate it. The pid of the VM is located in the second column of the results right after the username (typically root). As you can see in the below example, the first result with a pid of 25914 is the command itself and the second result with a pid of 23896 is the running VM.

[root@esx1 root]# ps auxfww | grep win2003-1

root     25914  0.0  0.2  3688  676 pts/0    S    13:17   0:00          \_ grep win2003-1

root     23896 0.0  0.2  2008  864 ?        S<   Feb13   4:12 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -# name=VMware ESX Server;version=3.5.0;licensename=VMware ESX Server;licenseversion=2.0 build-123630; -@ pipe=/tmp/vmhsdaemon-0/vmxd0af4bb011822fc5; /vmfs/volumes/442d541b-cb5a815d-6083-0017a4a9c074/ win2003-1/ win2003-1.vmx

Now that we know the pid of the VM (23896), to forcibly terminate it we type kill -9 23896. You can verify that the VM process has been terminated by running the ps command again. Only one result should be returned. Now that the VM has been stopped you can power it on using the VI Client and you should have no problems this time.

Using live CDs with virtual machines

Live CDs are bootable CD/DVD images that contain an operating system that you can boot from to perform various tasks on a server. When you boot from a live CD, the operating system from the server does not load. Instead the live CD operating system loads, allowing you access to the server’s file system. There are hundreds of free live CD’s available that you can download and use, and they are particularly useful for troubleshooting or repairing a problem with a server.

Most of these live CDs use one of the many Linux distributions as there is no cost for using them and they are very compact. There are also some Windows versions that use a special version of Windows called Windows PE. While physical servers usually rely on a physical CD/DVD which must be burned and placed in the server’s CD/DVD-ROM drive, virtual machines can make use of ISO images which are image files that can be directly mounted to a VM’s CD-ROM.

Live CDs have a wide range of functionality. It’s handy to have a good selection of them available. So when would you want to use a live CD?

•When you have a server that has been infected with a virus or spyware. Booting from a live CD will allow you to run scanning utilities to remove the infection without the server being active.
• When you have a corrupt operating system that will not boot or that crashes while booting. Booting from a live CD will allow you to repair corrupted files or configuration files.
• When you’re installing an application that renders a system unusable. Booting from a live CD will allow you to disable or uninstall the application.
• When you’ve forgotten the administrator password to log in to your server. Booting from a live CD can allow you to crack or reset it.

Almost all live CD images are available as an ISO download which is ready to use on your virtual machines. You simply edit the settings for the virtual machine and select the CD/DVD-ROM drive and map it to either your client CD or a host data store where you can select the ISO file. It’s a good idea to have an ISO data store available to all your hosts that contains the ISO images so you can use them when needed.

As mentioned, there are many live CD images that you can download and use, so to get you started I’ve compiled a list of some of the best ones that will make a good addition to your ISO library.

• Ultimate Boot CD - A Swiss Army knife boot CD which has many diagnostic, hard disk and file system tools as well as antivirus and network tools.
• Ultimate Boot CD for Windows - A Windows version of the Ultimate Boot CD with a huge array of tools including many antivirus and antispyware, disk, file, security, password reset, network tools and much more.
• Knoppix - A Linux Live CD that contains many tools and utilities.
• Backtrack - A huge selection of security tools that you can use to scan for vulnerabilities of your VMs.
• Kaspersky Resce CD - A Live CD with anti-virus scanner to clean infected systems.
• Gnome Partition Editor - A hard disk partition utility that lets you do all sorts of things with disk partitions and is particularly useful for resizing existing partitions on your VM after you have increased the size with vmkfstools or the VMware Infrastructure Client.
• Ophcrack - A utility that will crack forgotten Windows passwords.
• System Rescue CD - A multi-purpose rescue CD that includes partition editors, file system utilities, password recovery and much more.

These are just a few of the live CDs available. If you would like to build your own live CD based on Windows PE check out the Bart PE website for more info. You can also check out the live CD list website for a huge list of available live CD’s.

Avoid disk space problems; monitor your snapshots

In a VMware snapshot series I wrote a few months ago (How VMware snapshots work, Deleting virtual machine snapshots without wasting disk space and Troubleshooting VMware snapshots), I listed a few methods for finding snapshots that are running in your environment. I thought I would expand on those methods and also offer some new methods for finding snapshots. Snapshots are a handy tool but leaving them running for any longer then necessary is not a good idea, (read the snapshot series for an explanation).

VMware’s vCenter Server does not provide a centralized mechanism for managing snapshots so subsequently you must resort to other methods if you want to find them. Let’s review the various methods that you can use to find snapshots. If you know how to find them, you’ll be aware of snapshots that are still running and won’t be (as) surprised later when your volumes run out of disk space.

The first method is fairly simple and relies on the find command that is part of the VMware ESX service console. To use it, log in to the service console, switch to your VMFS volume directory and type the following command. The command will find all files that have ‘delta’ in their file names. Deltas indicate snapshot files.

find –iname “*-delta.vmdk”

Or, to find orphaned snapshots that have not been modified in a number of days, type:

find –iname “*-delta.vmdk” –mtime +7 ls

The second method requires one of the free Perl scripts (SnapHunter or SnapAlert) that were written by VMware users. This method requires use of a small Perl script which can be run inside the ESX service console or with the remote command-line utility (Remote CLI) and will search for snapshots and create a report. (You have the option to email the report.) To use these scripts you copy it to the service console or the workstation/appliance running the Remote CLI, modify the script with your vCenter or ESX server name and log in credentials, and then run them.

I also listed a third method that used a VB script to query the vCenter Server’s VPX_SNAPSHOT table for active snapshots. While this method works, it will only show snapshots that vCenter Server is aware of and may not show orphaned snapshots or those taken outside of vCenter Server. Also, since the database design is subject to change with any vCenter Server upgrades, this may break any existing SQL queries. So I’d like to recommend a different method instead which involves using PowerShell and the VMware Infrastructure Toolkit to query for snapshots. You can use simple PowerShell commands to check for snapshots. There are a few sample scripts that you can use. You can find them at the following URLS:

http://www.peetersonline.nl/index.php/vmware/powershell-oneliner-4/
http://communities.vmware.com/docs/DOC-6980

Lastly, there are a few more methods that you can take advantage of by using a reporting or search application. Check out the freely available RV Tools or vKernel’s SearchMyVM. Both enable you to search and report on your virtual environment. These free tools are a good addition to any VMware environment. RV Tools is a .NET application that installs on a workstation and SearchMyVM is a virtual appliance that you can download and import to an ESX host.

Whichever method you choose, you should make sure to periodically look at the snapshots running in your environment. You can automate many of these queries (i.e. using a cron job or Windows task scheduler) so that they run on a scheduled basis. By staying on top of your snapshots, you ensure optimal performance of your hosts and VMs and also avoid disk space problems.

Three ways to temporarily move VMs off a VMware ESX host

I recently came across an ITKE question from a user that had some virtual machines (VMs) on an ESX host. He wanted to move the VMs off of it so he could reconfigure the server with more storage, reinstall ESX and then move the VMs back on to it. Presumably he only had one ESX host with only local disk. There are several options for accomplishing what he wanted to do.

First, you need to find alternate storage to place the VM’s on. This could be a workstation, a Windows or Linux server or an NFS/iSCSI storage device with enough free space to temporarily hold the VMs. Once you find your temporary home, there are several methods that you can use to move the VMs.

Method 1: Create a network-based data store

The first method is the simplest and involves creating a network-based data store on the ESX host. You can do this with a network file system (NFS) or by using a software iSCSI initiator. If you have a Windows or Linux server that supports NFS, you can create a NFS share and then map your ESX host to it to use as a data store by creating a new data store. You can also setup one of the open-source iSCSI appliances like OpenFiler to use as a data store for your ESX host. Once you have a new data store you simply shut down the VMs and use the service console command-line tools to move the VM to the new data store. The vmkfstools –i command will allow you to clone the VMs disk to another VMFS datastore. Then, after you copy the VMs disk to the new data store, you simply add the data store back to the newly rebuilt ESX host and use vmkfstools –i to copy the disk back to it and create a new VM, and tell it to use an existing disk.

Method 2: Use VMware Converter

Another method is to use VMware Converter to migrate the VM to a local workstation or network drive. To do this just install Converter on the VM, choose to convert the local server, and for the destination choose “Other Virtual Machine.” You can then select a network path to store the new VM. It’s a good idea to resize the VM so there is less data to transfer; you can always resize it back to its original size when you move it back to the ESX host.

Method 3: Use a secure copy (SCP) file transfer utility

Finally, the last method is to use one of the secury copy (SCP) file transfer utilities to copy the VM’s files to a local workstation or network drive. FastSCP works best for this. It copies data the fastest because it does not encrypt the data being copied. Then you simply copy the data back to the host after it is rebuilt and either re-register the VM or create a new one and tell it to use an existing disk.

Once you copy virtual disks to an ESX host it’s always a good idea to clone them before using them using vmkfstools as it allocates all the disk space at once which results in a disk with less fragmentation. Tools like FastSCP only allocate space as the data is copied which can result in more fragmentation of the virtual disk file. To clone a disk with vmkfstools just type vmkfstools –i . Once the cloning process completes you can remove the existing disk from the VM and add a new one and browse to the newly created virtual disk.

Whichever method you choose, make sure you have a good backup of the VM before starting.