There are two methods for backing up a VM, traditional backup methods that install an agent inside the OS and back it up file-by-file and image-level backups that back up the single virtual disk VMDK file. Image-level backups are usually done by backup applications that are designed to specifically back up virtual machines, like Veeam Backup and Replication. These applications use the snapshot feature that is built in to VMware to stop disk writes to the virtual disk so it can be safely backed up. Backup applications read the original virtual disk file which is now read-only as new disk writes get written to a newly created delta virtual disk file. Once the backup application has read all the blocks from the original virtual disk file the snapshot is committed, which takes the data from the delta virtual disk and writes it to the original virtual disk. Once this is complete the delta virtual disk is deleted.

In virtual environments it’s more efficient to do image-level backups, but they also offer more options when you need to do a restore. If you want to restore the whole VM to a previous state you can simply restore the whole virtual disk file. Or, if you want to restore individual files, you can easily do this also. When using a backup application for virtual environments the process for individual file restores in virtual machines is easy and straight-forward. For example, Veeam Backup and Replication can quickly mount the backed-up VM disk file from the backup location so it is available for browsing; the files that need to be restored can then be selected and copied back to the original VM, a process that takes only minutes.

VMware’s own VM backup product, VMware Data Recovery can also do individual file restores via a command-line application that mounts the virtual disk from a selected VM/restore point to a Windows drive letter so the files can be accessed and copied. Even if you use simple scripts to back up a VM by copying its disk to another storage location you can restore files easily enough. Virtualization makes this fairly simple as you can just add the backed-up virtual disk file to another helper VM as an additional disk, browse the files on that disk and copy and paste them back to the original VM.

As you can see restoring individual files to virtual machines is a simple and easy process when using a backup application designed to work with virtual machines. It may not be the same process that is used with physical servers and traditional methods, but it is definitely possible and not all that difficult. If anything virtualization makes backup and recovery easier by providing many unique alternative backup and recovery methods then are not possible with traditional methods.

The other night our storage area network (SAN) admininstrators shut down both of our SANs so they could do a microcode upgrade. Part of this process involved shutting down all of our production virtual machines (VMs) that were on shared storage before the SAN was shut down.

But there are certain servers that you want to have available at all times. In my work this includes at least one DNS and Active Directory server as well as our VPN authentication server. Because these VMs are all on shared storage I decided to use Storage VMotion to move them temporarily to local storage so they would be available while the SAN was down. With Storage VMotion I was able to do this while all the VMs were powered on with no interruption to service.

What Storage VMotion is and how it works

Introduced with VMware ESX 3.5 and vCenter Server 2.5, Storage VMotion allows you to move VMs from one storage data store to another while the VM is running. The difference between VMotion and Storage VMotion is that VMotion simply moves a VM from one ESX host to another but keeps the storage location of the VM the same; Storage VMotion changes the storage location of the virtual machine while it is running and moves it to another data store on the same ESX host. The source and destination data stores can include any storage volume that is configured on an ESX host, which includes local and shared storage. The magic behind Storage VMotion involves several behind-the-scenes steps, which are outlined below:

1. A new virtual machine directory is created on the target data store, virtual machine configuration files and all non-virtual disk files are copied to the new target directory.
2. The VMware ESX host does a “self” VMotion to the target directory.
3. A snapshot (without memory) is taken of the virtual machine’s disks in the source directory.
4. Virtual machine disk files are copied to the target directory.
5. Snapshot that is located in the source directory is consolidated into the virtual machine disk files located in the target directory.
6. Source disk files and directory are deleted.

With ESX 3.5 you either had to initiate a Storage VMotion using a Remote Command Line Interface (Remote CLI) Perl script, or you could use a third-party plug-in developed by Andrew Kutz to do this inside the VMware Infrastructure Client. Storage VMotion is now fully integrated into the vSphere Client but you can still use the Remote CLI as well.

To be able to use Storage VMotion there are a few requirements that must be met, as outlined below.

1. Virtual machine disks must be in persistent mode or be a raw device mapping (RDM) that is in virtual compatibility mode.
2. If a virtual machine has any snapshots then it cannot be migrated, you must delete them before you can proceed.
3. The ESX host that the VM is running on must be licensed for VMotion and must also be configured to use VMotion.
4. The ESX host that the VM is running on must have access to the source and target data stores.
5. The ESX host that the VM is running on must have enough resources available to support two instances of the virtual machine running at the same time.

It’s not every day that you will use Storage VMotion, but it can come in very handy in certain situations when a VM needs to be moved to another data store and you can’t afford to shut it down.

You can also use Storage VMotion and VMotion together to move a VM from a local data store on one host to a local data store on another host by first using Storage VMotion to move it to shared storage, then using VMotion to move it to another host, and finally using Storage VMotion again to move it back to local storage on the new host.

iSCSI is quite easy to configure. ESX’s iSCSI support is fully available in the form of a software initiator that uses a VMkernel interface. “That easy?” you ask? Yes, it is really that easy.

Using Ethernet is convenient. Until this point, I have exclusively used fibre channel storage for virtual machine file system (VMFS) volumes. With the ESX iSCSI software initiator, I simply dedicated some gigabit network interface cards to the VMkernel interface and was ready to configure the iSCSI adapter. There is experimental support for a hardware initiator with the QLogic 4010 interface.

There is a minimal configuration for the storage adapter. ESX has an iSCSI software adapter listed in the storage adapters section of the VMware Infrastructure Client. Once you configure this interface, the system is ready to receive a LUN. The figure below shows the configuration of the software iSCSI interface:

After those pointers, I was quickly running with a LUN provided from the storage system. Once the LUN is presented to the host, it is indistinguishable from other VMFS volumes. Full VMotion, Distributed Resource Scheduler and other VMware tools are available on these volumes, including the esxcfg- series of commands.

If you are getting started with iSCSI, be sure to go through the drills related to configuration steps on ESX. Also, visit your system architecture plan and make sure that the iSCSI interfaces are provisioned well by not also holding other traffic, and be sure to check out VMware’s iSCSI configuration document available for download from the VMware website.

I do not back up the ESX host in a way that I would ever want to restore it. Occasionally, there are needs for onetime backups of VMDK files or such that are well suited for an agent backup, but for the most part the hosts are a transient set of resources that I present vCenter.

There are some practice issues that can optimize how this preference can be used, and I’ll share a few of them here. Some of these I currently use in production, some I have used in lab functions only.

Before you locally destroy a server’s state, it may be a good idea to run the vm-support tool or generate diagnostic bundles before the system is gone forever. Of course, this is not always possible, but it is a nice way to have the critical logs available before the installation is replaced.

One of the first points is the server reinstallation time requirement. Simply installing ESX is quite easy and can be done in twenty minutes or so. Some large ESX environments may want to look into an ESX kickstart script. These scripts can provide a scripted out answer file for the ESX install and can be made to work on a PXE boot. This can not only slightly reduce the reinstallation time, but can also ensure configuration consistency.

Now that ESX is configured, you may have a wonderful time reconfiguring all of the virtual switches and port groups. ESX has some native help here with the use of the esxcfg-vswitch series of commands, (check out the link for a blog post I did earlier that can get you started using this command). I’ll also pass along a site I recently came across with some good tools from Richard Garsthagen, a Netherlands-based VMware evangelist who has a cool blog with some good ESX tools that can help in this area, especially with the ITQ VLAN and portgroup manager tool.

The last series of configuration in a re-installation can revolve around storage pieces, such as multipath policies, iSCSI, or host bus adapter (HBA) information and configuration may also be optimized by making scripts with the esxcfg series of commands.

Finally, again echoing Edward’s comments, the best tool to have is good documentation and a confidence in the installation. This will permit the reinstallation to succeed smoothly and in a timely fashion.

Systems - What products are supported for installations for ESX and ESXi platforms
Storage and SAN – Allows searches for partners and their products for ESX and ESXi-based storage devices.
I/O devices – Has brand information for supported HBAs, RAID controllers, and SCSI adapters.
VMware View – Lists supported connecting devices to the new virtual desktop product.

This new hardware compatibility guide search website also has direct links to all of the relevant other configuration information. This includes resources on CPU configuration for VMotion, supported guest operating systems, as well as my trusty PDF documents that are available as a traditional download.

More information on the new tool can be found in the online help section of the hardware compatibility guide website.

VMware View Composer is simply a Windows service that resides on a vCenter server. VMware View Composer interacts with both vCenter and the View Connection Manager. For environments that already have a server based VMware environment with vCenter and ESX hosts, it is clear that a separate environment is a good idea. This would be best served through dedicated hosts, storage and a separate vCenter server. The figure below shows how the VMware View Composer and vCenter installations would work together:

The VMware View Composer service, or svid, interacts with vCenter from the configuration set forth from VMware View Connection manager, which functions as the broker for connections. Once the linked clone virtual desktops are created, they then deliver the storage optimization that we have been anticipating with the release of VMware View.

One other feature of View Composer is storage over-commit. This functionality is a configurable level of how the linked clones’ delta disk, or differencing disk, is allocated. Looking at a guest virtual machine, the delta disk would be a very small percentage of the parent or base VM. This setting will determine the behavior of determining how many VMs will fit on a datastore. A setting of conservative will enable less VMs to fit on a datastore, yet run less of a likelihood of running out of space. While a more aggressive level will attempt to put more VMs on the datastore and reserve less storage reserved for the delta disks. With that information, it is critically important to get an expectation of the delta disk behavior to best utilize the storage.

A final key component of View Composer is the Quickprep feature. Quickprep does the guest VM specific tasks such as domain membership, organizational unit placement in Active Directory and run any scripts on the guest VM.

With this information primer on VMware View Composer, it is important to isolate the vCenter and more importantly be aware of how the virtual desktop managment agents will interact with the vCenter server. More information on VMware View can be found on the VMware website.

[#2] [2008-11-08 08:16:33.171 'App' 1536 error] [imageProcessingTaskImpl,552] VmiImportTask::task{4}: Image processing task has failed with MethodFault::Exception: vim.fault.NoDiskSpace

After some deeper research, it turns out that this particular datastore was created with the default 1 MB block size, making VMDK files limited to 256 GB. Fellow SearchVMware.com blogger Eric Siebert has an excellent post on this site about the formatting options of the VMFS volume as it is brought into VI3. When you add a LUN, you select the block size during the format shown in the figure below:

While running VMware Converter on very large volumes is rare, this was one that made me stop and think for while. There are plenty of strategies for very large storage for a P2V conversion, but this was one where the other LUNs were configured with the 1 GB block size and that was fine thus far.

Luckily, there was space that I could evacuate all other virtual machines on that LUN so that I could remove the LUN and format it with a larger block size to get the VMDK file size I needed. The tricky part of this situation is that with VMware Converter experiencing the error on the block size, it was displayed in the log in a way that made it a little tricky to discover.

I recently upgraded from an HP MSA 1000 to an IBM DS3400 because I wanted to improve performance and lower my overall energy costs.  One of the reasons I decided to replace my old SAN is because it is much cheaper to have a single 2U device running than the three devices for the old SAN. In addition, I dropped from 42 drives to 12 drives with more storage. Minimally my SAN power costs should drop to just 1/3 the original. I also have gone from 2.5 TB to 3 TBs of storage. Not a huge increase in storage capability.

I will know next month if my energy cost reductions have been realized and will report back then.

The steps for replacing a SAN are not all that tricky, but there was a single gotcha that could be avoided with careful planning. To replace a SAN, follow these steps:

1. Plug in the new SAN to your existing fabric. Luckily I had a pair of unused fibre connections and Gbics available else this would have been another expense and a delay until the cables and Gbics arrived.

2. Find a system on which to install the management console. For the IBM DS3400 I chose my VirtualCenter and VMware Consolidated Backup (VCB) server to be the management console for the SAN. There are two methods to manage the IBM DS3400: in-band or over the fibre channel fabric, or out of band using Ethernet — even a VM would suffice given networking is connected to the SAN. Software exists for both 64-bit Linux and Microsoft Windows.

3. Create the LUNs on the new SAN. This is a good chance to correct any problems you may have with the LUN configuration on the old SAN. I did a one-to-one mapping, except I slightly increased the size of the LUNs.

4. Present the LUNs to your VMware ESX host(s) and VCB server(s).

5. Rescan the storage adapters for new LUNs using the VMware Infrastructure Client (VI Client) for the first VMware ESX host. Once this is completed, you can then add as many Virtual Machine File Systems (VMFSs) as required.

6. Rescan the storage adapters for new LUNs and VMFSs using the VI Client on all the other ESX hosts.

7. Employ Storage VMotion via the VI Client to migrate VMs from one LUN to another LUN. This works if you have the patience to move all the VMs one by one. If not you can employ other measures. If you do this, however, you will end up having to edit the VMX files for each VM migrated to change the location of the virtual disk files. There are scripts to do this for you as well. This second option, however, also requires you to power off all VMs. Use of Storage VMotion does not require any VM downtime. Be sure to move all files from the LUNs in use.

8. For a LUN with an RDM (mine was a Linux file server), use Storage VMotion to move any VMDKs related to the VM. Then map the new RDM to the VM. You will have to reboot the VM to complete. Then create a new filesystem on the new RDM mount the file system. Then you must copy all the files from the old RDM to the new RDM. I used the following command to complete this task to copy all files from /files to /files2.

• cd /files; rsync -ravlpog * /files2

9. Then I modified the mount point for /files within /etc/fstab to be the correct new location. Finally I powered off the VM, deleted the old RDM from the VM and powered on the VM picking up the new data.

Here is the gotcha. I missed it, but it will be extremely useful for you (and me) going forward: Remove the old SAN’s LUNs from each VMware ESX host. If you miss this step when you finally disconnect the old SAN, the ESX hosts will go into a state of constantly attempting to failover the old LUNs. This will spew massive failures into the log files. If this happens there is no recourse but to reboot the VMware ESX hosts.

Now the SAN has been replaced. With the exception of dealing with any RDMs, it is possible to migrate to a new SAN without any downtime.

The support matrix for storage and SAN configuration is an absolutely critical component to planning additional storage purchases or expanding current environments. Among my small circle of peers, I have been a little critical of VMware for releasing documentation that covers ESX 3 in a blanket format. This is a big step in the right direction, as the supported environments and their functionality vary by platform which was the crux of my frustration with blanket documentation.

This split in documentation is likely due to ESX 3i and Storage VMotion related supported environments. At first glance at the two guides they seem similar, but the following was taken from the 3.5 and 3i guide:

You will note that this guide is sparsely populated at present. The reason for this is that storage arrays require re-certification for ESX Server 3.5 and ESX Server3i, and while many re-certifications are in process or planned, relatively few have been fully completed to date. In contrast, servers and I/O devices do not require re-certification.

While this is somewhat of a surprise for a nearly 11-month-old product line, I still welcome the split documentation. Be sure to check these guides when making storage related infrastructure decisions, as they change frequently and based on the excerpt above should be updated. VMware’s supported configurations for storage are important to not only deliver a solution that works as expected, but to lay the framework for the virtual machine file system or VMFS, which I touched on during a prior blog post of why the proprietary drivers are important.

-The use of Volume Shadow Copy Service (VSS) for snapshots of Windows VMs
-More efficient communication between the host and guest or between the guest and multiple guests on the same host using the Virtual Machine Communication Interface (VMCI)
-The ability to add new disks to running VMs
-Guest VM direct SCSI support for devices such as a tape drive
-Additional browser support for VMware Infrastructure Web Access to include FireFox 3
-Redirection of devices on the guest VM to the web client

I had a chance to install the RC1 version, and the new features are welcome additions from the previous betas. One of the new features that I want to touch on is the ability to add a new disk to a running VM. This feature allows an administrator to add storage to the VM’s inventory with the normal options of disk size, disk type and the option to use an existing .VMDK file. The figure below shows the wizard within the web interface to add the hard disk to the running VM:

Once the wizard is completed, the drive is available to the operating system. At that point, different operating systems handle the arrival of a hard disk differently. The figure below shows the new hard drive on Windows Server 2008 after a rescan task:

In the example used with the VM above, the hard drive can be added and used in the operating system immediately. Other VMware products, such as ESX have had this functionality, but it is the first time for the Server product. This feature is a welcome addition to the free product series as many organizations use VMware Server in a live workload capacity.

More information on the VMware Server 2.0 RC1 version can be found in the release notes on the VMware website.