Storage

Using Storage VMotion to keep critical servers up when shared storage is down

What a great tool Storage VMotion is. I may not use it every day, but when certain situations arise I really appreciate having this feature available.

The other night our storage area network (SAN) admininstrators shut down both of our SANs so they could do a microcode upgrade. Part of this process involved shutting down all of our production virtual machines (VMs) that were on shared storage before the SAN was shut down.

But there are certain servers that you want to have available at all times. In my work this includes at least one DNS and Active Directory server as well as our VPN authentication server. Because these VMs are all on shared storage I decided to use Storage VMotion to move them temporarily to local storage so they would be available while the SAN was down. With Storage VMotion I was able to do this while all the VMs were powered on with no interruption to service.

What Storage VMotion is and how it works

Introduced with VMware ESX 3.5 and vCenter Server 2.5, Storage VMotion allows you to move VMs from one storage data store to another while the VM is running. The difference between VMotion and Storage VMotion is that VMotion simply moves a VM from one ESX host to another but keeps the storage location of the VM the same; Storage VMotion changes the storage location of the virtual machine while it is running and moves it to another data store on the same ESX host. The source and destination data stores can include any storage volume that is configured on an ESX host, which includes local and shared storage. The magic behind Storage VMotion involves several behind-the-scenes steps, which are outlined below: Continued »

Getting started with iSCSI and VMware ESX

Many VMware ESX administrators are quite comfortable with fibre channel storage but have not ventured into the iSCSI world. I recently set up my first iSCSI configuration for a small VMware Infrastructure 3 installation and it was quite successful. Here are some takeaways:

iSCSI is quite easy to configure. ESX’s iSCSI support is fully available in the form of a software initiator that uses a VMkernel interface. “That easy?” you ask? Yes, it is really that easy.

Using Ethernet is convenient. Until this point, I have exclusively used fibre channel storage for virtual machine file system (VMFS) volumes. With the ESX iSCSI software initiator, I simply dedicated some gigabit network interface cards to the VMkernel interface and was ready to configure the iSCSI adapter. There is experimental support for a hardware initiator with the QLogic 4010 interface.

There is a minimal configuration for the storage adapter. ESX has an iSCSI software adapter listed in the storage adapters section of the VMware Infrastructure Client. Once you configure this interface, the system is ready to receive a LUN. The figure below shows the configuration of the software iSCSI interface:

After those pointers, I was quickly running with a LUN provided from the storage system. Once the LUN is presented to the host, it is indistinguishable from other VMFS volumes. Full VMotion, Distributed Resource Scheduler and other VMware tools are available on these volumes, including the esxcfg- series of commands.

If you are getting started with iSCSI, be sure to go through the drills related to configuration steps on ESX. Also, visit your system architecture plan and make sure that the iSCSI interfaces are provisioned well by not also holding other traffic, and be sure to check out VMware’s iSCSI configuration document available for download from the VMware website.

VMware offers new searchable compatibility for support resources

In October of this year, I mentioned in a prior blog post that VMware updated their storage and compatibility guides to reflect a split of sorts between ESX 3.0.x and ESX 3.5 and ESXi. This is now available as a searchable by product name or hardware vendor for multiple solutions and provides a central resource for all supported configurations. This tool allows for the following search categories:

Systems - What products are supported for installations for ESX and ESXi platforms
Storage and SAN – Allows searches for partners and their products for ESX and ESXi-based storage devices.
I/O devices – Has brand information for supported HBAs, RAID controllers, and SCSI adapters.
VMware View – Lists supported connecting devices to the new virtual desktop product.

This new hardware compatibility guide search website also has direct links to all of the relevant other configuration information. This includes resources on CPU configuration for VMotion, supported guest operating systems, as well as my trusty PDF documents that are available as a traditional download.

More information on the new tool can be found in the online help section of the hardware compatibility guide website.

VMware View Composer and vCenter architecture

With the recent release of VMware View, one of the hottest components of the desktop virtualization component is the linked clone technology. In planning how VMware View works, it is important to understand the critical component - VMware View Composer.

VMware View Composer is simply a Windows service that resides on a vCenter server. VMware View Composer interacts with both vCenter and the View Connection Manager. For environments that already have a server based VMware environment with vCenter and ESX hosts, it is clear that a separate environment is a good idea. This would be best served through dedicated hosts, storage and a separate vCenter server. The figure below shows how the VMware View Composer and vCenter installations would work together:

The VMware View Composer service, or svid, interacts with vCenter from the configuration set forth from VMware View Connection manager, which functions as the broker for connections. Once the linked clone virtual desktops are created, they then deliver the storage optimization that we have been anticipating with the release of VMware View.

One other feature of View Composer is storage over-commit. This functionality is a configurable level of how the linked clones’ delta disk, or differencing disk, is allocated. Looking at a guest virtual machine, the delta disk would be a very small percentage of the parent or base VM. This setting will determine the behavior of determining how many VMs will fit on a datastore. A setting of conservative will enable less VMs to fit on a datastore, yet run less of a likelihood of running out of space. While a more aggressive level will attempt to put more VMs on the datastore and reserve less storage reserved for the delta disks. With that information, it is critically important to get an expectation of the delta disk behavior to best utilize the storage.

A final key component of View Composer is the Quickprep feature. Quickprep does the guest VM specific tasks such as domain membership, organizational unit placement in Active Directory and run any scripts on the guest VM.

With this information primer on VMware View Composer, it is important to isolate the vCenter and more importantly be aware of how the virtual desktop managment agents will interact with the vCenter server. More information on VMware View can be found on the VMware website.

VMware Converter failures due to block size

Recently during a P2V conversion, I had an issue where the conversion failed for reasons I could not explain. In looking at the logs of VMware Converter, there was no clear indication of the specific issue. Some of the relevant log entries are shown below:

[#2] [2008-11-08 08:16:32.953 'App' 1592 error] [managedVMCreator,1033] CreateVm task failed: Insufficient disk space on datastore ”.

[#2] [2008-11-08 08:16:33.171 'App' 1536 error] [imageProcessingTaskImpl,552] VmiImportTask::task{4}: Image processing task has failed with MethodFault::Exception: vim.fault.NoDiskSpace

After some deeper research, it turns out that this particular datastore was created with the default 1 MB block size, making VMDK files limited to 256 GB. Fellow SearchVMware.com blogger Eric Siebert has an excellent post on this site about the formatting options of the VMFS volume as it is brought into VI3. When you add a LUN, you select the block size during the format shown in the figure below:

While running VMware Converter on very large volumes is rare, this was one that made me stop and think for while. There are plenty of strategies for very large storage for a P2V conversion, but this was one where the other LUNs were configured with the 1 GB block size and that was fine thus far.

Luckily, there was space that I could evacuate all other virtual machines on that LUN so that I could remove the LUN and format it with a larger block size to get the VMDK file size I needed. The tricky part of this situation is that with VMware Converter experiencing the error on the block size, it was displayed in the log in a way that made it a little tricky to discover.

Replacing a VMware ESX SAN

Replacing or upgrading a SAN is no trivial task. There are a few tried-and-true steps to take when replacing a SAN which I’ll outline in this blog post, including a key step to the process that will ensure a successful switch.

I recently upgraded from an HP MSA 1000 to an IBM DS3400 because I wanted to improve performance and lower my overall energy costs.  One of the reasons I decided to replace my old SAN is because it is much cheaper to have a single 2U device running than the three devices for the old SAN. In addition, I dropped from 42 drives to 12 drives with more storage. Minimally my SAN power costs should drop to just 1/3 the original. I also have gone from 2.5 TB to 3 TBs of storage. Not a huge increase in storage capability.

I will know next month if my energy cost reductions have been realized and will report back then.

The steps for replacing a SAN are not all that tricky, but there was a single gotcha that could be avoided with careful planning. To replace a SAN, follow these steps:

1. Plug in the new SAN to your existing fabric. Luckily I had a pair of unused fibre connections and Gbics available else this would have been another expense and a delay until the cables and Gbics arrived.

2. Find a system on which to install the management console. For the IBM DS3400 I chose my VirtualCenter and VMware Consolidated Backup (VCB) server to be the management console for the SAN. There are two methods to manage the IBM DS3400: in-band or over the fibre channel fabric, or out of band using Ethernet — even a VM would suffice given networking is connected to the SAN. Software exists for both 64-bit Linux and Microsoft Windows.

3. Create the LUNs on the new SAN. This is a good chance to correct any problems you may have with the LUN configuration on the old SAN. I did a one-to-one mapping, except I slightly increased the size of the LUNs.

4. Present the LUNs to your VMware ESX host(s) and VCB server(s).

5. Rescan the storage adapters for new LUNs using the VMware Infrastructure Client (VI Client) for the first VMware ESX host. Once this is completed, you can then add as many Virtual Machine File Systems (VMFSs) as required.

6. Rescan the storage adapters for new LUNs and VMFSs using the VI Client on all the other ESX hosts.

7. Employ Storage VMotion via the VI Client to migrate VMs from one LUN to another LUN. This works if you have the patience to move all the VMs one by one. If not you can employ other measures. If you do this, however, you will end up having to edit the VMX files for each VM migrated to change the location of the virtual disk files. There are scripts to do this for you as well. This second option, however, also requires you to power off all VMs. Use of Storage VMotion does not require any VM downtime. Be sure to move all files from the LUNs in use.

8. For a LUN with an RDM (mine was a Linux file server), use Storage VMotion to move any VMDKs related to the VM. Then map the new RDM to the VM. You will have to reboot the VM to complete. Then create a new filesystem on the new RDM mount the file system. Then you must copy all the files from the old RDM to the new RDM. I used the following command to complete this task to copy all files from /files to /files2.

• cd /files; rsync -ravlpog * /files2

9. Then I modified the mount point for /files within /etc/fstab to be the correct new location. Finally I powered off the VM, deleted the old RDM from the VM and powered on the VM picking up the new data.

Here is the gotcha. I missed it, but it will be extremely useful for you (and me) going forward: Remove the old SAN’s LUNs from each VMware ESX host. If you miss this step when you finally disconnect the old SAN, the ESX hosts will go into a state of constantly attempting to failover the old LUNs. This will spew massive failures into the log files. If this happens there is no recourse but to reboot the VMware ESX hosts.

Now the SAN has been replaced. With the exception of dealing with any RDMs, it is possible to migrate to a new SAN without any downtime.

VMware updates storage and SAN compatibility guides

On October 1, VMware posted two important documentation updates related to the storage and compatibility guides for VI3 environments. The most visible indicator is that VMware has split the compatibility guides for ESX Server 3.0.X and ESX Server 3.5 including ESX Server 3i into separate guides. Both guides are available from the VMware website as a PDF. Here are links to both the ESX Server 3.0.X and ESX Server 3.5 and 3i guides.

The support matrix for storage and SAN configuration is an absolutely critical component to planning additional storage purchases or expanding current environments. Among my small circle of peers, I have been a little critical of VMware for releasing documentation that covers ESX 3 in a blanket format. This is a big step in the right direction, as the supported environments and their functionality vary by platform which was the crux of my frustration with blanket documentation.

This split in documentation is likely due to ESX 3i and Storage VMotion related supported environments. At first glance at the two guides they seem similar, but the following was taken from the 3.5 and 3i guide:

You will note that this guide is sparsely populated at present. The reason for this is that storage arrays require re-certification for ESX Server 3.5 and ESX Server3i, and while many re-certifications are in process or planned, relatively few have been fully completed to date. In contrast, servers and I/O devices do not require re-certification.

While this is somewhat of a surprise for a nearly 11-month-old product line, I still welcome the split documentation. Be sure to check these guides when making storage related infrastructure decisions, as they change frequently and based on the excerpt above should be updated. VMware’s supported configurations for storage are important to not only deliver a solution that works as expected, but to lay the framework for the virtual machine file system or VMFS, which I touched on during a prior blog post of why the proprietary drivers are important.

VMware Server 2.0 Release Candidate 1 offers hot new features

The VMware Server 2.0 RC1 version, released on July 1, has six new features previously not available in the beta releases. These features include:

-The use of Volume Shadow Copy Service (VSS) for snapshots of Windows VMs
-More efficient communication between the host and guest or between the guest and multiple guests on the same host using the Virtual Machine Communication Interface (VMCI)
-The ability to add new disks to running VMs
-Guest VM direct SCSI support for devices such as a tape drive
-Additional browser support for VMware Infrastructure Web Access to include FireFox 3
-Redirection of devices on the guest VM to the web client

I had a chance to install the RC1 version, and the new features are welcome additions from the previous betas. One of the new features that I want to touch on is the ability to add a new disk to a running VM. This feature allows an administrator to add storage to the VM’s inventory with the normal options of disk size, disk type and the option to use an existing .VMDK file. The figure below shows the wizard within the web interface to add the hard disk to the running VM:

Once the wizard is completed, the drive is available to the operating system. At that point, different operating systems handle the arrival of a hard disk differently. The figure below shows the new hard drive on Windows Server 2008 after a rescan task:

In the example used with the VM above, the hard drive can be added and used in the operating system immediately. Other VMware products, such as ESX have had this functionality, but it is the first time for the Server product. This feature is a welcome addition to the free product series as many organizations use VMware Server in a live workload capacity.

More information on the VMware Server 2.0 RC1 version can be found in the release notes on the VMware website.

Proprietary ESX drivers explained, enlightened

You may initially be disappointed that VMware ESX does not allow you to install drivers for storage and network controllers (depending on your environment). I have determined why this is, and it is for the better. Let’s go through why this is, what you loose and what you gain.

The drawback to ESX using the proprietary drivers is that tools that go along with your environment may not be available. This is most visible in the storage arena. For environments using a SAN, certain commands that are part of your driver on physical servers that are SAN-attached are not available. For both storage and networking, the esxcfg series of commands can usually get you the information you need if it can not be done within VirtualCenter. For me, this was initially a frustration, but then I saw the light.

The proprietary drivers are really a necessity for the VMware virtual machine file system (VMFS), which in my opinion is the most underrated technology provided by VMware. The benefit of using VMFS outweighs the learning curve required for the esxcfg series of commands.

One of the biggest issues I had when transitioning to VMware from general purpose servers was the storage provisioning. When the storage administrator would present storage to the VI3 environment, the logical unit number (LUN) serial number was the key identifier to the storage. Without the native SAN tool, determining the LUN serial number was a challenge which I shared earlier in a tip on this site. After a small investment in the commands available, I was able to address all required functionality within ESX.

Consider the benefits of VMFS coupled with the use of the proprietary drivers to enable other features such as traditional VMotion and Storage VMotion. More information on VMFS is available in the VMFS best practices white paper.

Adding a remote datastore to VMware Server 2.0 beta

I have been evaluating the VMware Server 2.0 beta on Windows and CentOS Linux systems since the release in November of 2007. The second beta of Server 2.0 was released in March, and both versions support adding remote datastores for guest virtual machines. Adding datastores can be a great way to store your lesser priority virtual machines without immense storage requirements on your VMware server system.

I have used the 1.0x version of VMware Server for years in testing and development systems. With version 2.0, I have started to use remote datastores for storing test systems. Datastores are storage locales in VMware Server 2.0. You can add NFS datastores for Linux-based installations of VMware Server. The Microsoft networking server message block (SMB) is available for Windows versions.

From a performance and configuration perspective, a non-local datastore is not ideal for live production systems. For situations like mine where a large number of infrequently used virtual machines are used from VMware server, a remote slower-speed disk suits the need very well. This storage configuration may also be good for archival of certain virtual machines, such as a project build that has gone into support mode.

In my example, I used a network attached storage (NAS) device, which can provide some of the cheapest storage available. NAS default configurations usually provide Windows file and print as a native file-serving option. Adding datastores from the “add datastore” command in the web interface after the initial installation is straightforward:

Once this is done, a virtual machine can be added to the local inventory. When virtual machines run from a remote datastore, the disk I/O is executed on the remote system. Adding a remote datastore also puts the CPU, network and memory functions on the host locally. There is definitely a performance hit from this configuration. But for archival purposes, you can free up more capable storage for your most frequently used virtual machines.

Administrators should also note that the virtual machines must be updated before being visible in 2.0 from a remote datastore before you make a remote datastore for all of your existing servers running VMware Server 1.0x from your VMware Server 2.0 beta system. Once updated to virtual machine version 6.0, the 1.0x systems cannot use the guest virtual machines without being upgraded to VMware Server 2.0.

The following section illustrates how to add a virtual machine to the inventory after upgrading:

This can save time by possibly eliminating the need to copy large virtual machine files back and forth over your network.