Storage

Making a P2V conversion: Tricks for systems with large storage

Converting a system with a large amount of locally attached storage can be a challenging task given the time required to perform the conversion. Here area a few tricks I’ve found that can help ease the pain on these types of conversion tasks.

• Private network: Making a private network between the physical host and the virtual host can provide two benefits. The main benefit is that conversion traffic will be isolated from the rest of the network traffic; the other advantage is that there is no risk of a user or process connecting to any resources and making changes during the conversion. The downside is that there may be special host-side configuration for a temporary network presence to allow the special network.
• Direct LUN mappings: For virtualization platforms that allow guest VMs to access a LUN directly, it can be much easier than performing a lengthy conversion of large data volumes that are already on a storage area network (SAN) and mapped to a physical system. Here is a blog post with a little more detail on that topic.
• Housekeeping: If there is junk on the physical system, does it need to be converted to the virtual environment, which may have more expensive storage? Clean up the candidate’s file system, and perform obvious tasks like emptying the Windows Recycle Bin. This allows for a more accurate re-sizing of the drives during the conversion.
• Agent backup and restore: For standard file volumes, such as a file server, it may make more sense to only convert the system drive and perform an agent-based restore to the virtual machine for the additional volumes. This does not necessarily save time from the entire conversion, but saves time within a tool like VMware Converter.
• Get a good time estimate: If you have to go at the large storage system as-is, make sure you have a baseline of about how many GB can be converted per hour. A good way to test this is to convert a good candidate system of about 100 GB and use that as a multiplier for your environment. There are a lot of factors, such as network speed and traffic, virtualization platform, storage systems (on both ends), and the conversion mechanism used. This allows for a good estimate on any downtime that needs to be coordinated if this applies to the selected workload.

These tricks can make converting a large amount of storage a little less daunting. What tricks have you employed to tackle physical systems with large amounts of storage in the course of being converted to a virtual machine? Leave a comment below and let us know.

Still mulling over a Greene-less VMware

It’s been covered to death, but something about Diane Greene’s ousting from VMware’s top spot still doesn’t sit right with me. Not the ousting itself but the chatter about why. There have been conversations about why she was let go, ranging from EMC’s CEO Joe Tucci wanting greater control of VMware to questions about whether she was more of a technology person and less of a business person. In the end, the appointment of Paul Maritz is the really big news, at least in my not-so-humble opinion.

It goes back to “it’s not what but who you know,” and Maritz knows Microsoft. He knows Ballmer, Gates and every other player there. He was one of the most influential and instrumental executives in Microsoft’s history. His reach is wide when it comes to pulling people into the fold — not necessarily by bringing ex-Microsoft folks in as employees, but rather by having high-level working relationships with all the partners that Microsoft has worked with and that EMC and VMware have worked with or would love to work with. He also knows the PC Revolution firsthand, having seen the rise and fall of Novell’s NetWare, Banyan’s VINES and the host of minis and mains that these replaced, only to be replaced by Windows a few years later.

Tucci also knows Microsoft — EMC’s storage products center around the Microsoft world as much as any other operating system. Exchange data stores, SQL databases, file shares — all of these are EMC’s bread and butter in selling storage to the modern data center. Its software, even though some products compete (like Documentum versus Sharepoint PS), is built around a Windows-centric world.

Then there’s the history — Microsoft knows how to win. It buys what it can’t make on its own, then drowns the competition in price wars and advertising battles. Novell, once Microsoft’s bitter rival for network OS sales, now sells Linux licenses to Microsoft. Netscape is gone and the ghost of its second cousin twice removed, the Mozilla Foundation’s Firefox, lives on to take what is really an insignificant chunk of Internet Explorer’s market share. Corel/Novell WordPerfect? Only if you’re working in a huge law firm will you see WP on an enterprise level.

Put these together and the fabled VMware versus Microsoft Hypervisor war starts to look less like an armed conflict between bitter rivals and more like a strategic partnership built through a demonstration of independence. Tucci’s no fool — Maritz is there for the day that the Redmond giant comes knocking. He’s there to build thin but sturdy roads between the two companies. He’s there to forge something like the Citrix/Microsoft alliance, where Citrix is an independent company but still acts in many ways like a subsidiary of Microsoft (or at least an extension). In Martiz’s VMworld keynote speech (not the parts about having “sins to atone for” in his early days of programming for the PC Revolution), he barely mentioned Greene and hardly touched on competition with Microsoft. He’s looking forward to the day when he can do what only Citrix has managed to do so far — preserve independence while under Redmond’s all-seeing eye.

In the end, we’ll see VMware’s VDC-OS as the dominant force in the virtualization space with Hyper-V as an acceptable but lesser alternative, much like Citrix’s MetaFrame/XenApp and Microsoft’s Terminal Services. I think this leaves one question: In the long run, what happens to Citrix now that it’s betting so heavily on Xen and taking on Microsoft and VMware directly in the systems virtualization market?

Take the time to learn direct disk access to a virtual machine

Recently I had the opportunity to go through a series of tests revolving around the use of a raw disk or dedicated logical unit number (LUN) to a virtual machine. I think that any virtualization administrator should go through the drill. The basic principle is to seamlessly move storage on a storage area network (SAN) from a physical system to a virtual machine. This can accommodate an emergency physical-to-virtual (P2V) conversion of a system with large storage, saving costly time on a system conversion.

The experience I reference was a VMware ESX Server and VirtualCenter configuration with Windows Server 2003 systems. With a LUN made available to a physical system, moving that storage over to a virtual machine was actually quite easy and uneventful. There are a few pointers to remember when doing this configuration, namely because the storage is not residing on a virtual machine file system (VMFS) LUN, it will not be eligible for a lot of VMware’s management and high-availability (HA) functionality. This includes Storage VMotion, host-to-host VMotion and HA features for failover onto another host automatically.

Taking the time to become familiar with the process and expected functionality is a worthwhile investment. This is a configuration that may not be something that you foresee using, but it may come in handy. The figure below shows a virtual machine configured with two LUNs made available to the host and assigned to a virtual machine:

One important step is not to add storage as you normally would for a VMFS volume. This will destroy the contents of the drive and make your preserved data unreadable. In the configuration shown below, the drive arrives seamlessly to the guest operating system and will be ready for use. There may be a drive letter change, but that is an easy correction.
Most storage systems and virtualization platforms should be able to support this functionality in some capacity. Taking the time to get the specifics down in your storage environment can avoid any surprises if this configuration is required due to an unplanned migration

VMware defends its upcoming fault-tolerance feature

During VMworld 2008 in Las Vegas last week, VMware Inc. announced its upcoming fault tolerance feature and gave a demonstration of it during one of the keynote sessions. It looked pretty good and simple to use, but Littleton, Mass.-based Marathon Technologies Corp., a company that specializes in fault tolerance software, had plenty to say otherwise.

In response to Marathon’s blog dissin’ the upcoming feature, Palo Alto, Calif.-based VMware’s Mike DePetrillo, a principal systems engineer, wrote a blog defending VMware Fault Tolerance.

For starters, Marathon complained that VMware does not provide component-level fault tolerance. “The most common failures that result in unplanned downtime are component failures such as storage, NIC [network interface card] or controller failures. Yet VMware Fault Tolerance doesn’t do anything to protect against I/O, storage or network failures.”

DePetrillo noted that VMware already has features to protect again component failure. “If your NIC fails you’ve got NIC teaming built into the system. To set it up simply plug in both NICs to the server, go into the network panel and attach both of them to the same virtual switch. Done. Four clicks. Same thing for storage with the built-in SAN [storage area network] multipathing drivers,” DePetrillo wrote. “I absolutely agree with the author that component failures are the cause of most crashes and that’s why VMware added these features in 2002. VMware FT is not designed for component failure because there’s no sense in moving the VM to another host if you’ve simply lost a NIC uplink. NIC teaming will take care of that with ease and is a LOT cheaper than using CPU and memory resources on another host to overcome the failure.”

Marathon’s second beef: VMware’s fault tolerance is too complex. “In order to use VMware Fault Tolerance, you’ll first have to install both VMware HA [High Availability] and DRS [Distributed Resource Scheduler]. No small feat in and of themselves. Then, because VMware FT requires NIC teaming, you’ll also have to manually install paired NICs. Then you’ll need to manually set up dual storage controllers (with the software to manage them) because it requires multipathing. And to top it all off, you’re required to use an expensive, and often complicated, SAN.”

DePetrillo said the process requires checking off two boxes - HA and DRS. That’s it. “If that’s too hard then please comment and let me know how it could possibly be easier. Even my dog has figured out how to do this now. Granted, it’s a pretty smart dog.”

“As for setting up the dual NICs and dual HBAs [host bus adapters], well, yes, you have to actually plug the physical devices in. After you’ve done that the **built-in** NIC teaming and HBA drivers will take over and configure most everything for you. The NIC teaming does require four extra clicks. The HBA drivers actually figure out the failover paths, match them up, and set up the appropriate form of failover all auto-magically. They’ve been doing this since ESX 1.5 (6 years ago),” DePetrillo blogged.

“Lastly, yes, this requires shared storage. Pretty sure that most environments that want FT (no downtime what-so-ever because our business could lose millions) already have a SAN to take advantage of other things virtualization related such as DRS and VMotion,” he wrote.

Also, VMware FT does not require dual NICs or dual HBAs because, DePetrillo said, “This is something you should have in every virtualization setup that’s running VMs you care anything about, but it’s not a requirement to get VMware FT [Fault Tolerance] running.”

The last point Marathon makes that’s worth spending any time on is that VMware  offers onlylimited CPU fault tolerance. “With VMware FT, you’ll need to set up what VMware refers to as a “record/replay” capability on both a primary and secondary server. If something happens to the primary server, the record is stored on the SAN and then restarted on the secondary server. … The whole thing depends on the quality of the SAN. Second, in the words of the VMware engineer who presented at VMworld, “this can take a couple of seconds.” So what happens to your application state in those couple of seconds?”

DePetrillo’s defense is that “if you’re the type of company that requires absolutely no downtime for an app — if the app is just that critical — then I’m pretty sure you’re going to have a decent SAN. … If you’re having so many problems with your SAN that you don’t trust it for FT, then you have much bigger issues at hand that VMware or Marathon or any of the other virtualization related vendors aren’t going to help you with.”

You can read more of VMware’s comments on DePetrillo’s blog, which gets into some details on how VMware Fault Tolerance will work, and vice versa for Marathon.

But I think it is obvious that Marathon is making VMware’s fault tolerance feature seem worse than it is, and VMware is making its new feature seem simpler than it is.

For the most part, this is a pissing contest between the incumbent fault-tolerance vendor and the “new guy,” but the fact of the matter is, if you use VMware virtualization, you can’t use Marathon Technologies because they don’t support VMware (obviously) and if you use Citrix Systems’ XenServer, you can’t use VMware Fault Tolerance, so these arguments are moot.

Getting to know Sun xVM VirtualBox snapshots

Desktop virtualization packages rely on snapshots and virtual drive functionality. The de facto functionality standard here is found in VMware Workstation and VMware Server, but the tools in Sun’s VirtualBox may be setting a new standard. Let’s take a quick look at how snapshots and virtual drives work within Sun xVM VirtualBox.

VirtualBox snapshot technology provides the same basic functionality as the VMware products in that they can be taken while the virtual machine (VM) is running or offline.  The snapshots are taken from two different places depending on the state of the VM. For a running VM, the snapshot is taken from the running console as shown in the figure below.

When a VM is powered off, snapshots may be taken in the properties of the VM. This difference is a slight inconvenience, but is an easy learning curve to overcome. Further, if a VM needs to revert to a saved snapshot, this same location is where the VM would be reverted. VirtualBox gives the option to build from the snapshots, so there can be multiple point-in-time restores for a single VM. Snapshots in VirtualBox are kept in the .VirtualBox\Machines\VMName\Snapshots location by default, and are a collection of .VDI and .SAV files. The figure below shows three point-in-time restores for a single VM:

As with all snapshot restores, you should be sure that you want to restore as the reverting process is authoritative to the VM. Reverting to a VirtualBox snapshot taken while the system is running reverts precisely to that point with the VM running, rather than a powered off state. Overall, the functionality inventory of VirtualBox snapshot functions as advertised and brings another positive view to this exciting virtualization platform.

More information on the VirtualBox 1.6.x product can be found in the online user guide.

Ensuring disk resources with SCSI reservations

You may hear the term SCSI reservations frequently when dealing with VMware servers that utilize shared storage. SCSI reservations are used to ensure exclusive access to disk-based resources when multiple hosts are accessing the same shared storage resources. In addition to being used by VMware hosts, SCSI reservations are also used by Microsoft Cluster Server.

SCSI reservations are only used for specific operations when metadata changes are made and are necessary to prevent multiple hosts from concurrently writing to the metadata to avoid data corruption. Once the operation completes the reservation is released and other operations can continue. Because of this exclusive lock, it is important to minimize the concurrent number of reservations that are made. When too many reservations are being made at once, you may receive I/O failures because a host is unable to make a reservation to complete an operation because another host has locked the logical unit number (LUN). When a host is unable to make a reservation because of a conflict with another host, it will continue to retry at random intervals until it is successful; however, if too many attempts are made the operation will fail.

Some examples of operations that require metadata updates include:

• Creating or deleting a VMFS datastore
• Expanding a VMFS datastore onto additional extents
• Powering on or off a VM
• Acquiring or releasing a lock on a file
• Creating or deleting a file
• Creating a template
• Deploying a VM from a template
• Creating a new VM
• Migrating a VM with VMotion
• Growing a file (e.g., a Snapshot file or a thin provisioned Virtual Disk)

Having a minimal amount of reservation conflicts is generally unavoidable and will not have a big impact on your hosts and VMs. To avoid having too many conflicts, try to limit the number of operations that can cause reservations and stagger them so too many are not happening simultaneously. All reservation errors are logged to the /var/log/vmkernel log file on each ESX host. To reduce the amount of conflicts:

• Limit the number of snapshots you have running, as snapshots grow in 16MB increments and every time they grow they cause SCSI reservations.
• Only vMotion a single VM per LUN at any one time.
• Only cold migrate a single VM per LUN at any one time.
• Do not power on/off too many VMs simultaneously.
• Limit VM/template creations and deployments to a single VM per LUN at any one time.
• Consider using smaller LUN sizes (<600GB) and do not use extents to extend a VMFS volume

Symantec, Citrix take on VMware with block storage management product

On Monday, June 9, Symantec Corp. of Cupterino, Calif., announced the release of Veritas Virtual Infrastructure (VxVI), a server and storage virtualization product built on Citrix Systems Inc.’s XenServer technology. By exploiting Veritas’ block storage management model, VxVI hopes to compete with VMware-Infrastructure-3-in-production environments by offering increased capabilities for storage and availability-critical systems.

The new Xen-based virtual infrastructure platform from Symantec provides storage management and high availability with cross-platform connectivity for the virtual data center. It’s essentially XenServer with the Veritas storage management layer on top — all wrapped in a Symantec management console.

According to Symantec Senior Vice President of Storage and Availability Management Rob Soderbery, the time is right for a product that addresses the needs of testing and development, needs that have been underserved by VMware. “Users understand the storage management challenges with VMware,” he said. Symantec has delivered something “fundamentally new” in how server virtualization works with storage management, he noted.

The key difference between VMware and Veritas is in how each handles virtual machines (VMs). Soderbery argues that the Virtual Machine File System (VMFS) file-based system that VMware uses can’t compete with the block storage system of Veritas VxVI.

As enterprises build out the x86 data center, Symantec’s product seeks to serve those who want to bring physical capabilities into the virtual environment. Veritas Virtual Infrastructure brings dynamic storage layouts, enclosure and array mirroring and storage-area network (SAN) multipathing/load balancing to server virtualization, adding features such as shared VM boot images with which Symantec hopes to lure VMware customers that are not satisfied with the storage capabilities of the leading server virtualization platform. 

Soderbery says that VxVI will work well with Microsoft’s forthcoming Xen-inspired hypervisor, Hyper-V. “Microsoft has done something pretty interesting here in being open to the Xen community and encouraging the Xen community to be open with Microsoft,” says Soderbery. “Veritas Virtual Infrastructure is technology that we can apply across the Xen ecosystem and Hyper-V as well.”

With another Xen-based virtualization product on the market engineered to be more compatible with the forthcoming Hyper-V, VMware may feel the pinch as users see more options with the other big players in the server virtualization market. But will the $4,595 per two-socket server for Veritas discourage VMware users from even running a demo?

What do you think? If you plan on deploying Veritas VxVI, we want to hear from you. Send us your thoughts via email.

Storage utilization is a new battle

I was recently asked, “do you have any visibility of the storge utilization you provide your virtual machines?” I stopped, thought about it and said “no”. However, in my situation, this is not yet a problem.

A pitfall for most enterprise server virtualization strategies is in a reservation for storage, regardless of what the virtual machine has written on the virtualized filesystem to the defined maximums. For example, if I have a base installation of a Windows Server 2003 system, the footprint as I do my server builds will be around 5 GB. My standard build allocation is 32 GB. This makes this system only 15.6% utilized from inception. This rule of thumb applies to most servers, and a standard build has 32 GB as an accepted footprint per system. 

Excluding backend storage virtualization and de-duplication strategies, what about systems that have a storage footprint larger than 32 GB? Well, luckily we’ve been down this path before:

The storage is the storage, virtual or physical.

Managing the percentage of utilization for shared storage should be a task of continuing diligence. I don’t (yet) have a large number of virtual servers with a footprint above the standard build, these systems face the same battles we have had for years with general purpose servers.  As an example, take a main file and print server that is 2 TB on a general purpose server: It will be about 2 TB on a virtual server as well from the storage perspective. For large storage footprints using iSCSI or storage-area network (SAN) technologies, the difference in configuration is minimal.

However, how do we address the first question about under-utilized storage footprints for the virtualized systems? Is it best to look only at operating system metrics? That may be an adequate solution for each operating system, but the aggregation will be from different sources and outputs. What are you doing to address storage utilization when you are not using storage virtualization?

VMware white paper review: “Comparison of storage protocol performance”

Since I just love to read white papers (n.b., sarcasm), I grabbed a copy of VMware’s Comparison of Storage Protocol Performance. Actually, I found it to be a good read. It’s short and to the point. This sums it up quite nicely:

“This paper demonstrates that the four network storage connection options available to ESX Server are all capable of reaching a level of performance limited only by the media and storage devices. And even with multiple virtual machines running concurrently on the same ESX Server host, the high performance is maintained. “

The big four storage connections are:

• Fibre Channel (2 GB was tested)
• Software iSCSI
• Hardware iSCSI
• NFS NAS

The paper infers that network file system (NFS) is perfectly valid for virtual machine (VM) storage, performing in all of the tests at a level comparable with software iSCSI, very close to hardware iSCSI and lagging behind 2 GB Fibre Channel (FC). This doesn’t surprise me one bit: I like NFS network-attached storage (NAS) for VM storage. I prefer storage area network, or SAN-based storage because I prefer to store on a virtual machine file system; but for low-criticality VMs, NAS’s price is right (well, as long as you don’t count Openfiler, IET, etc.) Also, it’s plausible to build out a virtual infrastructure storage architecture using nothing but Fedora Core and be supported.

I was particularly interested in the FC vs. iSCSI performance results presented in this VMware white paper. At the lowest end of the scale, iSCSI beat FC. Granted, the low end of the scale isn’t what will be seen in most production environments but it is  interesting data. What I liked most was that nowhere did 2GB FC truly outclass 1Gb iSCSI. It was faster in most of the higher I/O testing, but never did it double the performance. 2 GB FC did show a big performance improvement in the multiple VM scalability test but not double (about 185 MB per second vs. about 117 MB per second).

On to what I didn’t like in this white paper:

• No 4 GB FC comparisons. 4 GB FC is the sweet spot for high-performance enterprise SANs being put in place to support the big iron now being virtualized. It should have been covered, even if it is still a little bit of a nascient technology (well, not in terms of maturity but in terms of it’s market segment.)
• No 1 GB FC connections. (There are still plenty out there.)
• No NIC Teaming comparisons. I want to know how much additional CPU overhead is involved. I want to know how much performance is improved if you team NICs on your software iSCSI targets and initiators.
• No multipathed comparisons. This should have been done. Mutipathing is a way of life for anything as mission critical as a server that hosts multiple servers.
• No 10 GB Ethernet iSCSI comparisons. VI 3.5 is out. 10 GB Ethernet support is built into VI 3.5 (see the HCL, page 29.) Not to test this is a big oversight.
• No internal-disk storage was tested. Ok, maybe it’s not reasonable for me to expect this to be tested. Maybe I’m just grouching now.

I was surprised to see that software iSCSI got its tail handed to it in CPU workload testing. I’ve never done this testing but I knew there was a big overhead involved. I just didn’t expect it to be that big, especially compared to NAS, which I expected to be right there with iSCSI rather than much more CPU efficient (FC was the 1.0 baseline, NAS scored about 1.8-ish 1.9-ish, and SW iSCSI was about 2.75.) This means one thing: while performance is great across all protocols, plan on extra CPU power for software iSCSI.

I was pleasently surprised to see hardware iSCSI dead-even with 2 GB FC. I had expected some additional overhead even with dedicated hardware, but that wasn’t the case. I would expect to find that in a dedicated iSCSI solution–unless you’re using really cheap equpment like hooking up a couple of big drives to that old desktop–you won’t hit the CPU-use ceiling unless you fail badly at planning.

All of these protocols are perfectly valid. There could have been more meat in the paper, but it did a good job of accurately testing four of the most common storage architectures used with VMware’s products.

Overall, I give this white paper seven “pokers.” Why pokers? Because stars and 1-10 ratings are common. Pokers are mine. Because fireplace pokers can jar you into action if you get bit by one, seven pokers means you should read this paper if you have any responsibility for virtualization.

What to do with internal disk space on VMs

If one thing annoys me to no end, it is unused capacity.

That’s why I like virtualization. And it’s also why I like grid computing. Heck, that’s why I like cheeseburgers (there’s no empty space in my stomach after a trip to In-and-Out.) Virtualization makes efficient use of existing hardware to control costs. VMware environments often have host servers with more than a small RAID1 array in them because they were existing servers retasked as ESX hosts. Sometimes this space gets used for ISO file storage, sometimes it gets used for virtual machine toys (so called gray- or black- boxes) and sometimes it’s used for production VMs. Labs are often set up on the local storage with machines that have no production value hosted, which makes great use of that space (but perhaps not as good of use of CPU or memory for production machines).

A case study in space, storage and VMs
Then there are times when I walk into sites like the one I saw last week. They had a virtual-machine iSCSI SAN set up on each ESX host homed to the local storage. This was in addition to their FC SAN, by the way. They even ran part of their production environment off of it, using the unused internal disk space on the ESX servers to store virtual appliances that ran iSCSI Targets in them, similar to what I described in an earlier post. What a great use of space. Kudos to them!

I was concerned about how they were to keep those virtual machines up in the event of host failure. First I was told that they put in a poor-man’s round robin, which is to say, host 1 has iSCSI SAN 1, host 2 has iSCSI SAN 2, 3 had 3, and 4 had 4, and they all replicated to each other; VMs hosted by ESX 1 were on SAN 2, those on ESX 2 were on SAN 3, those on ESX 3 were on SAN 4, and those on ESX4 were on SAN1. Then, anti-affinity rules were used to prevent VMotioned VMs from winding up on the same host as the SAN on which their files existed. The replication prevented any single SAN failure from becomming a nightmare. They hadn’t done anything unusual on the network side though, which bothered me (I would prefer dedicated physical NICs for the SAN VMs!), but their performance testing showed no need for the extra NICs to be added. It’s a little hard to follow-the-leader, but it worked reasonable well. This was done with a variety of open-source packages, some of which I had never seen before. I recognized IET right away, but the SAN appliances were all custom builds, and it took me some time to figure out what was what and what was going on where. It was not an efficient use of that internal disk space because of all the replication across servers being, essentially, mirrors of mirrors.

Ingenious? Yep. Complicated to track? Yep. Functional. Yep. Needing of a little less work to manage? Yep.

There are commercial products to do this, one of them is LeftHand Network’s Virtual Storage Appliance. I’ll post more about my experiences with this very soon.