April, 2008

Announcing the VI Toolkit for .NET

VMware resource website l o s t c r e a t i o n s has just announced the immediate availability of the VI Toolkit for .NET. From the projects webpage:

The VI Toolkit for .NET is a .NET library that is patterned after the VI Perl Toolkit. It makes connecting to VI, finding a VM, and getting its properties a snap! The toolkit is written in C# 2.0, but can be used by any language capable of loading .NET 2.0 assemblies. One of the reasons that the VI Toolkit for .NET is so powerful is because it is the first .NET application to make use of the recent .NET port of the Apache Commons CLI library for parsing command line options and their arguments. We know it is the first, because we ported CLI! Administrators and developers who like the ability to quickly create scripts with the VI Perl Toolkit but appreciate the depth of .NET will love the VI Toolkit for .NET.

VirtualCenter custom roles for delegated access

The built-in roles for access to VirtualCenter and the managed objects are okay for many common scenarios, but some situations require additional configuration. In some cases, creating custom roles has been a viable solution. Here is how I created a role that would allow a user only to view a virtual machine’s console.

Create the role and deployment model first

Before permissions are assigned, some thought should be given to user rights and how they would be administered. In most VirtualCenter environments, the permissions would be retrieved from a Windows Active Directory domain. To make that process easier, all permissions should be assigned to VirtualCenter through Windows groups.

To create a role or modify an existing role, select the Administration button within the VMware Infrastructure Client (VI Client). From here, you can create or modify a role for your desired access. If you wanted to allow a user to view a virtual machine’s console, for example, you would create a custom role such as the specified permission below:

Applying the custom roles

The ability to view the console of specified systems can come in handy for certain situations, particularly when traditional network connectivity to the guest operating system is not possible for normal methods such as remote desktop or VNC. Creating a console view-only role would be done in the VIC on a per-object basis. A per-object basis is one in which you can assign the permission to view just the console to an ESX host, a resource pool, a data center or even a virtual machine individually. Roles to objects in VirtualCenter are always applied via the Permissions tab for the object.

To make configuration consistent, create a Windows security group with the same name in the Active Directory domain. For my custom role, the Windows group (MSS\VMSpecified-Roles-ConsoleOnly) and the VirtualCenter role (VMSpecified-Roles-ConsoleOnly) are assigned to the object below:

From this point, the clients can log into a locally install the VIC and connect per the specified permissions. Be careful, however, as a username that may have multiple roles would have the permissions of the combined roles. You can work with some propogation, but singular assignment would be a better practice. VMware provides a document fully outlining the roles architecture available for download from their website.

Audit trail of connections into the VirtualCenter

With this functionality, an auditing requirement is fully justified. Within the VirtualCenter database, you can monitor the authentication log-on and log-off events. A January SearchVMware.com ITKE post has this outlined well.

Fortune 500 company uses VMware to slash long term power costs

Santa Ana, Calif.-based First American Corp., a Fortune 500 company that provides business information and analytics, is standardizing its data center with virtualization software from VMware, Inc. to improve power efficiency and better server utilization rates.

First American’s Enterprise Architect, Jake Seitz, said the increasing cost of power was the most significant driver for virtualization.

“In Q407 we started looking at our utility costs and our data center footprint, and realized that embracing virtualization would help us maximize what we have and shrink our utility footprint,” Seitz said. “My hope is that we can get to 80% virtualzed environment – we are at about 30% right now.”

Low hardware utilization rates also prompted the company to invest in VMware virtualization. “We wanted to better utilize our existing hardware. We had some hardware utilization as low as 4%, and rarely would be see anything above 15% utilization,” Seitz said.

When Seitz began looking at virualization about two years ago, he also deployed small Microsoft Virtual Server and Citrix XenServer environments - both less expensive than VMware - but decided to standardize on VMware going forward because he liked the management tools and support.

When the project started, the data center at First American was equipped with 2,800 physical servers running that many OSes. After virtualizing 700 of those servers, First American runs 3,500 OSes, Seitz said.

The OSes are mostly Microsoft Windows, though there are some Unix systems on mainframes and in Xen containers, he said. The physical servers supporting the virtual machines are x86 servers rom Hewlett Packard and Dell, and many are equipped with Intel Xeon quad-core processors, Seitz said.

First American hasn’t gotten rid of many physical servers because they lease them, but it will save power in the long run because with better server utilization, they won’t have to add more physical servers in the future, Seitz said.

Also, because their hardware is leased, it is refreshed every three years. When this happens, the IT staff has to port all of the applications and OSes from the old hardware to the new hardware. Normally, this takes months to accomplish, but it should be a lot easier using virtualization, Seitz said.

“Now, instead of doing a one-to-one refresh, we will virtualize our environment and it will shave off a few months in man hours,” Seitz said.

The company is starting to virtualize their mission critical applications now, and have even adopted a “virtual machine first” policy.

“You need a really good reason to get a physical server,” Sietz said. “Everything we do now is virtual.”

VMware releases official plug-in guide

Congratulations to the VMware engineering team releasing the official guide to creating VMware plugins. Their awesome, hard work is culminated in the document Getting Started with VI Client Plug-ins. Several of you have already asked me how their official methods and plug-ins compare to my previously published guide and plug-ins. After a cursory review of the official guide, here are my thoughts:

- The VMware guide to plug-ins is official. Mine is not. Although the VMware guide is experimental, it is more likely to be supported by VMware than any plug-in you write using my guide. That said, I have heard that the team responsible for plug-ins plans on developing a shim that will continue to support my plug-in methods in the next version of VMware Infrastructure (VI). That is hearsay, however, and it could change.

- Official plug-ins function very similarly to mine (they should, they are built using the same principals), but they could be considered inferior in one very important manner. Although the user interface to activate a plug-in is the same (context menus, tabs, menu items), the interface for official plug-ins can only be a web page. For instance, a user right-clicks on a virtual machine and clicks on the context-menu item labeled Migrate storage which launches my Storage VMotion plug-in rewritten as an official plug-in. Instead of having a Windows form appear that maintains a consistent user interface, instead a web browser appears and runs a script or web application that has authentication information and object information passed to it from the VI client (much like my Invoke plug-in).

- Official plug-ins must reside on a web server to which clients have access. This can be considered a good thing and a bad thing. On one hand, it centralizes plug-in updates. On the other hand, if the web server is offline ESX admins could lose plug-in functionality that they have come to rely on. Plug-ins written using my guide reside on the local client; always online and accessible.

- Given how hard it is for developers to build consistent web applications between browsers (even the VI online web interface fails to work properly on Safari), it is a tall order to expect plug-in developers to create plug-ins that look the same between the four major web browsers available to Windows (IE, Firefox, Safari and Opera). That said, you could just force users to use IE since you know it will be there.

- One big win with the official method is that the plug-ins could be written to be standalone web applications. This means that they could be accessed outside the context of the client.

The official plugin guide is a great achievement on the part of VMware; it shows their commitment to giving users what they want. Ultimately, though, the plug-ins created with it are forced to be web applications or scripts hosted on a web server. The biggest problem with this is that it forces users to leave the consistent look and feel of the VI client, ripping them out of their experience. The official plug-in guide is probably best thought of as an online alternative to the Invoke plug-in, but not as a replacement to the plug-in architecture that I have already exposed.

Hopefully with VI4 we will see a more fully fleshed out, official plug-in architecture.

Update 4/22/08:

Carter Shanklin, Product Manager of End User Enablement at VMware, sent me an email with the following note:

One point you make is the inconsistency of browsers in rendering, etc. Inside VI Client, if you create a custom tab it will be rendered using an Explorer control, regardless of what your preferred browser might be. This may be problematic if you want to use the same interface from outside of VI Client, but from within the client it should look fairly consistent.

Carter is absolutely correct. These controls should be rendered within the VI client, however you still need to make sure your web application looks and feels like a part of the VI client. Just because your web application is displayed from within the VI client, it does not mean that the VI client is rendering your list box or text box with its own style sheets.

VirtualCenter virtual machine removal best practices

Creating virtual machines (VMs) is quite easy, but how you remove them can make a big difference in storage space.

When removing a virtual machine, one popular method is to move the virtual machine to a resource pool with no assigned memory or processor resources. There are some concerns and strategies for doing this, however there are no virtual appliances or fully automated solutions within the VMware Infrastructure Client to help you with this process.

While moving the VM to a different resource pool is a convenient way to roll the system back online if necessary, the VM would still have its storage requirements. From an ongoing management perspective, storage is the biggest concern. I have found that using the local storage for the tentative decommission stage is acceptable in lieu of using the shared storage configurations across the other ESX hosts.

Once the virtual machines are in the resource pool with no assigned resources, you can then remove them on an interval that suits the ‘just in case’ situations. This can be a daily, weekly or monthly timeframe — whatever suits your environment and business needs. Of course, if the storage situation inhibits keeping these systems for long amounts of time, then the interval may be dynamic.

What are you doing from a matter of practice standpoint for the deletion of your virtual machines? Share your comments and ideas below.

Is the mixed virtualization vendor data center possible?

I recently floated the idea of implementing a mix of virtualization products in your data center as a way to better customize your virtual environments. The query was part of a larger discussion I wanted to get going about how VMware will compete when Hyper-V is generally released. I threw out the notion that data center managers might use, for example, Hyper-V for end-user file servers; VMware ESX for apps that require dynamic load balancing, sophisticated disaster recovery and migration; and Xen for commodity Linux boxes.

The idea behind that supposition was to match your enterprise investments to appropriate workloads because, let’s face it, running everything on ESX is going to be expensive compared to other options. Big deal if you don’t get ESX-level features because you may get enterprise level features on silver-medal products.

Now, I wish I could take credit for that idea (for better or for worse), but that really came from SearchServerVirtualization.com’s editor, Jan Stafford. At any rate, a few people had comments about that note and I thought it would be fitting to continue that conversation with their input:

Hi,

One of the reasons I think Hyper-V is interesting is the server core concept. The ability to create appliance virtual machines for specific roles such as DNS, DHCP, DC etc. means you can drastically reduce the manageability overhead and the attack surface for those servers. Weird - I’m finding myself suggesting that an all-MS platform may actually be more secure than the alternatives!

On the other hand, you may be correct that fully-fledged servers with complex HA needs will be better off on ESX.

Cheers,
Dan Flower

There are flaws to this query, the base of which stem from whether ESX, Hyper-V and Xen-based virtualization products can even be compared; never mind that there are disparities between Xen-based virtualization vendors (xVM isn’t Red Hat isn’t Citrix Xen etc). This was the point that some folks took issue with:

Just a comment about an “apples and oranges” comparison you made:

I understand the point you were trying to make about using the right virtualization product for the right job, and at times this might mean using multiple solutions, but you can’t really compare Hyper-V or VMware to Xen. Both Virtual Iron and Citrix offer a virtualization solution based on Xen technology, but neither sells just Xen. Your reference to Xen brings to mind what a Suse or Red Hat shop might do with Xen technology, but not those that would consider Hyper-V or VMware as a virtualization solution.

David Roden

So, there are still many problems that have to be approached before we even ask the question of implementing a mixed enterprise virtualization infrastructure. Ultimately, people will have to decide on what functionality, support and management interfaces are important. Is quick migration support good enough, or do you need live migration? How does each platform approach P2V conversions? Does VirtualCenter have the right kind of management options for your installation? Will you be better off using System Center Virtual Machine Manager?

Let’s keep prodding at this idea and brainstorm ways that using multiple virtualization products in one environment could work. Send us your comments and feedback. If you’ve tried the mixed virtualization environment, we’d love to hear about it.

Double-check your HBAs on ESX 3.02 with IBM SVC connectivity

Like many administrators, I have been quite happy with the performance of my ESX environment. However, we recently had an observation that avoided a potentially disastrous issue.

In my environment, I am using an IBM SAN Volume Controller (SVC) for storage with 4 GB/s host bus adapters (HBA). The driver is proprietary to ESX for connectivity. I am currently running version 3.02, but I recently came across some unexpected behavior.

ESX was only using one of the HBAs for the SAN storage. We wanted to determine what would happen if that path was lost. Would ESX would continue operating as expected? So we performed the following tests and came up with these results:

-Dropped connectivity on first HBA / active path rolled to next HBA, port status ‘dead’
-Restored connectivity on first HBA / port went to ‘on’, active path remained on second HBA
-Dropped connectivity on second HBA / lost all connectivity

Yes, all connectivity was lost in the third step. It was better to correct this now before learning this the hard way. My expectation was that the connectivity would use both HBAs at all times and failover as needed. Luckily, this is easily corrected.

There are two ways to address this issue. One is to run a command to instruct ESX to use both HBAs, and the other is to apply an update. The first option would use the following command:

esxcfg-mpath –policy=rr –lun=vmhba1:0:1

This command would be run per LUN per ESX host, but is a very quick and easy way to address the functionality issue immediately and can be done outside of maintenance mode. This behavior is spelled out in VMware KB article 1003270 online. The solution is to install a critical-class patch to the ESX system to address this, as well as a few other issues. The native behavior in ESX 3.5 has this issue corrected with no updates or commands.

One simple way to see if your ESX host is using the different HBA’s is to look at the LUN properties. From the VMware Infrastructure Client, select an ESX host, select the Configuration tab, select Storage, right-click on a LUN, select Properties, click the Manage Paths button, and look at the path listed as Active. If it is always on the first path and that ESX host has a virtual machine running on that LUN, the host may not be using both paths. Below is a figure showing a LUN that is using the second HBA:

You can also run the following command to see who is active at that moment:

esxcfg-mpath -l

The far-right column has the role of active and preferred assigned to a path within a LUN. If the active designation never leaves one path where there is an active virtual machine, you may be at risk of the behavior we observed initially. This makes a case for ESX host patching, as well as ensuring that all redundant components function as expected during installation.

Use the VMware Converter bootable CD for the difficult conversions

Depending on the scope of your virtual environment, it is likely that physical-to-virtual (P2V) conversions have taken place. The P2V process truly enables VMware administrators to put physical systems into virtual environments. However, you may have come across a system that for some reason will not go through the normal conversion. In such cases the VMware Converter bootable CD may be an option. It provides a zero-transaction state that may be a favorable environment to perform P2V conversions.

Good candidates for using the VMware Converter bootable CD include:

• Systems that run a database engine,
• have real-time systems that may not convert correctly,
• or systems where the VMware Converter agent otherwise fails.

The bootable CD is licensed to enterprise customers, so the download requires advance purchase. The VMware Converter bootable CD is a Windows XP Pre Installation (PE) environment. The initial screen loads as follows:

The behavior is very similar to that of the full installation version once the VMware Converter interface loads. The only difference is that you can only convert the local system instead of being able to convert a remote system. This is to be expected, as the bootable environment should only be used when the traditional mechanisms fail. Once in the application, you can push the conversion to a VMware ESX server or to a flat .vmdk file for use in VMware Server or VMware Workstation:

I had a chance to use the VMware Converter bootable CD for a Windows 2000 system conversion that would not complete correctly in the installed, online environment. The bootable environment is also referred to as a cold clone environment, and with no transactions occurring on the file system a clean backup environment is available. The unfortunate circumstance is that this functionality can transport a poorly configured system to your virtual environment - so you may be able to keep it and its issues running forever.

Get answers to all of your ESX/VirtualCenter questions

This PDF has been a long time coming. It is a a document that lists just about every maximum value to do with VI3 possible. For example, how many iSCSI HBAs can you install on one ESX host? This document covers about 80% of the questions I receive on VMware a week, “How much/many of X does ESX/VirtualCenter support?” This document has the answers to your questions!

Kudos to the team at VMware that put this together!

Configuration Maximums for VMware Infrastructure 3

VMware’s competitive edge shrinking in light of Hyper-V hype, experts say

My Digg reader kicked out a great RedmondMag.com interview with VMware’s “product guru” Raghu Raghuram today in which he discusses the company’s product philosophy and how it translates to the VMware product line. During the interview, Raghuram says that there is a “stark difference” between Microsoft’s and VMware’s approach to virtualization. He had this to say about how he positions VMware ESX Server against Hyper-V:

Our view is that the core virtualization layer belongs in the hardware. It also has to be much smaller in order to reduce its surface area for attacks. This is why we introduced the 3i architecture . . . The Microsoft approach is to have virtualization be an adjunct to the OS . . . With the Hyper-V architecture, they’re still maintaining the same dependency on the OS.”

VMware ESX and Hyper-V are both bare metal virtualization products. To belabor an explanation, this means that they both sit in a thin OS layer abstracted from the hardware. This veritably eliminates hardware dependencies. However, Raghuram seems to be suggesting that Hyper-V is more of a hosted virtualization approach. This could be a misunderstanding on his part, questionable editing, or just a case of Microsoft being Microsoft.

At any rate, one difference that is certain can be logically approached when considering ESX versus Hyper-V. Something that virtualization expert Andrew Kutz said at a recent virtualization seminar keeps flashing in my mind. In his (and others’) view, Hyper-V will be the virtualization vendor to beat. This isn’t because Hyper-V is particularly a better product, but because VMware can’t compete with Microsoft on the level of supporting applications and interoperability.

In other words, after years of development and being the big guy in the computing space, Microsoft has a support cloud of applications and services all designed to work together that VMware will need to emulate in order to remain the leader in enterprise virtualization. This remains to be seen. But especially considering the low Hyper-V price tag ($28), VMware must be prepared to counter, at least with lower pricing.