Andrew Kutz

Friends don’t let friends, like VMware, act like Google

I like VMware. I like Google. Heck, both of them keep me more than busy with development ideas. But I have a problem with them. Google started it with Gmail. Although it is hard to remember now, Gmail was in beta forever. Oh wait, it still is? Huh. I guess I just figured it *must* have hit production by now. Then there is Google News, Google Apps, Google Page Creator, Google everything else — all beta . I am honestly surprised search hits don’t come back with the “beta” tag next to them. I guess they thought ICQ was the cat’s meow, and that the whole beta thing had a nice ring to it.

Enter VMware, which is perilously close to become the next Google in terms of heavily pushing new features, but then labeling them as beta or experimental. Take for example Storage VMotion (SVMotion). VMware played up this new feature to VI 3.5 last fall at their North American VMworld conference, but when it was release there was no graphical user interface (GUI) option for it. How is that ready for prime-time? And then there is virtual machine (VM) high availability (HA),  another marketed feature that is so experimental you have to edit an advanced setting (as a free-form string) just to enable the functionality.

I wouldn’t actually have a problem with VMware doing this if they didn’t market the heck out these new “features.” Excuse me for being old fashioned, but it isn’t enterprise-ready if it is beta or labeled experimental. And VMware makes no bones about this; they plainly state that these features should not be used in production. However, on the other hand they make a big show about the same set of features, whipping the crowd to a fever pitch of excitement. You can’t have it both ways, guys.

Take VMware Fusion 2 or VMware Server 2. These products are in beta stages right now and VMware is not making a big deal about them. Sure, they are out there for people to get, but VMware isn’t throwing them at customers, not the way they revolved last year’s North American and this year’s European VMworld conferences on features that were not even ready for production.

Then there is the other end of the spectrum as well. I recently discovered that VMware is strategically hiding a long sought feature of ESX in the bowels of its software development kit (SDK). Since version 2.5 of the SDK (VI 3.5), VMware has included the ability (although it does not yet appear to be working correctly) to create network address translation (NAT) and dynamic host control protocol (DHCP) devices directly on ESX servers for VMs to use. This is awesome! Prior to this, the only way to create NATd networks on an ESX host was to dual home a VM to a public and private port group, have it act as the NAT and DHCP server, and then attach other VMs to the same network as its private interface. This solution was cumbersome and did not work well when VMotioning VMs. If I was VMware, I would make a little bit more noise about the fact that they are working on this feature.

I want to reiterate that I like, if not love, VMware. I just hate getting jazzed about a new feature that they have thrown at me, only to find out that it is a curve ball. VMware needs to make sure that features that are experimental should be announced with an asterisk next to their headline, while at the same time working a little harder to ensure that some other upcoming features get the love they deserve.

VMotion and RDMs

Yesterday I was a speaker at “Virtualization: Getting from Pilot to Production.” During my second session I claimed that you could VMotion a virtual machine (VM) that uses a Raw Device Mapping (RDM) to access a raw logical unit number (LUN). Two audience members challenged this claim, saying that they had previously run into a scenario where it was not possible to VMotion a VM that makes use of RDMs. I was sure I was right, and they were positive they were correct. It turns out we were *all* spot on. You can VMotion a VM that uses RDM as long as the RDM is configured in virtual compatibility mode. When you map a SAN LUN using a RDM, you choose between two modes of operation: physical and virtual. Per VMware documentation:

Virtual mode for an RDM specifies full virtualization of the mapped device. It appears to the guest operating system exactly the same as a virtual disk file in a VMFS volume. The real hardware characteristics are hidden. Virtual mode allows customers using raw disks to realize the benefits of VMFS such as advanced file locking for data protection and snapshots for streamlining development processes. Virtual mode is also more portable across storage hardware than physical mode, presenting the same behavior as a virtual disk file.

Physical mode for the RDM specifies minimal SCSI virtualization of the mapped device, allowing the greatest flexibility for SAN management software. In physical mode, the VMkernel passes all SCSI commands to the device, with one exception: the REPORT LUNs command is virtualized, so that the VMkernel can isolate the LUN for the owning virtual machine. Otherwise, all physical characteristics of the underlying hardware are exposed. Physical mode is useful to run SAN management agents or other SCSI target based software in the virtual machine. Physical mode also allows virtual-to-physical clustering for cost-effective high availability.

Additionally, you can also VMotion a VM with an RDM that uses network port ID virtualization (NPIV), as long as you use virtual compatability mode.

So there you have it. The audience members were right. My memory is not as shot as I thought it was, and everyone is happy.

VMware releases official plug-in guide

Congratulations to the VMware engineering team releasing the official guide to creating VMware plugins. Their awesome, hard work is culminated in the document Getting Started with VI Client Plug-ins. Several of you have already asked me how their official methods and plug-ins compare to my previously published guide and plug-ins. After a cursory review of the official guide, here are my thoughts:

- The VMware guide to plug-ins is official. Mine is not. Although the VMware guide is experimental, it is more likely to be supported by VMware than any plug-in you write using my guide. That said, I have heard that the team responsible for plug-ins plans on developing a shim that will continue to support my plug-in methods in the next version of VMware Infrastructure (VI). That is hearsay, however, and it could change.

- Official plug-ins function very similarly to mine (they should, they are built using the same principals), but they could be considered inferior in one very important manner. Although the user interface to activate a plug-in is the same (context menus, tabs, menu items), the interface for official plug-ins can only be a web page. For instance, a user right-clicks on a virtual machine and clicks on the context-menu item labeled Migrate storage which launches my Storage VMotion plug-in rewritten as an official plug-in. Instead of having a Windows form appear that maintains a consistent user interface, instead a web browser appears and runs a script or web application that has authentication information and object information passed to it from the VI client (much like my Invoke plug-in).

- Official plug-ins must reside on a web server to which clients have access. This can be considered a good thing and a bad thing. On one hand, it centralizes plug-in updates. On the other hand, if the web server is offline ESX admins could lose plug-in functionality that they have come to rely on. Plug-ins written using my guide reside on the local client; always online and accessible.

- Given how hard it is for developers to build consistent web applications between browsers (even the VI online web interface fails to work properly on Safari), it is a tall order to expect plug-in developers to create plug-ins that look the same between the four major web browsers available to Windows (IE, Firefox, Safari and Opera). That said, you could just force users to use IE since you know it will be there.

- One big win with the official method is that the plug-ins could be written to be standalone web applications. This means that they could be accessed outside the context of the client.

The official plugin guide is a great achievement on the part of VMware; it shows their commitment to giving users what they want. Ultimately, though, the plug-ins created with it are forced to be web applications or scripts hosted on a web server. The biggest problem with this is that it forces users to leave the consistent look and feel of the VI client, ripping them out of their experience. The official plug-in guide is probably best thought of as an online alternative to the Invoke plug-in, but not as a replacement to the plug-in architecture that I have already exposed.

Hopefully with VI4 we will see a more fully fleshed out, official plug-in architecture.

Update 4/22/08:

Carter Shanklin, Product Manager of End User Enablement at VMware, sent me an email with the following note:

One point you make is the inconsistency of browsers in rendering, etc. Inside VI Client, if you create a custom tab it will be rendered using an Explorer control, regardless of what your preferred browser might be. This may be problematic if you want to use the same interface from outside of VI Client, but from within the client it should look fairly consistent.

Carter is absolutely correct. These controls should be rendered within the VI client, however you still need to make sure your web application looks and feels like a part of the VI client. Just because your web application is displayed from within the VI client, it does not mean that the VI client is rendering your list box or text box with its own style sheets.

Get answers to all of your ESX/VirtualCenter questions

This PDF has been a long time coming. It is a a document that lists just about every maximum value to do with VI3 possible. For example, how many iSCSI HBAs can you install on one ESX host? This document covers about 80% of the questions I receive on VMware a week, “How much/many of X does ESX/VirtualCenter support?” This document has the answers to your questions!

Kudos to the team at VMware that put this together!

Configuration Maximums for VMware Infrastructure 3

VirtualCenter 2.5 database index defragmentation

I, like many virtualization administrators, have worked very hard to get the VMware virtual environment set up and running as expected. Now, one of my main tasks is to make sure that we do not do anything to adversely effect server performance. A good place to start in this regard is the VirtualCenter (VC) database. That being said, the VC database is critically important to a successful ESX implementation, so do not do anything that is not advised by VMware documentation or support services. Let’s discuss index defragmentation in particular here when using Microsoft SQL server 2000 for the VC database.

Index defragmentation on statistics

I will save you some work in what to look for in determining which tables will need index defragmentation - statistics. While we all like the statistics and graphing options available in the VMware Infrastructure Client and virtual appliances that may use the table, there can be a great amount of data in that table and it can quickly become fragmented. A fragmented index in a database is similar to a fragmented file system where the ordering of an index is not in the order of the index.

In my VC 2.5 environment, the VPX_HIST_STAT1 table is the heavy hitter. For this database maintenance, I’m going to start with the white paper entitled “VirtualCenter Database Maintenance” available from the VMware website. Here there is a command to check your current fragmentation level:

USE
GO
DBCC SHOWCONTIG (VPX_HIST_STAT1)
GO

I have modified the command to use the table name, as the white paper is VC 2.0 based on the table name, whereas this example is on VC 2.5. The result will look something like the following:

DBCC SHOWCONTIG scanning ‘VPX_HIST_STAT1′ table…
Table: ‘VPX_HIST_STAT1′ (800721905); index ID: 1, database ID: 16
TABLE level scan performed.
- Pages Scanned…………………………..: 505458
- Extents Scanned…………………………: 78097
- Extent Switches…………………………: 457307
- Avg. Pages per Extent……………………: 7.4
- Scan Density [Best Count:Actual Count]…….: 22.34% [113183:905308]
- Logical Scan Fragmentation ………………: 3.81%
- Extent Scan Fragmentation ……………….: 0.47%
- Avg. Bytes Free per Page…………………: 187.2
- Avg. Page Density (full)…………………: 97.69%
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

The key takeaway is the Scan Density percentage. The white paper advises that a number close to 100% is good, meaning that the table index in the example above is quite fragmented. The white paper goes on to identify two correction levels for improving the index. Index defragmentation and, more aggressively, rebuild are the standard options to address the index. If the scan density after a index defragmentation does not do enough to improve the index, database admins will have to begin the rebuild process. A caveat: rebuilding requires VC downtime to perform the database maintenance.

By comparison, here the same command on the VPX_EVENT table. This table is busy, but not near as much as the statistics table:

DBCC SHOWCONTIG scanning ‘VPX_EVENT’ table…
Table: ‘VPX_EVENT’ (36195179); index ID: 1, database ID: 16
TABLE level scan performed.
- Pages Scanned…………………………..: 594
- Extents Scanned…………………………: 86
- Extent Switches…………………………: 132
- Avg. Pages per Extent……………………: 6.9
- Scan Density [Best Count:Actual Count]…….: 66.39% [75:133]
- Logical Scan Fragmentation ………………: 6.73%
- Extent Scan Fragmentation ……………….: 98.84%
- Avg. Bytes Free per Page…………………: 150.5
- Avg. Page Density (full)…………………: 98.14%
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

This table is in better shape, but is much smaller than the statistics table.

Configure statistics logging level

As virtualization administrators, one safeguard we can perform is to limit the logging levels within the VMware Infrastructure Client. To access the logging levels select Administration menu, then VirtualCenter Management Server Configuration, then Statistics. Here you want to limit the number of high level logging to keep the VPX_HIST_STATx tables in check:

In selecting which level works best for your environment, be sure to identify any monitoring tools or virtual appliances that may read the selected tables. Also be sure to benchmark your database size and index fragmentation to see if you gain any improvements. Identifying the parts of the entire VMware Infrastructure environment that you can keep in maintenance mode will make your job as a virtualization administrator much easier.

Coming soon: Application High Availability (AHA)

Scott Lowe recently blogged about things that he loves but are not yet available. VMware, it seems, is getting as bad as Apple in announcing products far ahead of their release dates. This generates great market buzz and sends stock prices soaring (or in the case of VMworld Europe leaving them as low as they have been), but it does little to satiate the desires of systems administrators, it only serves to increase want and provide little respite. Well, then I am about to send you into the Sahara with the promise of an oasis, only to find a sign-post that says “Coming Soon!” at the end of your journey.

Please join me in anticipation of VMware Application High Availability (AHA).

AHA has not been announced by VMware, but it is coming. How do I know this? It is next logical step for their HA portfolio. ESX 3 brought us server HA. 3.5 introduced us to VM HA. And the recent announcement of the VMsafe all but secures the eventual release of AHA.

What is AHA? Simply put, you will be able to right-click on a VM from the VI client and indicate that if Microsoft Exchange fails then the Exchange service should be restarted, or the VM should be failed over to another ESX server. Or perhaps you want to monitor the Apache web server — just check a box. How will VMware achieve this level of fine-grained control? Allow me to refer to the VMsafe product page:

Process execution
VMsafe provided in-guest, in-process APIs that enable complete monitoring and control of process execution.

This API will allow VMware to monitor and control processes within the guest OS. That, my friends, is how AHA will work.

AHA will allow VMware to take even more market share away from companies like Sun and Microsoft, both who have their own clustering technologies. Why cluster at the OS-specific level when you can create clusters the same way no matter the OS or application underneath!

VMsafe is set to debut later this year, and I am quite certain that alongside VMsafe, or shortly thereafter, we will see VMware announcing and releasing its application level high availability software. I hope you aren’t too parched : )

Embed SSH connections to ESX with console plug-in

Fellow SearchVMware.com blogger Andrew Kutz has done it again with his release of the integrated SSH plug-in for use in the VMware Infrastructure Client (VIC). This plug-in saves ESX administrators the hassle of launching Putty or another mechanism to start a session to the ESX host. Details on the plug-in and the download link are available on this post in the VMware communities thread and I had a chance to use the plug-in for a while successfully.

Installing the console plug-in is straight forward from the ConsoleClientSetup-0.1.5.msi and it is easily added from the plug-ins menu in the VIC. Once added, each ESX host in your inventory will have a new tab called Console that performs in a similar fashion that the Console tab does for virtual machines. The difference is that authentication to the ESX host is passed through the plug-in. This requires that SSH be enabled on the ESX host, and should you wish to use the root login there is a slight configuration to enable root SSH access which is explained here on the ITKE by David Davis.

The console tab now visible on the far end of the VIC and is context-sensitive so it is only displayed when you have an ESX host selected in the left windows pane. Once installed the plug-in allows you to authenticate directly to the host with a username and password:

Now it is very easy to log into the ESX host and perform simple tasks such as running esxtop, pinging the console, and enabling visibility to the file system within ESX. Note that as long as you have the console connection established, it runs as long as your VIC session is running. So use the disconnect link when you are complete with a session. If you close the VIC, the session is disconnected from the ESX host. Here is my session running the esxtop command while in maintenance mode from within the VIC:

Limitations

While this plug-in definitely makes a quick look to the ESX console quite accessible, there is one limitation that I came across. The resolution of the embedded VIC console is not as scalable as a Putty session. This is especially noticeable if you are running some of the esxcfg-xxxxxxx commands. Take for example running the following command:

esxcfg-mpath -l -v

This command has 140 positions of text on the longest line, which is visible on one line in Putty if you have the resolution. For the console plug-in, the width is limited to 80 positions of text, and for long commands they should be run in pipe fashion to show the full output. For the example above, this iteration would be as follows:

esxcfg-mpath -l -v | more

Important note on plug-ins

While the ESX plug-ins work in the VIC, they are not supported by VMware. With this warning, adequate testing is required to ensure that you do not have any issues in your environment that may effect live systems. As a matter of practice, I do not install any non-essential components on the Virtual Center server so a tool like this would be best suited on client systems only. All that said, Andrew has put out another great plug-in to make administration of an ESX environment more centric to the VIC. I can’t wait for his next plug-in!

VMware Infrastructure 3.5 Plugin and Extension Programming Guide

The VMware Infrastructure 3.5 Plugin and Extension Programming Guide - Revision 1 is now available at VIPlugins.com. This document is not sponsored or supported by VMware in any way. In fact, an excerpt from the text:

“While the succeeding pages may give the impression that this paper was written in cooperation with VMware, this work is the result of hours of using Lutz’s Reflector to peer into VMware’s intermediate language (IL), Lutz’s Resourcer to figure out where icons come from (it’s not the icon stork), ProccessMon, FileMon, and RegMon to take a look at things happening in real time, and finally the Microsoft structured query language (SQL) manager to explore the new VI 3.5 database schema. In summary, although the knowledge from these explorations resulted in an idea of the VI plugin architecture and working plugin, do not consider it to be the final word on anything. We will simply have to wait for VMware to provide finality to this matter.

In short, all the information contained in this document may be entirely and completely wrong. Read it at your own risk. If you find yourself stuck in an infinite time loop once you finish, remember two things: 1) ice sculptures impress the heck out of the ladies and 2) you are not god. You may be a god, but not the god. That honor is left to Mr. Morgan Freeman.”

This paper focuses on educating developers on:

Client Plugin Architecture
This includes where plugins are installed, how the VI client discovers local plugins and ones advertised on the VC server, and finally how to create a client plugin.

Server Extension Architecture
Includes how to register server extensions and how to make client plugins centrally available.

Tomcat Architecture
Discusses how the VirtualCenter Tomcat installation impacts server extension daemons.

Database Schema
Review the new VirtualCenter database tables that are related to extensions.

Creating Windows Installers
Reveals some problems with creating Windows Installers for server extensions.

Namespaces
Details the namespaces and assemblies VMware provides to create plugins and extensions.

Terminology
Coins new terminology that developers can use when discussing the above concepts.

Hope this helps!

Storage VMotion through the VI client

The VMware Infrastructure client allows for plug-ins for third party applications and other enhancements. Andrew Kutz of lostcreations has released a plug-in for Storage VMotion that you can use outside of the VirtualCenter Remote Command Line Interface (Remote CLI). I had a chance to test the plug-in, and I can tell you that it works and is quite easy to use.

The plug-in is discussed here on the VMware communities Website with some feedback from other ESX administrators. A link for the download as well as installation instructions are available there as well. Once the plug-in is installed, a right-click option Migrate Storage appears when clicking on a virtual machine. When the plug-in activates, a window appears that lets you select what storage you want to migrate the running virtual machine to:

Once you select the correct storage, the task is initiated to VirtualCenter and at this point it is indistinguishable to the task as it would be performed in the Remote CLI. The plug-in does not break compatability rules. For example, if you attempt to migrate a virtual machine’s storage running on an ESX 3.02 or lower host, VirtualCenter will prohibit the task due to compatability.

The plug-in is freely available under the new BSD license. Give it a test drive and post your feedback here or on the community board..

Pre and post backup scripts for VMware Server 1.0.4

So I use BackupPC for a lot of my SMB backup needs. For my money it is the best open-source backup program hands-down. However, one of the problems that I have run into is when I backup my VM directories on VMware Server systems, the VMs sometimes go wonky because they do not like being backed up when they are running. So I need a way to suspend the VMs prior to a backup and resume them after a backup.

BackupPC implements what are called Pre and Post commands that allow an IT administrator to supply commands to be before and after a backup operation is to take place. The two Pre and Post commands we are concerned with are DumpPreUserCmd and DumpPostUserCmd. The BackupPC documentation explains how to use the Pre and Post commands to send a command to the server that is going to be, or just was, backed up via SSH. Now obviously this works great for Linux, UNIX, and OS X servers, but for an OS that lacks certain fundamental necessities like an SSH server, and I am of course talking about Windows, the burden is on the IT administrator to install one (it is possible to obtain a free SSH server for Windows).

So now that we know how to issue commands to a server before and after it is to be backed up, we need to be able to tell that server to suspend and then resume its VMs. However, it is not enough to simply issue a suspend and resume command to all VMs, there needs to be some logic involved. For example:

- The VM should only be suspended if it is running.
- The VM should only be resumed if it is suspended and was suspended as part of the backup operation.

Lucky for you I have written two bash scripts (Linux/Unix/OS X compatible) that do just that.

Note: The IFS variables have a letter O in them instead of the numeral 0 because Wordpress keeps escaping the 0. So if you copy and paste the scripts, please replace the letter O with a number 0.

suspend_vms - This script will suspend all of the VMs currently running that are listed in the server’s VM list file. A note will be placed into a temporary file /tmp/vms_to_be_resumed that this VM was suspended as part of a backup operation. This temporary file will be read by the start_vms script in order to determine which VMs to resume.

#!/bin/bash
IFS=$'\O12'
for F in $(grep "config" /etc/vmware/vm-list | sed "s/(config )(.*)/2/g")
do
        VMCFG=$(echo $F | tr -d \")
        VMSTATE=$(/usr/bin/vmware-cmd $VMCFG getstate)

        echo -n "$VMCFG - $VMSTATE"
        if [ $(echo $VMSTATE | grep on) ]
        then
                echo $VMCFG >> /tmp/vms_to_be_resumed
                echo " - suspending"
                /usr/bin/vmware-cmd $VMCFG suspend
        fi
done
echo

start_vms - This script will parse the contents of /tmp/vms_to_be_resumed and resume the VMs that are listed in the file and are currently suspended. The temporary file will be deleted at the end of this script’s execution.

#!/bin/bash
IFS=$'\O12'
for F in $(cat /tmp/vms_to_be_resumed)
do
        VMCFG=$(echo $F | tr -d \")
        VMSTATE=$(/usr/bin/vmware-cmd $VMCFG getstate)
        if [ $(echo $VMSTATE | grep suspended) ]
        then
                echo "$VMCFG - $VMSTATE - resuming"
                /usr/bin/vmware-cmd $VMCFG  start
        fi
done
rm /tmp/vms_to_be_resumed
echo

These scripts should allow you to successfully back up VMs running on VMware Server on a Linux server. Simply place them in an appropriate location on the server to be backed up that has VMs, such as /usr/local/bin, and use BackupPC’s pre and post commands to call them via SSH. If someone would like to write the equivalent scripts for Windows I would be happy to post those here as well (if I do not write them soon myself).

Hope this helps!