November, 2007

Managing VMware Infrastructure 3 (VI3) from OS X (10.5.1 Leopard)

Up until now there has not been a way to manage VI3 from OS X clients. This stinks for those of us with our shiny PowerBooks and MacBook Pros (I just can’t let my 12″ go!). What to do, what to do… Then, suddenly and without warning BOMP BOMP BOMP (10 points if you know where that line is from) VMware released the VI Perl Toolkit, or as I like to call it, the viper toolkit (look at the directory name from a shell). Unfortunately for us there is no OS X version of the installer, but since VMware released the source, all that is required is a little ingenuity and we have ourselves a working VI client.

Step 1, cut a hole in a box. Oops, wrong series of steps. Step 1, download the viper toolkit from VMware at http://www.vmware.com/download/sdk/index.html. Make sure you get the source code (VI Perl Toolkit - source), not one of the pre-built installers.

Once the tarball is downloaded, deflate it and change directories into it. Go ahead and attempt to create a makefile with:

perl MakeFile.pl

You will receive something similar to:

[0]akutz@amends:viperltoolkit$ perl Makefile.PL

Checking if your kit is complete…

Looks good

Warning: prerequisite Class::MethodMaker 2.08 not found.

Warning: prerequisite Crypt::SSLeay 0.51 not found.

As you can see, some perl modules are missing. Good thing OS X (10.5.1 Leopard) comes with cpan. Install the missing modules by typing:

cpan Class::MethodMaker Crypt::SSLeay

Accept all of the default for both of the modules. Now you can create the makefile needed to build the viper toolkit:

perl MakeFile.pl

After you create the makefile, test it with:

make test

You should receive something similar to:

[0]akutz@amends:viperltoolkit$ make test

PERL_DL_NONLAZY=1 /usr/bin/perl “-MExtUtils::Command::MM” “-e” “test_harness(0, ‘blib/lib’, ‘blib/arch’)” t/*.t

t/VIPerlToolkit….ok

All tests successful.

Files=1, Tests=1, 1 wallclock secs ( 0.75 cusr + 0.06 csys = 0.81 CPU)

You want it to say “All tests successful.” Next install the viper toolkit by typing:

sudo make install

This will place the viper toolkit common runtimes and libraries and man pages in the appropriate locations. For OS X that is ‘/Library/Perl/5.8.8/’ and ‘/usr/local/share/man/man3/VMware’. You should see the following output:

[0]akutz@amends:viperltoolkit$ sudo make install

Installing /Library/Perl/5.8.8/VMware/VILib.pm

Installing /Library/Perl/5.8.8/VMware/VIM2Runtime.pm

Installing /Library/Perl/5.8.8/VMware/VIM2Stub.pm

Installing /Library/Perl/5.8.8/VMware/VIRuntime.pm

Installing /usr/local/share/man/man3/VMware::VIM2Runtime.3pm

Installing /usr/local/share/man/man3/VMware::VIM2Stub.3pm

Writing /Library/Perl/5.8.8/darwin-thread-multi-2level/auto/VIPerlToolkit/.packlist

Appending installation info to /System/Library/Perl/5.8.8/darwin-thread-multi-2level/perllocal.pod

Well, now you have a working VI client for OS X! Stay tuned as I show you more how the viper toolkit can make your life as an ESX administrator much easier.

Hope this helps!

Tips and tricks for general infrastructure needs using VMware VI3

VMware and Foedus joint issued a white paper some time called Tips and Tricks of Implementing Infrastructure Services on ESX Server. It’s a good read, although the first two paragpahs are meant for the virtualization-uninitiated and can be skipped if you are familiar with how to use VMware, VMotion, DRS, and HA to acheive redundancy and uptime improvements. The paper does focus largely on Windows networking, with Active Directory being repeatedly referenced. Since similar non-Windows services (such as OpenLDAP, OpenDirectory, eDirectory, etc. etc.) are affected in similar ways, it’s still a good cross-platform read despite the focus on Microsoft technologies.

If you are only going to scan the paper, there is one section I highly recommend reading, because it’s very ofen overlooked - Shutdown and Startup Sequence.

* I * Cannot * Stress * the * Importance * of * this * Enough *

Things crash - even in the best environment, you can expect to have problems now and again, and not planning this out just because you have other forms of reliable failover is no excuse. I have personally seen, in the lab and in practical reality, what happens when this tool isn’t used - it gets very nasty when you have a database server come up AFTER the server hosting and application that needs to access a database comes up, or when VPN servers come up early in the order and can’t find an authentication source.

A great tip for larger companies:

User environments with many Active Directory driven policies will want to ensure that the ESX Server %Ready time as viewed in the ESXTOP utility stays under 10

There is also the obligatory mention of time-syncing. Going back to my view on startup sequence… if you don’t install VMware Tools, you will have clock drift. If you have clock drift, services that depend on timed replication (particularly DNS, Active Directory, File Replication Services, and the Distributed File System). If you have an operating system that isn’t compatible with VMware Tools, the simple answer is this - don’t virtualize it.

What I was pleasantly surprised to see was a strong chunk of space allocated to setting up DMZ environments in VMware, which is somewhat of a sticky subject because of the two main ways it can be done (in virtual switching, or in the physcial network using additional nics in the host). The paper references one of my favorite open source projects of all time IPCop, a firewall that supports mulitple DMZs and is available as a VMware applicance. The paper’s approach is, appropriately for the material, focussed on the virtual swithcing, and they cover it well. Truthfully, I think the papwer should be renamed “How to use VMware and IPCop to make a DMZ, plus some other generic advice” because of the time they spend on DMZ vs. everything else.

An excerpt:

To setup up IPCOP as a DMZ firewall, create two virtual switches on one or more ESX Server hosts named DMZ-EXT and DMZ-INT. Plug the red NIC of IPCOP into DMZ-EXT and the orange NIC into DMZ-INT. Plug the green NIC into whatever virtual switch is associated with the internal LAN address space.

This paper is less a tips and tricks paper and more of a general advice paper, so I wasn’t overly impressed. From the title I was expecting more focus on VMware tips and tricks, but because of the DMZ section, it still gets 7 pokers.

x7

Use Virtual Infrastructure Client with Caution on Vista

The Virtual Infrastructure Client used to manage your VMWare ESX hosts may not perform all tasks correctly when running on Windows Vista. There are a handful of issues reported from users in the VMWare communities board such as “Object reference not set to an instance of an object” when trying to use the VMWare cloning features. I was fortunate enough to experience this issue just today. This thread mentions to update some time zone information into the registry. On another Vista system, we performed this registry tweak, and the VI client worked. However, this is not a good resolution from the Vistal environment perspective.

Critical VI Tasks Use on XP

If you are performing some critical VMWare tasks that are prone to failure when the VI client is installed on Vista, a better idea would be to perform those tasks from a Windows XP system. Though the issues reported with Vista are minor, and the VMWare server-side parts have intelligence to manage the issues as they occur to avoid integrity issues with the virtual machines or other pending tasks.

Why You Should Listen To Me

No, the title is not a typo. No, I am not that arrogant (I’m telling my wife to hush right about now.) :). I simply think that you should listen to me. Why? Because unlike a lot of other developers I know, I will gladly admit that there may be easier ways of accomplishing a task than writing new code. Developers who disagree have simply lost their way. We become developers because we are ultimately lazy — we want to create shortcuts, hack the OS to find hidden pathways we can exploit. But sometimes we lose our way and reinvent the wheel. Why? Why create what has been created if the prior creation has already been created and was created in a way that resulted in a good creation (I’m on a creation kick apparently)? Which brings me to my point (I bet you thought I didn’t have one) — the VMware VI Perl Toolkit is a wonderful thing and should be used at every opportunity in lieu of writing your own code.

I know writing code is fun — I do it all the time (just like using the em dash). However, writing code for writing code’s sake is a waste of time. Remember, code has a point. We are supposed to use it to carve out new pathways that have not been discovered. However, if someone has already hiked the trail and left their boots and walking stick behind we only need to pick up their tools and walk in their foot steps. And the beautiful part is that we don’t have to follow their trail completely — we can use the tools they left us to forge new trails, new pathways.

The VI Perl Toolkit comes with a lot of handy functions — creating VMs, cloning VMs, migrating them, requesting statistics on them. Yes, all these things are possible by leveraging the SDK directly (how do you think the Perl Toolkit accomplishes what it does — it uses the SDK too), but VMware has taken the time to forge the path and leave us walking sticks and hiking boots. Let’s not look a gift horse in the mouth people (unless we’re Trojans, in which case maybe we should inspect gift horses a little more closely).

Now, all that said, I am not asserting we completely forget about the SDK. I would be stupid to suggest such a thing as I write far too many articles on the subject to recommend that people should stop reading them! Remember, the VI Perl Toolkit uses the SDK too, so anything you learn about the SDK can be applied to increasing your knowledge of how the Perl Toolkit works, and more importantly, how to extend it. For example, did you know that you can set up asynchronous callback notifications based on events like a VMotion? I didn’t either until I told me, but boy was I surprised to find out what I knew! I will be writing an upcoming article/blog/SSV ATE on how to set up such a notification (for, oh, I don’t know, ensuring that you are maintaining Windows license compatibility) using the SDK and .NET. Be the first kid on your block to port my example to Perl using the VI Perl Toolkit.

Java. C#. Perl. I give them all just the right amount of love. I’m not a language bigot — I don’t see syntax, only 1s and 0s. And that is why you should listen to me.

Oh, and because I really, really like writing these things and it helps to imagine that someone is reading them. Otherwise I cry.

A lot.

Hope this helps!

Citrix on VMware - Going there, slowly

Welcome to my first blog on the new site… but not my last on SSV (hmm… I’ll be in both places at the same time… maybe I can get that prize money from Uri Gellar and Chris Angel). This may be part of a long series of posts, as soon I am destined to begin a Citrix implementation project without a hardware budget. Yes, that’s right… no hardware. We’re going to be using our existing VMware infrastructure. For those who follow my blogs, you probably have inferred that the mid-size organization I work for is making the jump from ESX 2.5 to VI3 next month. Truthfully, I’m not so positive about how this is going to turn out - there are far too many reported issues with the scheduler, storage controllers, and virtual SMP to make me entirely comfortable. We’re going to limit our Citrix servers to 1 vCPU each to circumvent a lot of the problems that occur with Citrix on ESX deployments, and try it out on NFS NAS to circumvent the storage controller problems that are frequently reported (though our iSCSI SAN would be more of an ideal solution and will most likely end up the final storage medium).

Mind you, we’ve tried other things in our VMware Server lab… 2X Application server came the closest to making the grade, but it’s not quite there yet. The same with Graphon. We looked at SSL-Explorer as a way to utilize Terminal Services directly over an SSL VPN, but it’s not quite what we were looking for (more the Terminal Services, less the SSL-Explorer… that’s a great product). In the end, it was time for Citrix, we knew it, and we’re going forward with it for a few users in a small environment first, hoping to build steam and justify a non-virtualized farm sometime next year, or perhaps an expansion of our VMware farm if performance on VMware works out.

We did some discovery, because there’s been so much negativity around Citrix performance in a VMware environment under 2.x and VI3. The first thing we found when we went to start testing was that Citrix offers a downloadable demo as a virtual appliance for Microsoft Virtual Server, but they don’t have one for VMware. This means going an extra step for conversion, and not really being certain whether any problems in performance are conversion-related or virtualization related. That really fried my bacon past crispy and into blackened-charcoal bits. To offset that gritty taste, we found a great post by Brian Madden, Citrix guru of long-standing grand repute, that helps explain some of the steps to take in order to improve performance. I highly recommend it as a great read.

As the demo progresses and (maybe) goes live, I’ll be posting more on the topic of Citrix and VMware playing in each other’s sandboxes. Since we’re going to be running some fairly deep applications (financial, mainly) applications as well as Office and other cross-departmental apps, I expect to be getting a lot of good feedback on the subject from my user base. I find their new head-to-head competition with Citrix XenServer interesting too…

Populating VMWare Server 2.0 with Existing Virtual Machines

As you may know, VMWare Server 2.0 Beta has been released for public (and still free!) beta. TechTarget covered the release, and I had a chance this week to install VMWare 2.0 on my CentOS Linux system. I chose to install on Linux without an X window manager not as an exercise in futility, but to really make a purpose built system that I can run the test virtual machines. With the new version of VMWare Server 2.0, I wanted to run my virtual machines there to see how they performed. My previous VMWare Server 1.03 system was running on Windows, and then I posed the question of how to get the virtual machine files there.

Depending on how you access your filesystem in Linux, there are many ways to get the files over. I choose to run Samba (smbd) on this Linux box as my means of getting the files to the system and with my particular version of CentOS, after VMWare installed based on my parameters, I could not access the path via the existing Samba share.

Here is what happened, after the installation I added a datastore in VMWare Server. The datastore, for those like me who are used to the Windows world, is the path in which the virtual machines are kept on the filesystem. For my own ease, I chose the path /home/rick/vms which resembles the \My Documents\My Virtual Machines format that I am used to when using the Windows version of VMWare Server. This path was also my home directory for my username. After the datastore was added, I could not access the path via the samba connection while using correct credentials. After some poking around I determined the following command will allow you to restore the user’s access to the path that contained the VMWare Server datastore as root:

chmod 777 /home/rick/vms

This changed the permissions for the this path for the rick username in the home path, to all users read, write, and execute. In my situation, there is only root and rick users configured on the system - so you may want to run the man chmod command to set a more appropriate set of permissions on the path. Once that was set, I restarted the Samba service and I was able to access the path to populate with my existing virtual machines and upgrade them for the VMWare Server 2.0 Beta.

VMware heads 50K network user school district data center and DR plan in hurricane country

If the Collier County Public School district IT department can trust VMware to to head its disaster recovery program in hurricane-ridden Florida, then you can probably trust it with your company’s IT and DR needs, too.

A SearchDataCenter.com coworker sent me this YouTube video in which Tom Petry of Collier County School District talks (rather quickly) about the disaster recovery realities of running a school district that has 44,000 students and 6,000 staff members “for a total of 50,000 network users” in hurricane-ridden southwest Florida.

Collier uses VMware to leverage failover sites to ensure robust business continuity — even if a rogue hurricane decides to eat the district’s data center for lunch.

In the video, you’ll get glimpses of Collier County school district’s data centers’ hardware, the generators on stand-by in case power goes down, you’ll see how VMware helps the IT department bridge the 120-mile distance between the main data center site and the failover site.

How to Install VMWare Tools Through Windows Group Policy

With all of the migration strategies available to the virtualization host administrators, it is easy to overlook the guest operating system details. When it comes to upgrading or installing VMWare Tools, the idea of touching every server interactively is not very appealing. For a Windows guest operating system (OS) on a domain, a group policy object can be one of your quickest options for getting the tools installed.

Some Guidelines for Installing VMWare Tools through Group Policy

The VMWare Tools are available for your guest OS as an .ISO CD-ROM file. Find the file, and get access to it from another system. Open the CD .ISO file or write it to disk and retrieve the files. Be sure to place them in a share accessible to the systems you want to install VMWare Tools with an indicator of the version. For example, \\SERVER1\Applications\VmwareTools\ESX3.02\ would indicate that this path would hold VMWare Tools for guest OSs on ESX server 3.02. Be sure to use the correct version of VMWare Tools with the corresponding guest OS if you have different VMWare host platforms in use. Each version of VMWare tools is optimized with the version provided with the installation.

Making the GPO

To make a new Group Policy Object (GPO) that would install the VMWare tools do the following from the Group Policy Management Console (gpmc.msc):

-Select the Organizational Unit (OU) you want to install VMWare tools
-Expand Computer Configuration
-Expand Software Configuration
-Click Software Installation
-In the right pane, select “New-Package”
-Browse to the VMWare Tools.msi file in your shared path
-Click Open
-Sending the package to “Assigned” will install the VMWare Tools package on the next refresh of the policy on the systems in the OU to which this GPO is linked.
-Once the package is listed, rename it from “VMWare Tools” to something more clear like “VMWare Tools - Windows - ESX 3.02″
-Once this is complete the VMWare tools will install on the server on the next boot without interaction. The Windows event log will have an entry similar to the following:


The install of application VMWare Tools - Windows - ESX 3.02 from policy Default Domain Policy succeeded.Changes to software installation settings were applied successfully.

Upgrades to VMWare Tools

In the situation where hosts are upgraded and guest OSs are moved dynamically, upgrading the VMWare Tools through the same fashion can be done as well. Be sure to use the version of the tools again in the package, and remove the old one if needed. Some proper testing is of course due, and a best practice would be to set up a test OU to apply the VMWare tools package before rolling out to more virtual machines.

VI3 SDK - The Windows Hosts File

It seems to be a little known fact that Windows has a hosts file similar to the one found in Unix and Linux at /etc/hosts. In the case of Windows the file is found at %SystemRoot%\System32\Drivers\etc\hosts. The contents of the file look like this:


# Copyright (c) 1993-1999 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a ‘#’ symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host127.0.0.1 localhost

As you can, the Windows hosts file allows you to make arbitrary host name to internet protocol (IP) address assignments. Why am I telling you this? Because if you are programming with the VI3 SDK you may need to alter this file or have your application crash miserably:


Unhandled Exception: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —>System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.


The above text is the error that occurs when you attempt to access the SDK’s web service with a host name that does not match what is on the SSL certificate that the SDK’s web server is using. By default the host name is “VMware”. So, if you have not changed the SSL certificate on the VirtualCenter server then you need to add a line to your Windows’ hosts file that points the host “VMware” (it is not case sensitive) to the IP address of your VirtualCenter server. Mine looks like this:


192.168.0.226 VMware


Voila! I can now access my VirtualCenter’s web server, and thus the VI3 SDK web service, programatically and without error.

Hope this helps!

Revisiting the VI3 SDK? Visit this link first!

How long has it been since you have taken a look at the VMware Virtual Infrastructure 3 (VI3) SDK? A month? Two months? Six? Maybe you were originally turned off by the SDK because of its hard-to-use traversal mechanism, or maybe you could never get your application to take less than a good minute to connect to the VI3 SDK web service. If your bad experience was because of the latter, you are in luck. Earlier this fall VMware released a patch for there VimApi namespace that changes the way that the serialization code is generated. The KB article that resolves this issue is located at VimApi.

I myself have tried solution #1 and found it to rapidly speed the connection times up. However, truthfully, the process of querying or setting data via the VI3 SDK is still slow. Hopefull VMware is working on this issue and a solution will make it into future releases of ESX.

Hope this helps!