VI3

Virtualization virtual tradeshow offers VMware networking opportunities

Trade shows are great – if you have time to attend, have staff to cover while you’re away learning about a new technology, can avoid summons back to the office during the show, can find a show in your local area or can get budget approval to attend a show that requires flight or hotel reservations.

Enter the virtual trade show (VTS); an online conference conceived to mitigate the above challenges. Last week, sister sites SearchDataCenter.com and SearchServerVirtualization.com hosted an advanced enterprise virtualization VTS. I helped staff the networking lounge and editorial booth where I had the opportunity to chat with VMware users about two of the virtualization provider’s newest tools, Site Recovery Manager (SRM) and Storage VMotion.

IM chatting with attendees
Conversations ranged from general IT talk (“Anyone use virtual desktops?”) to small talk (“What’s the weather like in Maine?”). Trying to be the friendly host, I said “Good morning” to the room. I immediately got the reply “Good evening” and was subsequently told this particular user was signed on from –literally–the other side of the world.

I ended up chiming-in on another user’s question about if anyone was familiar with VMware Site Recovery Manager (SRM). The respondent had said that he was, and I ended up asking him about his experience via private IM. SRM orchestrates your virtual machine disaster recovery (DR) plan in the event that your main data center goes down. It prioritizes which virtual machines (VMs) are brought up at the failover site based on available resources, syncs your VM configurations between the main site and the failover site, and allows for DR plan testing without having to take the system offline. It’s a relatively new addition to the VI3 lineup, having been on the market for four months (at the time of publication).

Our conversation turned to plug-ins, and he raved about Andrew Kutz’s Storage VMotion plug-in. The plug-in adds a user interface to the out-of-the-box product, which operates through a command line interface. The attendee explained that he’s primarily a “Windows guy,” so a graphical user interface makes using Storage VMotion much easier.

Kutz recently released an update to the Storage VMotion plug-in.

“The new release now ignores raw device mapping,” Kutz said. “Previously, if you had a raw device that pointed to a 300 Gig disk, the plug-in would look at it as an actual disk and screw up the disk size map.”

He also removed the majority VMware’s internal code from the plug-in (excepting the code that loads the plug-in), replacing it with code based on the VI Toolkit for .NET.

Impressive user interface
The VTS emulates the look of a physical tradeshow floor, which makes navigation a bit friendly, though not as intuitive as I would have liked. You could either move around with the help of a clickable navigation bar, or point-and-click your way from the main entryway to the desired location, be it the conference hall, vendor hall, networking lounge or “library” where you can download PDFs of presentations and various information from vendors, which then moves into your “suitcase,” displayed on your personal page.

VTSs are essentially fancy webcast packages displayed in unconventional ways. In this particular show, the topics were “Protecting your Virtual Environment: Backup and Storage,” “Virtual Infrastructure Automation and High Availability Best Practices” and “Virtual Infrastructure Tuning and Advanced Management.” The speaker was displayed on the left side of the screen presenting his slides via streaming video. The slides were displayed on the right hand side. Users could ask questions via a box at the bottom of the screen.

The VTS, if done correctly, has many more plusses than minuses. As long as there is a reliable Internet connection, there’s no need to leave the data center (if you don’t have a reliable connection in your data center, you might think about leaving for good). The content is almost exactly the same as at a physical trade show (that’s how they got the video of the speaker to begin with). And editorial staff can send IT pros direct links to helpful guides that they know of if an IT pro wants to know about, for example, virtual desktop drawbacks.

If any SearchVMware.com readers passed up the opportunity to “attend” a virtual trade show, I suggest you test it out next time a topic of interest comes around. It’s actually fun to use (think AOL in the 90’s minus the “you’ve got mail”) and offers great learning potential and networking opportunities.

An archived version of the advanced enterprise virtualization virtual trade show is available online, short registration required.

Automatic recovery for failed virtual machines

A little known feature, called Virtual Machine Failure Monitoring (VMFM), was introduced in ESX 3.5. VMFM offers the ability to leverage HA to monitor VMs for operating system failures, such as blue screens, and have them automatically restarted. Previously, HA would only deal with ESX host failures by automatically restarting VMs on alternate ESX hosts in the event of a problem with the host server.

VMFM also extends HA to monitor VMs through a heartbeat sent every second when using VMware Tools. This new feature is disabled by default and is considered ‘experimental’ by VMware. This typically means it works but it is not officially supported for production use yet. In order for this feature to function properly you must first ensure the following conditions exist:

• ESX hosts are version 3.5
• VirtualCenter is version 2.5
• VMware Tools is installed on VMs and is the latest version
• You have a Cluster configured and HA enabled

To enable it follow the below steps:

• Edit the Settings for your Cluster
• Choose VMware HA and click the Advanced Options button
• Add the following Options and Values

das.vmFailoverEnabled – true (true or false)
das.FailureInterval – 30 (declare virtual machine failure if no heartbeat is received for the specified number of seconds)
das.minUptime – 120 (After a virtual machine has been powered on, its heartbeats are allowed to stabilize for the specified number of seconds. This time should include the guest operating system Boot up time)
das.maxFailures – 2 (Maximum number of failures and automated resets allowed for the time that das.maxFailureWindow specifies. If das.maxFailureWindow is ‐1 (no window), das.maxFailures represents the absolute number of failures after which automated response is stopped and further investigation is necessary)
das.maxFailureWindow – 86400 (Either -1 or a value in seconds. If das.maxFailures is set to a number, and that many automated resets have occurred within that specified failure window , automated restarts stop and further investigation is necessary)

I enabled this on a Cluster and tested it by simulating a blue screen on a VM running Windows 2003 Server and it worked perfectly. After 30 seconds the loss of heartbeat was detected and the VM was automatically restarted. Currently there are no notification alerts that can be configured when this occurs. That is, if you check the events for the VM you will see no evidence of this happening. The only mention of it that I found in the logs was in the hostd log on the ESX server ([2008-06-26 11:47:22.552 ‘ha-eventmgr’ 3076440992 info] Event 101 : VM1 on Esx1.xyz.com in ha-datacenter is reset). Hopefully this will change in a later version when the feature is no longer considered ‘experimmental’. You can read more about this new feature in a white paper that VMware has provided on it.

What makes VMotion so great?

To many of us, some VMware product features and functionalities still seem like magic. Sure, I understand what the products do, but considering that my education and background is in writing and editing (read: not computer science), how they work still remains a mystery.

At least a few pieces of the VMware Infrastructure feature set were explained to me by Alliance Technologies solutions architect and Central Iowa Virtualization User Group (CIVUG) member Sean Clark. In a recent conversation, Clark explained how VMotion works, what Distributed Resource Scheduler (DRS) is and how the competition stacks up. Check out this audiocast if you’re still using VMware Server or just the ESXi hypervisor but are considering moving into the rest of the suite as it more provide more information to help you with your decision.

Isolate ESX hosts in separate clusters for maintenance, upgrades

Performing key maintenance on a VMware Infrastructure 3 (VI3) host OS or can be an involved process, as can the process of simply adding a host. Because of the length of time that a system may be offline or because of what exactly is being performed, having the hosts in an isolated cluster can allow a safe environment for virtually any task. An isolated cluster will not apply to the same DRS and high availability rules that may apply to the live cluster. Further, you can reboot the host as needed without needing to wonder if there will be any effect to the live workload.

With the isolated cluster, the following tasks can be safely performed:

-Version upgrades (ESX 3.0x to 3.5 Update 1)
-Hardware maintenance
-Adding a new host to an existing cluster
-Testing network connectivity to various port groups
-Confirming VMware HA and DRS performance with in an isolated set of rules
-Importing or configuring new storage types

A good practice for the isolated cluster would be to keep it less visible to the live workload and named accordingly within the VMware Infrastructure Client, so a name like “TestingCluster” or “ZZUpgradeCluster” can distinguish the collection to be different from that of the live workload.

The figure below shows a cluster with a name and position that contains one host in maintenance mode for any task better suited in an isolated environment:

It is important to note that this cluster would still require licensing as it would be in live workload cluster. More information on creating a VI3 cluster can be found in the VI3 online library.

Choosing a block size when creating VMFS datastores

When you create a VMFS datastore on your VMware ESX servers many administrators select the default 1MB block size without knowing when or why to change it. The block size determines the minimum amount of disk space that any file will take up on VMFS datastores. So an 18KB log file will actually take up 1MB of disk space (1 block) and a 1.3MB file will take up 2MB of disk space (2 blocks). But the block size also determines the maximum size that any file can be, if you select a 1MB block size on your data store the maximum file size is limited to 256GB. So when you create a VM you cannot assign it a single virtual disk greater then 256GB. There is also no way to change the block size after you set it without deleting the datastore and re-creating it, which will wipe out any data on the datastore.

Because of this you should choose your block size carefully when creating VMFS datastores. The VMFS datastores mainly contain larger virtual disk files so increasing the block size will not use all that much more disk space over the default 1MB size. You have the following choices when creating a datastore:

• 1MB block size – 256GB maximum file size
• 2MB block size – 512GB maximum file size
• 4MB block size – 1024GB maximum file size
• 8MB block size – 2048GB maximum file size

Besides having smaller files use slightly more disk space on your datastore there are no other downsides to using larger block sizes. There is no noticeable I/O performance difference by using a larger block size. When you create your datastore, make sure you choose your block size carefully. 1MB should be fine if you have a smaller datastore (less than 500GB) and never plan on using virtual disks greater then 256GB. If you have a medium (500GB – 1TB) datastore and there is a chance that you may need a VM with a larger disk then go with a 2MB or 4MB block size. For larger datastores (1TB – 2TB) go with a 4MB or 8MB block size. In most cases you will not be creating virtual disks equal to the maximum size of your datastore (2TB) so you will usually not need a 8MB block size.

So remember, choose carefully, once you create your datastore there is no changing it later.

Ensure VMware Infrastructure Client is current

When VMware issued VirtualCenter 2.5 Update 1 and ESX 3.5 Update 1 in April of this year, I felt that this release was mature enough to upgrade my environment. One observation I have made is related to the VMware Infrastructure Client (VIC) versioning. The VMware Infrastructure 3 release notes mention many configurations related to supported environments of VirtualCenter and ESX, but do not distinguish between the base release of the VIC for VirtualCenter 2.5 and 2.5 Update 1.

The VIC for VirtualCenter 2.5 (base release) has a build number of 64192 whereas the VIC for VirtualCenter 2.5 update 1 has a build number of 84767. Unlike the upgrade from VirtualCenter 2.0 to VirtualCenter 2.5, connections into VirtualCenter are not prompted to download and install the new version of the VIC. Therefore, you can have VIC installations with the older version interacting with the newer VirtualCenter server. I recommend that you enforce the version currency of the VIC on all systems connecting to a VirtualCenter 2.5 Update 1 to be at build 84767 so that commands match the capabilities of VirtualCenter. For most situations, this is not going to be an issue, but should you use VirtualCenter plug-ins, advanced features, or have a large amount of VIC traffic there may be some anomalies in the version mismatch.

The figure below shows two information screens from two installations of the VIC. Both of these installations are interfacing to the same VirtualCenter server, but the top one is at the current version:

During the VirtualCenter upgrade process, the VIC is updated locally on the server. Further, unlike the VIC for VirtualCenter 2.0 the VIC for VirtualCenter 2.5 and 2.5 Update 1 have the same binary update path, so you will not have to manage two shortcuts for access.

Upgrading VirtualCenter does not update certificate store

When logging into the VMware Infrastructure Client (VIC) or using the web interface to the VirtualCenter server, you are presented with a certificate message. Best practice or not, I usually accept the certificate and instruct the software to not ask me again about this topic. I have just completed end-to-end testing of VirtualCenter 2.5 update 1 and ESX 3.5 update 1, however, and noticed something about the certificate store for VirtualCenter.

The VirtualCenter certificate store is valid for two years from the date of initial installation. Since this instance of VirtualCenter was installed, I have upgraded to version 2.5 base release and, most recently, to the version 2.5 update 1. But neither installation updated the certificate. Below are the details of my test certificate:

The good news is that you now know about this issue. The bad news is that you better correct it before the two year anniversary of your installation of VirtualCenter as it is required to process logins. VMware has a comprehensive PDF that outlines the certificate procedures for VirtualCenter and the ESX hosts. The ESX hosts, however, have a much longer lifespan for the local certificate, around 20 years, and do not exhibit this behavior. The VMware server certificate documentation is available for download from the VMware website.

Friends don’t let friends, like VMware, act like Google

I like VMware. I like Google. Heck, both of them keep me more than busy with development ideas. But I have a problem with them. Google started it with Gmail. Although it is hard to remember now, Gmail was in beta forever. Oh wait, it still is? Huh. I guess I just figured it *must* have hit production by now. Then there is Google News, Google Apps, Google Page Creator, Google everything else — all beta . I am honestly surprised search hits don’t come back with the “beta” tag next to them. I guess they thought ICQ was the cat’s meow, and that the whole beta thing had a nice ring to it.

Enter VMware, which is perilously close to become the next Google in terms of heavily pushing new features, but then labeling them as beta or experimental. Take for example Storage VMotion (SVMotion). VMware played up this new feature to VI 3.5 last fall at their North American VMworld conference, but when it was release there was no graphical user interface (GUI) option for it. How is that ready for prime-time? And then there is virtual machine (VM) high availability (HA),  another marketed feature that is so experimental you have to edit an advanced setting (as a free-form string) just to enable the functionality.

I wouldn’t actually have a problem with VMware doing this if they didn’t market the heck out these new “features.” Excuse me for being old fashioned, but it isn’t enterprise-ready if it is beta or labeled experimental. And VMware makes no bones about this; they plainly state that these features should not be used in production. However, on the other hand they make a big show about the same set of features, whipping the crowd to a fever pitch of excitement. You can’t have it both ways, guys.

Take VMware Fusion 2 or VMware Server 2. These products are in beta stages right now and VMware is not making a big deal about them. Sure, they are out there for people to get, but VMware isn’t throwing them at customers, not the way they revolved last year’s North American and this year’s European VMworld conferences on features that were not even ready for production.

Then there is the other end of the spectrum as well. I recently discovered that VMware is strategically hiding a long sought feature of ESX in the bowels of its software development kit (SDK). Since version 2.5 of the SDK (VI 3.5), VMware has included the ability (although it does not yet appear to be working correctly) to create network address translation (NAT) and dynamic host control protocol (DHCP) devices directly on ESX servers for VMs to use. This is awesome! Prior to this, the only way to create NATd networks on an ESX host was to dual home a VM to a public and private port group, have it act as the NAT and DHCP server, and then attach other VMs to the same network as its private interface. This solution was cumbersome and did not work well when VMotioning VMs. If I was VMware, I would make a little bit more noise about the fact that they are working on this feature.

I want to reiterate that I like, if not love, VMware. I just hate getting jazzed about a new feature that they have thrown at me, only to find out that it is a curve ball. VMware needs to make sure that features that are experimental should be announced with an asterisk next to their headline, while at the same time working a little harder to ensure that some other upcoming features get the love they deserve.

VMware’s Virtualization Forum 2008 in New York vendor centric

When attending a vendor sponsored conference, you expect that most - or all - of the sessions will focus on that vendor’s products along with partner products. So, I wasn’t all that surprised when I found that to be the case at VMware Inc.’s Virtualization Forum 2008 in New York, NY on May 8.

But I felt a tad cheated when I left.

You see, the agenda I received from VMware prior to the event looked great. There would be analysis from Gartner’s Distinguished Analyst, Thomas Bittman, about virtualization technology adoption and trends. Also, Quest Diagnostics was scheduled to talk about their specific pain points and challenges and how virtualization saved the day. Good, and good.

Other sessions included Introduction to Server Virtualization, Datacenter Management and Automation, Business Continuity, Application Virtualization and Remote and Branch Office Management.

The agenda I received in my inbox before the event also included three afternoon sessions scheduled from 2:15 to 3:15 for Platinum Sponsor / Customer Story; the latter being of most interest to me.

So, I got up at 5 a.m. on Forum day and drove from Rhode Island to New York City - a lovely four and a half hour trek with traffic.

I registered, got my Conference Guide, ID badge, marketing package full of VMware ads, and headed to hear Bittman talk. It was interesting enough. Then Quest Diagnostics did a quick talk about their VMware implementation. Cool.

Then lunch. Which was free. Which is awesome.

After that, I hit the Datacenter Automation and Management Session, where John Suit, CTO and co-founder of Fortisphere, discussed how their product,Fortisphere Virtual Forsight, helped Duke University Hospital set policies for virtual machines. Then John Brock, senior product marketing manager at VMware, stepped up to bat to talk about all the wonderful things Virtual Infrastructure can do for IT.

Fine.

Now for the 2:15 session - Platium Sponsor/ Customer Story. Much to my chagrin, the Customer Story portion of those sessions were not included in the eight page guide. All I see listed are Platinum Sponsor EMC Corp., Platinum Spronsor HP, Platinum Sponsor Dell/ Equalogic Corp., Platinum Sponsor NetApp. Brows furrowed, I flipped the pages looking for the customer stories. Were the times changed? Confused, I asked the nice VMware marketing representative if VMware mistakenly ommitted the customer stories from the guide.

“No, there won’t be anymore customer stories this afternoon,” she said.

“Oh, but the schedule I received last month had customer stories as part of the afternoon sessions,” I said.

Uncomfortable silence.

“The sponsors will probably talk about some customer stories,” she said.

What the *%#@?

But, I was not too surprised by this program change. It happened at the last VMware event I attended as well.

The VMware Virtualization Seminar Series in Providence, RI in February was the same way - customer story on the schedule, but not at the podium.

I complain about this not because of the horrid commute I made expecting to hear from more than one real user, but because I can safely assume that, like me, data center managers want to hear about virtualization implementations from their peers–the folks who manage virtual environments in real data centers every day. You know, the people who don’t have the word “marketing’ in front of their name. I’m sure that attendees would have like perspective from the system engineers and architects who can give honest, detailed descriptions about pitfalls to avoid, the real costs and ways to deploy virtualization successfully.

I understand users also want to learn about products that can solve their IT issues, and I am sure that some of the information vendors provide is very useful. But, VMware, I beg of you–please include real users in your future seminars, and if you don’t, then don’t put it in your agendas.

Use VI3 maps to visualize storage distribution

You may not have used the maps feature within the VMware Infrastructure Client because large environments become difficult to decipher. The maps view has become an important part of visualizing certain elements of the overall configuration of an ESX environment. One of the more useful mapping views that can help to illustrate relationships is the virtual machine (VM) to datastore map. This shows how many virtual machines are contained in each datastore. It will also show the virtual machines that have connections in multiple datastores.

To use the maps view to show the correlation between VMs and datastores, select the maps button from the main toolbar and then deselect all options except “VM to Datastore”. Then select the datastores, hosts, clusters or resource pools you wish to have represented in the map. It is best to construct your diagram based on the storage configuration. So, if your storage is per datacenter, diagram from the datacenter level down. Once you click the “Apply Relationships” button, a map is drawn based on all of the datastores. You can zoom out of the map by pressing the “-“ key or [ctrl] and scroll on a wheeled mouse. Below is a sample map of a datacenter and the VMs connected to each datastore:

In this example, there are three datastores without a VM assigned within the map. This is the local datastore on the ESX servers in this environment. As a result, it can be quickly determined if a VM is on the local disk. Depending on the environment, locally stored VMs may be prohibited if you use VMotion or certain VMware DRS configurations. This visibility presents an opportunity to see the variance from your standards. For example, lets say I build an environment of Windows Server 2003 VMs with 32 GB of storage assigned. In this example the shared storage resources are in increments of 320 GB, making it so that 9 VMs comfortably fit in each logical unit number which then becomes the datastore through the fiber channel interface, while leaving room for snapshots or .ISO files. If I see a datastore with a very large number of VMs attached, I can easily assume that they are very small. Or, in the situation where a VM is connected to multiple datastores, I can look into why that’s the case if it isn’t the norm. The most common example of a VM attached to multiple datastores is a CD-ROM image mapping to an .ISO image.