VMware ESX

Performance tips from the summer New England VMUG; attendees sound off

425 IT geeks invaded Brunswick, Maine last Thursday for the New England VMware Users Group summer meeting at Brunswick High School. The event, which started at 10 AM with a sponsor showcase and proceeded with after-lunch sessions, concluded in a Lobster/Clambake at Gritty’s in Freeport, Maine.

 “Good performance starts with proper planning and design, configuration best practices and operational awareness,” said Mike Burke (left), Virtual Infrastructure Practice Director for Virtera, a CT-based consulting firm during the first session I attended equivocally titled “Getting the most out of your ESX Environment – Performance tuning.”

“Tuning is not a replacement for proper planning and design,” Burke said. “Don’t guess; gather the data.”

Planning trumps tuning in VMware deployments
Performance, Burke said, hinges on examining and understanding how CPU, memory, networking and disk space will work together in a virtual infrastructure. Planning, then, is the key to better ESX performance, and not necessarily tuning.

Generally speaking, more cores are better; so plan on going big. “How does VMware license their product? Per socket. If you can choose a quad-core, it’s a lot more economical,” Burke said, citing that AMD series processors are now “dirt cheap” compared to earlier years, with eight core boxes going for around $2000.

“You need to weigh what the workload is against what the goals of virtualizing it are,” Burke said. For example, a four-CPU physical server may not be a good candidate for a server consolidation project if it’s currently using the four CPUs to the fullest extent, because you won’t have a good consolidation ratio on those systems. It may, however, be a candidate if the goal is overall server virtualization instead of consolidation.

Another reason for this type virtualization is hardware capability. You can’t VMotion from AMD to Intel. Even in servers from the same line, Burke said, older hardware instruction sets coded in the CPU won’t look the same, preventing VMotion capability.

More host RAM gives better overall performance, configure a VM’s memory based on actual need – are you really using the full two GHz, or would it better used elsewhere? On a general note, though, Burke said “oversubscribing,” or provisioning more memory for virtual machines in the physical host server, is better than not.

Advanced configurations
A systems administrator can, however, tweak a few advanced variables in a virtual machine’s configuration or vmx file. For memory, you can reset the time interval for transparent page sharing (TPS), which is automatically set at 60 seconds, by using mem.ShareScanTime. Adjust the memory pages to scan per 1 GHz idle cycle, automatically set at 4 MB/sec per 1GHz by using mem.ShareScanGhz. mem.CtlMaxPercent limits memory reclamation by ballooning.

But tweaking these parameters, Burke warned, can detract from virtualizing your systems. “You’re instructing ESX to spend more time processing and looking for memory savings rather than sitting in the background and running the VMs, which diverts resources away from virtualizing guests to handling these configurations on the backend.”

On a per-VM level, you can use the same mem.CtlMaxPercent command, and also: sched.mem.maxmemstl, which caps the max memory reclamation by ballooning (vmmemctl); sched.mem.pshare.enable – enables/disables memory sharing (TPS); lastly, using sched.swap.persist can enable vSwap to persist after power-off.

New England VMware Users Group summer meeting attendees sound off
Other sessions spanned from technical sessions about VMware DRS and HA to VDI demonstrations, SRM implementation and general topics, such as managing the virtual data center and how to minimize VM sprawl. Some sessions were led by speakers from consulting companies; others were 45 minute long vendor sales pitches disguised as learning sessions, but overall attendees had a good experience.

“For me, the sessions are good, but I can learn the same material in a textbook or on a website,” said Lee Pullen of the Maine Education Association. “The largest benefit is the networking, meeting other users and finding out what they’re doing in their own environments. That’s the real value in attending.”

“I found a few of the titles of the sessions misleading,” said State of Maine Office of Information Technology employee Lori Blier. “The last session I was at seemed like it was going to be about getting more out of your virtual infrastructure, which is what it was called. But they mainly talked about virtual desktops.”

Perhaps next year, the organizers could include descriptions of the presentations so that participants know exactly what to expect, Blier suggested. She said that she liked the sessions that discussed managing and using ESX Server, which is what the state of Maine plans to upgrade to pending management approval. They currently use VMware Server in production.

“I was a little disappointed that there were no sessions or vendors that focused on the networking side,” commented Kris Kirby from the Vermont Agency of Natural Resources, who said that’s his main concern with his VMware environment. “I also thought it was going to start a little earlier.”

Chris Harney, a systems engineer from Maine who organizes the New England Users Group meetings, said that feedback from the last session indicated increased interest in virtual desktop information sessions, which is why several of the sessions catered to that track. He also said that if networking came up as a trend on the feedback sheets, it would be voted on by a focus group and most likely included in the program for the January winter meeting, which will be held in Massachusetts.

“We’ve only been doing this for a couple years, and we’re still trying to figure out what works,” Harney said. “This is just a hobby for me, but it’s almost a full-time job. When we first started, we had 35 people. Now we’re looking at almost 500 per meeting.”

Virtualization virtual tradeshow offers VMware networking opportunities

Trade shows are great – if you have time to attend, have staff to cover while you’re away learning about a new technology, can avoid summons back to the office during the show, can find a show in your local area or can get budget approval to attend a show that requires flight or hotel reservations.

Enter the virtual trade show (VTS); an online conference conceived to mitigate the above challenges. Last week, sister sites SearchDataCenter.com and SearchServerVirtualization.com hosted an advanced enterprise virtualization VTS. I helped staff the networking lounge and editorial booth where I had the opportunity to chat with VMware users about two of the virtualization provider’s newest tools, Site Recovery Manager (SRM) and Storage VMotion.

IM chatting with attendees
Conversations ranged from general IT talk (“Anyone use virtual desktops?”) to small talk (“What’s the weather like in Maine?”). Trying to be the friendly host, I said “Good morning” to the room. I immediately got the reply “Good evening” and was subsequently told this particular user was signed on from –literally–the other side of the world.

I ended up chiming-in on another user’s question about if anyone was familiar with VMware Site Recovery Manager (SRM). The respondent had said that he was, and I ended up asking him about his experience via private IM. SRM orchestrates your virtual machine disaster recovery (DR) plan in the event that your main data center goes down. It prioritizes which virtual machines (VMs) are brought up at the failover site based on available resources, syncs your VM configurations between the main site and the failover site, and allows for DR plan testing without having to take the system offline. It’s a relatively new addition to the VI3 lineup, having been on the market for four months (at the time of publication).

Our conversation turned to plug-ins, and he raved about Andrew Kutz’s Storage VMotion plug-in. The plug-in adds a user interface to the out-of-the-box product, which operates through a command line interface. The attendee explained that he’s primarily a “Windows guy,” so a graphical user interface makes using Storage VMotion much easier.

Kutz recently released an update to the Storage VMotion plug-in.

“The new release now ignores raw device mapping,” Kutz said. “Previously, if you had a raw device that pointed to a 300 Gig disk, the plug-in would look at it as an actual disk and screw up the disk size map.”

He also removed the majority VMware’s internal code from the plug-in (excepting the code that loads the plug-in), replacing it with code based on the VI Toolkit for .NET.

Impressive user interface
The VTS emulates the look of a physical tradeshow floor, which makes navigation a bit friendly, though not as intuitive as I would have liked. You could either move around with the help of a clickable navigation bar, or point-and-click your way from the main entryway to the desired location, be it the conference hall, vendor hall, networking lounge or “library” where you can download PDFs of presentations and various information from vendors, which then moves into your “suitcase,” displayed on your personal page.

VTSs are essentially fancy webcast packages displayed in unconventional ways. In this particular show, the topics were “Protecting your Virtual Environment: Backup and Storage,” “Virtual Infrastructure Automation and High Availability Best Practices” and “Virtual Infrastructure Tuning and Advanced Management.” The speaker was displayed on the left side of the screen presenting his slides via streaming video. The slides were displayed on the right hand side. Users could ask questions via a box at the bottom of the screen.

The VTS, if done correctly, has many more plusses than minuses. As long as there is a reliable Internet connection, there’s no need to leave the data center (if you don’t have a reliable connection in your data center, you might think about leaving for good). The content is almost exactly the same as at a physical trade show (that’s how they got the video of the speaker to begin with). And editorial staff can send IT pros direct links to helpful guides that they know of if an IT pro wants to know about, for example, virtual desktop drawbacks.

If any SearchVMware.com readers passed up the opportunity to “attend” a virtual trade show, I suggest you test it out next time a topic of interest comes around. It’s actually fun to use (think AOL in the 90’s minus the “you’ve got mail”) and offers great learning potential and networking opportunities.

An archived version of the advanced enterprise virtualization virtual trade show is available online, short registration required.

Automatic recovery for failed virtual machines

A little known feature, called Virtual Machine Failure Monitoring (VMFM), was introduced in ESX 3.5. VMFM offers the ability to leverage HA to monitor VMs for operating system failures, such as blue screens, and have them automatically restarted. Previously, HA would only deal with ESX host failures by automatically restarting VMs on alternate ESX hosts in the event of a problem with the host server.

VMFM also extends HA to monitor VMs through a heartbeat sent every second when using VMware Tools. This new feature is disabled by default and is considered ‘experimental’ by VMware. This typically means it works but it is not officially supported for production use yet. In order for this feature to function properly you must first ensure the following conditions exist:

• ESX hosts are version 3.5
• VirtualCenter is version 2.5
• VMware Tools is installed on VMs and is the latest version
• You have a Cluster configured and HA enabled

To enable it follow the below steps:

• Edit the Settings for your Cluster
• Choose VMware HA and click the Advanced Options button
• Add the following Options and Values

das.vmFailoverEnabled – true (true or false)
das.FailureInterval – 30 (declare virtual machine failure if no heartbeat is received for the specified number of seconds)
das.minUptime – 120 (After a virtual machine has been powered on, its heartbeats are allowed to stabilize for the specified number of seconds. This time should include the guest operating system Boot up time)
das.maxFailures – 2 (Maximum number of failures and automated resets allowed for the time that das.maxFailureWindow specifies. If das.maxFailureWindow is ‐1 (no window), das.maxFailures represents the absolute number of failures after which automated response is stopped and further investigation is necessary)
das.maxFailureWindow – 86400 (Either -1 or a value in seconds. If das.maxFailures is set to a number, and that many automated resets have occurred within that specified failure window , automated restarts stop and further investigation is necessary)

I enabled this on a Cluster and tested it by simulating a blue screen on a VM running Windows 2003 Server and it worked perfectly. After 30 seconds the loss of heartbeat was detected and the VM was automatically restarted. Currently there are no notification alerts that can be configured when this occurs. That is, if you check the events for the VM you will see no evidence of this happening. The only mention of it that I found in the logs was in the hostd log on the ESX server ([2008-06-26 11:47:22.552 ‘ha-eventmgr’ 3076440992 info] Event 101 : VM1 on Esx1.xyz.com in ha-datacenter is reset). Hopefully this will change in a later version when the feature is no longer considered ‘experimmental’. You can read more about this new feature in a white paper that VMware has provided on it.

Isolate ESX hosts in separate clusters for maintenance, upgrades

Performing key maintenance on a VMware Infrastructure 3 (VI3) host OS or can be an involved process, as can the process of simply adding a host. Because of the length of time that a system may be offline or because of what exactly is being performed, having the hosts in an isolated cluster can allow a safe environment for virtually any task. An isolated cluster will not apply to the same DRS and high availability rules that may apply to the live cluster. Further, you can reboot the host as needed without needing to wonder if there will be any effect to the live workload.

With the isolated cluster, the following tasks can be safely performed:

-Version upgrades (ESX 3.0x to 3.5 Update 1)
-Hardware maintenance
-Adding a new host to an existing cluster
-Testing network connectivity to various port groups
-Confirming VMware HA and DRS performance with in an isolated set of rules
-Importing or configuring new storage types

A good practice for the isolated cluster would be to keep it less visible to the live workload and named accordingly within the VMware Infrastructure Client, so a name like “TestingCluster” or “ZZUpgradeCluster” can distinguish the collection to be different from that of the live workload.

The figure below shows a cluster with a name and position that contains one host in maintenance mode for any task better suited in an isolated environment:

It is important to note that this cluster would still require licensing as it would be in live workload cluster. More information on creating a VI3 cluster can be found in the VI3 online library.

Choosing a block size when creating VMFS datastores

When you create a VMFS datastore on your VMware ESX servers many administrators select the default 1MB block size without knowing when or why to change it. The block size determines the minimum amount of disk space that any file will take up on VMFS datastores. So an 18KB log file will actually take up 1MB of disk space (1 block) and a 1.3MB file will take up 2MB of disk space (2 blocks). But the block size also determines the maximum size that any file can be, if you select a 1MB block size on your data store the maximum file size is limited to 256GB. So when you create a VM you cannot assign it a single virtual disk greater then 256GB. There is also no way to change the block size after you set it without deleting the datastore and re-creating it, which will wipe out any data on the datastore.

Because of this you should choose your block size carefully when creating VMFS datastores. The VMFS datastores mainly contain larger virtual disk files so increasing the block size will not use all that much more disk space over the default 1MB size. You have the following choices when creating a datastore:

• 1MB block size – 256GB maximum file size
• 2MB block size – 512GB maximum file size
• 4MB block size – 1024GB maximum file size
• 8MB block size – 2048GB maximum file size

Besides having smaller files use slightly more disk space on your datastore there are no other downsides to using larger block sizes. There is no noticeable I/O performance difference by using a larger block size. When you create your datastore, make sure you choose your block size carefully. 1MB should be fine if you have a smaller datastore (less than 500GB) and never plan on using virtual disks greater then 256GB. If you have a medium (500GB – 1TB) datastore and there is a chance that you may need a VM with a larger disk then go with a 2MB or 4MB block size. For larger datastores (1TB – 2TB) go with a 4MB or 8MB block size. In most cases you will not be creating virtual disks equal to the maximum size of your datastore (2TB) so you will usually not need a 8MB block size.

So remember, choose carefully, once you create your datastore there is no changing it later.

64-bit guest operating system issues avoided with BIOS configuration

On more than one occasion, I have built a new ESX host system and had immediate migration issues with 64-bit guest operating systems. This is due to an easy-to-forget BIOS default that disables 64-bit processing. To save you the hassle of trying to diagnose why the migration may fail, here is a specific configuration that can prevent this issue.

The symptom is that the virtual machine migration of a 64-bit guest VM fails with the “CPU on host is incompatible with CPU feature requirements of virtual machine” message due to the lack of 64-bit support on the BIOS. To resolve the configuration issue, check the server’s processor configuration to ensure that proper support is enabled. On Dell PowerEdge series servers, the option to enable the functionality so that ESX permits the 64-bit migrations correctly is shown below:

This option can be accessed by pressing F2 during the boot process to gain entry into the BIOS. Go to the CPU Information section to enable the 64-bit support functionality option. Check your server documentation for other platforms.

Once this setting is enabled, you can migrate the 64-bit virtual machines without issue. This situation makes advanced testing on BIOS updates on your ESX servers a very good idea as the migration functionality may be impacted. More information on this issue and related processor topics related to this are discussed on a VMware Communities thread.

Storage and network planning takeaways from Iowa virtualization user group meeting

Sean Clark is a VMware Certified Professional (VCP) and a member of the Des Moines, Iowa-based Central Iowa Virtualization Users’ Group (CIVUG). The CIVUG emerged from the Central Iowa Linux Users’ Group as a way for virtualization users to learn more without being tied to one vendor. That is, the CIVUG discusses Hyper-V and Citrix Systems’ Xen in addition to VMware.

Clark has cut his teeth on VMware and other virtualization platforms as a solutions architect at Alliance Technologies, where he works with small and medium—sized businesses and some enterprise businesses to develop strategies to best implement virtualization, be it storage, server or application virtualization. Clark posted notes on a presentation he did at a recent meeting on Red Hat Network File System (NFS) configuration being used with VMotion, High Availability (HA) and Distributed Resource Scheduler (DRS).

NFS vs. Fibre Channel and iSCSI
The NFS configuration how-to Clark followed was posted by Mike Laverick on RTFM Education and was chosen for the demo not only because of it’s easy configuration, but also because implementing Red Hat NFS is a good low-cost alternative to shelling out big cash for more expensive devices for virtual machines (VM) storage. “Most people think that you need a Fibre Channel SAN [storage area network] or an iSCSI SAN; you’ll spend at least 20 or 30 grand on that device. But all you really need is a reasonably new server with some pretty fast discs, and you can basically have a SAN for the cost of your hardware.”

Clark also says that despite what some benchmarking stats say, performance won’t suffer on the cheaper NFS. “If you do your homework, you’ll find that in many benchmarks when you compare the performance of NFS versus iSCSI, they’re almost nearly identical. You can spin your benchmarking tasks so that one will come out on top every time, but for the majority of uses out there, and especially for test and dev, NFS is just as good as iSCSI.”

The storage system lines blur when you scale out to enterprise-level requirements because the needs go beyond the hardware and software backing your VM storage. Clark says that it’s really about having enough discs. “It doesn’t matter if you have NFS, Fibre Channel or iSCSI if you don’t have enough discs to meet the I/O demand that the number of virtual machines can put on a SAN. Most of the time, it becomes almost a religious decision — whatever you’re most comfortable with.”

HA snafu highlights proper network configuration
After walking through the NFS configuration, Clark went on to demo VMotion, HA and DRS, but ran into some problems with the HA portion. At his home in Pella, Iowa, Clark has a small test lab of two 1U servers running two ESX VMs, which he brought to Des Moines for his presentation.

With little time to reconfigure the servers (Clark was asked only a few days prior to the meeting to give the presentation), Clark decided against bringing his PC that he had used as his NFS server in favor of a Red Hat VM running on his laptop, which served as the basis for his presentation. Although both the ESX servers were able to see the Red Hat VM through a little Linksys switch, the demonstration came to a standstill during the HA portion.

The problem? Because the conference room that the VUG meets in is an island network, there is no default gateway address available as there was at Clark’s lab in Pella. An available, ping-able gateway address is required for HA to work. This is because when HA is set up, each ESX server establishes and communicates through a heartbeat and that’s how it determines whether each server is awake or not. If that heartbeat ceases, then HA makes a decision to restart the VMs that are running on a different ESX server. Sometimes the network becomes unavailable, but the other ESX server continues to run.

The isolation address was the pain point in Clark’s HA demo. The default gateway that used to exist in Pella didn’t exist and HA failed. As Clark explains, “It’s just an additional IP address that an ESX server can gain to say, ‘OK, I lost my buddy, but I need to make sure that the network is still up.’ So as long as it can reach that other IP address, then it can assume that the other host is actually truly dead and that it needs to restart its failed VMs.”

Proper planning and redundancy is key to successful deployments
Clark says that he almost never has problems because he is careful to use VMware-supported hardware and plans out each deployment carefully. But not all IT departments are so careful. “I ran into a customer the other day that didn’t want to invest in redundant networking; they had a four-host cluster and all their networking was going through one physical switch. They had HA set up, and when that switch went down, ever single VM that was on one of their ESX servers powered down.”

Clark says that this can happen as a result of not properly planning your configuration. “One of the configuration options for HA [concerns] what to do if an ESX host becomes isolated. Because it lost its network, there wasn’t a second physical switch for redundancy. Each host, even though it was still running, became isolated according to the HA configuration. It was probably a default setting at the time when they set up the HA cluster, and it powered down all the VMs.” Clark says that with proper preparation, the ESX hosts may not have appeared isolated to HA and this organization may have been able to save the headache of restarting all its VMs and troubleshooting what caused the power-down.

Ensure VMware Infrastructure Client is current

When VMware issued VirtualCenter 2.5 Update 1 and ESX 3.5 Update 1 in April of this year, I felt that this release was mature enough to upgrade my environment. One observation I have made is related to the VMware Infrastructure Client (VIC) versioning. The VMware Infrastructure 3 release notes mention many configurations related to supported environments of VirtualCenter and ESX, but do not distinguish between the base release of the VIC for VirtualCenter 2.5 and 2.5 Update 1.

The VIC for VirtualCenter 2.5 (base release) has a build number of 64192 whereas the VIC for VirtualCenter 2.5 update 1 has a build number of 84767. Unlike the upgrade from VirtualCenter 2.0 to VirtualCenter 2.5, connections into VirtualCenter are not prompted to download and install the new version of the VIC. Therefore, you can have VIC installations with the older version interacting with the newer VirtualCenter server. I recommend that you enforce the version currency of the VIC on all systems connecting to a VirtualCenter 2.5 Update 1 to be at build 84767 so that commands match the capabilities of VirtualCenter. For most situations, this is not going to be an issue, but should you use VirtualCenter plug-ins, advanced features, or have a large amount of VIC traffic there may be some anomalies in the version mismatch.

The figure below shows two information screens from two installations of the VIC. Both of these installations are interfacing to the same VirtualCenter server, but the top one is at the current version:

During the VirtualCenter upgrade process, the VIC is updated locally on the server. Further, unlike the VIC for VirtualCenter 2.0 the VIC for VirtualCenter 2.5 and 2.5 Update 1 have the same binary update path, so you will not have to manage two shortcuts for access.

VMware adds ODM relationships with Tyan, ASUS, Gigabyte, Inventec

At Computex Taipei 2008on June 3, VMware, Inc. announced new relationships with original design manufacturers ASUS, Gigabyte, Inventec and Tyan, adding to VMware’s current relationship with Supermicro announced in February 2008.

The relationships lets channel partners and system builders make virtualization available to a broader range of customers by certifying various one-, two- and four-socket servers and blade servers for the VMware virtualization platform.

The vendors are certifying their server systems for the VMware ESX and ESXi hypervisor, with availability expected by the end of the third quarter of 2008.

VMware acquires performance management software company B-hive Networks

VMware, Inc. today announced it has entered into a definitive agreement to acquire B-hive Networks, Inc., a privately-held application performance management software company with headquarters in San Mateo, California and principal R&D facilities in Herzliya, Israel.

By acquiring the company, VMware will add B-hive’s technology for performance management and service level reporting for applications running within VMware virtual machines on both servers and desktops. In addition, B-hive’s R&D facility and team will be the core of VMware’s new development center in Israel.

The terms of the B-hive acquisition, which is expected to be completed during the third quarter of 2008, subject to customary closing conditions, were not disclosed. This is VMware’s seventh acquisition in the past year.

VMware’s President and CEO Diane Greene said during a JP Morgan technology conference in Boston last week that VMware will continue acquiring companies as a growth strategy. The company grew 69% last quarter compared to the same quarter in 2007.

“We are always looking for technologies we don’t have. In the past year we have bought six small companies; we look to grow organically and through acquisitions,” Greene said at the conference.

What is B-hive?

Founded in 2005, B-hive developed a technology that gives visibility into application performance in virtual environments, such as end-user transaction response time, virtual machine utilization and cross-virtual machine dependencies. Unlike operating system-based performance monitoring products, B-hive’s product is designed to measure performance across multi-tier or service-oriented architecture applications that are distributed across clusters of ESX hypervisors and virtual machines.

B-hive’s flagship product, an agentless virtual appliance called B-hive Conductor, which was a “Best of VMworld” finalist at VMworld 2007, monitors end-user performance and issues service level reports, and also proactively resolves application performance problems by automatically triggering actions such as dynamically allocating more resources, migrating the application to a different server, provisioning additional virtual machines, changing transaction routing, or system reboots, accoridng to VMware.

For example, if B-hive identifies degradation in application response time, it can remediate the problem by automatically instructing VMware Infrastructure to adjust the resources allocated to the application or provision an additional virtual machine with an additional instance of the application, according to VMware.