When you create a VMFS datastore on your VMware ESX servers many administrators select the default 1MB block size without knowing when or why to change it. The block size determines the minimum amount of disk space that any file will take up on VMFS datastores. So an 18KB log file will actually take up 1MB of disk space (1 block) and a 1.3MB file will take up 2MB of disk space (2 blocks). But the block size also determines the maximum size that any file can be, if you select a 1MB block size on your data store the maximum file size is limited to 256GB. So when you create a VM you cannot assign it a single virtual disk greater then 256GB. There is also no way to change the block size after you set it without deleting the datastore and re-creating it, which will wipe out any data on the datastore.
Because of this you should choose your block size carefully when creating VMFS datastores. The VMFS datastores mainly contain larger virtual disk files so increasing the block size will not use all that much more disk space over the default 1MB size. You have the following choices when creating a datastore:
• 1MB block size – 256GB maximum file size
• 2MB block size – 512GB maximum file size
• 4MB block size – 1024GB maximum file size
• 8MB block size – 2048GB maximum file size
Besides having smaller files use slightly more disk space on your datastore there are no other downsides to using larger block sizes. There is no noticeable I/O performance difference by using a larger block size. When you create your datastore, make sure you choose your block size carefully. 1MB should be fine if you have a smaller datastore (less than 500GB) and never plan on using virtual disks greater then 256GB. If you have a medium (500GB – 1TB) datastore and there is a chance that you may need a VM with a larger disk then go with a 2MB or 4MB block size. For larger datastores (1TB – 2TB) go with a 4MB or 8MB block size. In most cases you will not be creating virtual disks equal to the maximum size of your datastore (2TB) so you will usually not need a 8MB block size.
So remember, choose carefully, once you create your datastore there is no changing it later.
On more than one occasion, I have built a new ESX host system and had immediate migration issues with 64-bit guest operating systems. This is due to an easy-to-forget BIOS default that disables 64-bit processing. To save you the hassle of trying to diagnose why the migration may fail, here is a specific configuration that can prevent this issue.
The symptom is that the virtual machine migration of a 64-bit guest VM fails with the “CPU on host is incompatible with CPU feature requirements of virtual machine” message due to the lack of 64-bit support on the BIOS. To resolve the configuration issue, check the server’s processor configuration to ensure that proper support is enabled. On Dell PowerEdge series servers, the option to enable the functionality so that ESX permits the 64-bit migrations correctly is shown below:
This option can be accessed by pressing F2 during the boot process to gain entry into the BIOS. Go to the CPU Information section to enable the 64-bit support functionality option. Check your server documentation for other platforms.
Once this setting is enabled, you can migrate the 64-bit virtual machines without issue. This situation makes advanced testing on BIOS updates on your ESX servers a very good idea as the migration functionality may be impacted. More information on this issue and related processor topics related to this are discussed on a VMware Communities thread.
Sean Clark is a VMware Certified Professional (VCP) and a member of the Des Moines, Iowa-based Central Iowa Virtualization Users’ Group (CIVUG). The CIVUG emerged from the Central Iowa Linux Users’ Group as a way for virtualization users to learn more without being tied to one vendor. That is, the CIVUG discusses Hyper-V and Citrix Systems’ Xen in addition to VMware.
Clark has cut his teeth on VMware and other virtualization platforms as a solutions architect at Alliance Technologies, where he works with small and medium—sized businesses and some enterprise businesses to develop strategies to best implement virtualization, be it storage, server or application virtualization. Clark posted notes on a presentation he did at a recent meeting on Red Hat Network File System (NFS) configuration being used with VMotion, High Availability (HA) and Distributed Resource Scheduler (DRS).
NFS vs. Fibre Channel and iSCSI
The NFS configuration how-to Clark followed was posted by Mike Laverick on RTFM Education and was chosen for the demo not only because of it’s easy configuration, but also because implementing Red Hat NFS is a good low-cost alternative to shelling out big cash for more expensive devices for virtual machines (VM) storage. “Most people think that you need a Fibre Channel SAN [storage area network] or an iSCSI SAN; you’ll spend at least 20 or 30 grand on that device. But all you really need is a reasonably new server with some pretty fast discs, and you can basically have a SAN for the cost of your hardware.”
Clark also says that despite what some benchmarking stats say, performance won’t suffer on the cheaper NFS. “If you do your homework, you’ll find that in many benchmarks when you compare the performance of NFS versus iSCSI, they’re almost nearly identical. You can spin your benchmarking tasks so that one will come out on top every time, but for the majority of uses out there, and especially for test and dev, NFS is just as good as iSCSI.”
The storage system lines blur when you scale out to enterprise-level requirements because the needs go beyond the hardware and software backing your VM storage. Clark says that it’s really about having enough discs. “It doesn’t matter if you have NFS, Fibre Channel or iSCSI if you don’t have enough discs to meet the I/O demand that the number of virtual machines can put on a SAN. Most of the time, it becomes almost a religious decision — whatever you’re most comfortable with.”
HA snafu highlights proper network configuration
After walking through the NFS configuration, Clark went on to demo VMotion, HA and DRS, but ran into some problems with the HA portion. At his home in Pella, Iowa, Clark has a small test lab of two 1U servers running two ESX VMs, which he brought to Des Moines for his presentation.
With little time to reconfigure the servers (Clark was asked only a few days prior to the meeting to give the presentation), Clark decided against bringing his PC that he had used as his NFS server in favor of a Red Hat VM running on his laptop, which served as the basis for his presentation. Although both the ESX servers were able to see the Red Hat VM through a little Linksys switch, the demonstration came to a standstill during the HA portion.
The problem? Because the conference room that the VUG meets in is an island network, there is no default gateway address available as there was at Clark’s lab in Pella. An available, ping-able gateway address is required for HA to work. This is because when HA is set up, each ESX server establishes and communicates through a heartbeat and that’s how it determines whether each server is awake or not. If that heartbeat ceases, then HA makes a decision to restart the VMs that are running on a different ESX server. Sometimes the network becomes unavailable, but the other ESX server continues to run.
The isolation address was the pain point in Clark’s HA demo. The default gateway that used to exist in Pella didn’t exist and HA failed. As Clark explains, “It’s just an additional IP address that an ESX server can gain to say, ‘OK, I lost my buddy, but I need to make sure that the network is still up.’ So as long as it can reach that other IP address, then it can assume that the other host is actually truly dead and that it needs to restart its failed VMs.”
Proper planning and redundancy is key to successful deployments
Clark says that he almost never has problems because he is careful to use VMware-supported hardware and plans out each deployment carefully. But not all IT departments are so careful. “I ran into a customer the other day that didn’t want to invest in redundant networking; they had a four-host cluster and all their networking was going through one physical switch. They had HA set up, and when that switch went down, ever single VM that was on one of their ESX servers powered down.”
Clark says that this can happen as a result of not properly planning your configuration. “One of the configuration options for HA [concerns] what to do if an ESX host becomes isolated. Because it lost its network, there wasn’t a second physical switch for redundancy. Each host, even though it was still running, became isolated according to the HA configuration. It was probably a default setting at the time when they set up the HA cluster, and it powered down all the VMs.” Clark says that with proper preparation, the ESX hosts may not have appeared isolated to HA and this organization may have been able to save the headache of restarting all its VMs and troubleshooting what caused the power-down.
When VMware issued VirtualCenter 2.5 Update 1 and ESX 3.5 Update 1 in April of this year, I felt that this release was mature enough to upgrade my environment. One observation I have made is related to the VMware Infrastructure Client (VIC) versioning. The VMware Infrastructure 3 release notes mention many configurations related to supported environments of VirtualCenter and ESX, but do not distinguish between the base release of the VIC for VirtualCenter 2.5 and 2.5 Update 1.
The VIC for VirtualCenter 2.5 (base release) has a build number of 64192 whereas the VIC for VirtualCenter 2.5 update 1 has a build number of 84767. Unlike the upgrade from VirtualCenter 2.0 to VirtualCenter 2.5, connections into VirtualCenter are not prompted to download and install the new version of the VIC. Therefore, you can have VIC installations with the older version interacting with the newer VirtualCenter server. I recommend that you enforce the version currency of the VIC on all systems connecting to a VirtualCenter 2.5 Update 1 to be at build 84767 so that commands match the capabilities of VirtualCenter. For most situations, this is not going to be an issue, but should you use VirtualCenter plug-ins, advanced features, or have a large amount of VIC traffic there may be some anomalies in the version mismatch.
The figure below shows two information screens from two installations of the VIC. Both of these installations are interfacing to the same VirtualCenter server, but the top one is at the current version:
During the VirtualCenter upgrade process, the VIC is updated locally on the server. Further, unlike the VIC for VirtualCenter 2.0 the VIC for VirtualCenter 2.5 and 2.5 Update 1 have the same binary update path, so you will not have to manage two shortcuts for access.
When logging into the VMware Infrastructure Client (VIC) or using the web interface to the VirtualCenter server, you are presented with a certificate message. Best practice or not, I usually accept the certificate and instruct the software to not ask me again about this topic. I have just completed end-to-end testing of VirtualCenter 2.5 update 1 and ESX 3.5 update 1, however, and noticed something about the certificate store for VirtualCenter.
The VirtualCenter certificate store is valid for two years from the date of initial installation. Since this instance of VirtualCenter was installed, I have upgraded to version 2.5 base release and, most recently, to the version 2.5 update 1. But neither installation updated the certificate. Below are the details of my test certificate:
The good news is that you now know about this issue. The bad news is that you better correct it before the two year anniversary of your installation of VirtualCenter as it is required to process logins. VMware has a comprehensive PDF that outlines the certificate procedures for VirtualCenter and the ESX hosts. The ESX hosts, however, have a much longer lifespan for the local certificate, around 20 years, and do not exhibit this behavior. The VMware server certificate documentation is available for download from the VMware website.
I have been using VMware Server 2.0 (beta 2) on both Windows and Linux platforms for a while now. For Windows systems that are a member of an Active Directory domain, there are inherited permissions that may be assigned from Group Policy. If you want to change that, here are a couple of pointers in changing the security model.
To start looking at the permissions of the server installation, click the server on the left side of the browser view and then click the permissions tab at the top. The default permission is the local Administrators of the Windows system will be in the VMware Server Administrators role as shown below:
Before you make any modifications to the security model, add your desired configuration. That way, you can protect yourself from orphaning your administrative access to the server. Click the New Permission link in the command section, and add the user from either the local accounts of the Windows system or a user from the Active Directory domain or security group from the domain. The figure below shows the addition of the RWVVMwareAdmins group to the role of Administrator within the VMware Server 2.0 web interface:
Once that is added, the new configuration should be tested to ensure the proper access is available. Once the new access is verified, it would be safe to then remove the previous default access (if needed). If you get stuck, you can save off the .VMDK files and reinstall the product if needed.
While the web interface for VMware Server 2.0 takes some getting used to when compared to the thick client for versions 1.0x, features continue to be added to the free virtualization product that can be suited to test and development or live environments.
At Computex Taipei 2008on June 3, VMware, Inc. announced new relationships with original design manufacturers ASUS, Gigabyte, Inventec and Tyan, adding to VMware’s current relationship with Supermicro announced in February 2008.
The relationships lets channel partners and system builders make virtualization available to a broader range of customers by certifying various one-, two- and four-socket servers and blade servers for the VMware virtualization platform.
The vendors are certifying their server systems for the VMware ESX and ESXi hypervisor, with availability expected by the end of the third quarter of 2008.
VMware, Inc. today announced it has entered into a definitive agreement to acquire B-hive Networks, Inc., a privately-held application performance management software company with headquarters in San Mateo, California and principal R&D facilities in Herzliya, Israel.
By acquiring the company, VMware will add B-hive’s technology for performance management and service level reporting for applications running within VMware virtual machines on both servers and desktops. In addition, B-hive’s R&D facility and team will be the core of VMware’s new development center in Israel.
The terms of the B-hive acquisition, which is expected to be completed during the third quarter of 2008, subject to customary closing conditions, were not disclosed. This is VMware’s seventh acquisition in the past year.
VMware’s President and CEO Diane Greene said during a JP Morgan technology conference in Boston last week that VMware will continue acquiring companies as a growth strategy. The company grew 69% last quarter compared to the same quarter in 2007.
“We are always looking for technologies we don’t have. In the past year we have bought six small companies; we look to grow organically and through acquisitions,” Greene said at the conference.
What is B-hive?
Founded in 2005, B-hive developed a technology that gives visibility into application performance in virtual environments, such as end-user transaction response time, virtual machine utilization and cross-virtual machine dependencies. Unlike operating system-based performance monitoring products, B-hive’s product is designed to measure performance across multi-tier or service-oriented architecture applications that are distributed across clusters of ESX hypervisors and virtual machines.
B-hive’s flagship product, an agentless virtual appliance called B-hive Conductor, which was a “Best of VMworld” finalist at VMworld 2007, monitors end-user performance and issues service level reports, and also proactively resolves application performance problems by automatically triggering actions such as dynamically allocating more resources, migrating the application to a different server, provisioning additional virtual machines, changing transaction routing, or system reboots, accoridng to VMware.
For example, if B-hive identifies degradation in application response time, it can remediate the problem by automatically instructing VMware Infrastructure to adjust the resources allocated to the application or provision an additional virtual machine with an additional instance of the application, according to VMware.
Microsoft recently published their Integrated Virtualization – ROI Tool, and I thought “That’s great. When the time comes I will have the ability to provide Microsoft branded reports to support a Microsoft virtual infrastructure opportunity.” I did not take the time to check out calculator first, but I assumed (or hoped) that it would provide clear answers about licensing costs and the confusing licensing options for virtualization. It wasn’t until I read VMware’s post Microsoft’s Virtualization ROI/TCO Calculator: Our Take that I decided I’d better understand what Microsoft’s ROI calculator produced.
Specifically, VMware’s post asked me to consider the inaccuracies they found:
“… evaluate the Microsoft calculator yourself – let us know what else you find! “
So I did just that, but it wasn’t how Microsoft calculated the numbers that bothered me most. I struggled to understand why a TCO and ROI calculator included a competitive analysis. After all, VMware’s TCO calculator doesn’t compare the cost of competitor’s products. What does that have to do with return on investment? It just seems out of place to me. Furthermore, if you go back and review VMware’s points they are mostly about the competitive cost comparison, too. It’s easy to forget we are discussing a TCO / ROI calculator.
As for using the calculator for ROI, it’s fine, I guess. But it did not live up to my basic expectations of helping with licensing. In fact, intentional or not, Microsoft comes across as trying to hide accurate licensing costs, as VMware points out:
“We did find a one-line disclaimer buried in the 66-page document: “Warning – Check pricing advice and rules as the automated recommendations here may not reflect all licensing rules.” Come on, guys – licensing is such a basic component for accurate TCO estimates. The disclaimer feels pretty weak.”
On the other hand, I think I can use the Windows Server Virtualization Calculators to help estimate licensing costs. But I shouldn’t have to use another calculator to verify the first one. Overall, I am left with the same feeling I get when trying to buy a new car. It’s similar to that doubt about the “dealership transport” charges or the frustration of feeling that I’m missing hidden costs even though the price is right. I am being forced to do way too much research.
I understand that the next few years will be filled with explaining the technical and financial differences between Microsoft, Citrix, VMware and all the other virtualization products. A competitive analysis calculator would come in handy. A single, unbiased virtualization competition calculator might be impossible to create, but even separate tools from each vendor that let you enter your own pricing numbers would be a great start. Call these tools what they really are. Don’t hide them in an ROI Calculator.
I like VMware. I like Google. Heck, both of them keep me more than busy with development ideas. But I have a problem with them. Google started it with Gmail. Although it is hard to remember now, Gmail was in beta forever. Oh wait, it still is? Huh. I guess I just figured it *must* have hit production by now. Then there is Google News, Google Apps, Google Page Creator, Google everything else — all beta . I am honestly surprised search hits don’t come back with the “beta” tag next to them. I guess they thought ICQ was the cat’s meow, and that the whole beta thing had a nice ring to it.
Enter VMware, which is perilously close to become the next Google in terms of heavily pushing new features, but then labeling them as beta or experimental. Take for example Storage VMotion (SVMotion). VMware played up this new feature to VI 3.5 last fall at their North American VMworld conference, but when it was release there was no graphical user interface (GUI) option for it. How is that ready for prime-time? And then there is virtual machine (VM) high availability (HA), another marketed feature that is so experimental you have to edit an advanced setting (as a free-form string) just to enable the functionality.
I wouldn’t actually have a problem with VMware doing this if they didn’t market the heck out these new “features.” Excuse me for being old fashioned, but it isn’t enterprise-ready if it is beta or labeled experimental. And VMware makes no bones about this; they plainly state that these features should not be used in production. However, on the other hand they make a big show about the same set of features, whipping the crowd to a fever pitch of excitement. You can’t have it both ways, guys.
Take VMware Fusion 2 or VMware Server 2. These products are in beta stages right now and VMware is not making a big deal about them. Sure, they are out there for people to get, but VMware isn’t throwing them at customers, not the way they revolved last year’s North American and this year’s European VMworld conferences on features that were not even ready for production.
Then there is the other end of the spectrum as well. I recently discovered that VMware is strategically hiding a long sought feature of ESX in the bowels of its software development kit (SDK). Since version 2.5 of the SDK (VI 3.5), VMware has included the ability (although it does not yet appear to be working correctly) to create network address translation (NAT) and dynamic host control protocol (DHCP) devices directly on ESX servers for VMs to use. This is awesome! Prior to this, the only way to create NATd networks on an ESX host was to dual home a VM to a public and private port group, have it act as the NAT and DHCP server, and then attach other VMs to the same network as its private interface. This solution was cumbersome and did not work well when VMotioning VMs. If I was VMware, I would make a little bit more noise about the fact that they are working on this feature.
I want to reiterate that I like, if not love, VMware. I just hate getting jazzed about a new feature that they have thrown at me, only to find out that it is a curve ball. VMware needs to make sure that features that are experimental should be announced with an asterisk next to their headline, while at the same time working a little harder to ensure that some other upcoming features get the love they deserve.