High availability and virtualization

VMware releases update to VirtualCenter and VMware Converter

Last week, VMware released VirtualCenter 2.5 Update 3. The release fixes issues with Update 2 (build 104263), which was released in July, before the infamous product expiration issue. In the immediate releases after the problem, a corrected version of Virtual Center was not released.

This update fixes 19 various issues but does not provide any new functionality. Several of the resolved issues involve VMware High Availability and Distributed Resource Scheduler functionality — in the previous version, various situations prevented the functionality from working correctly.

VMware Converter Enterprise, now on build 62407, also has an update that will correct two minor issues with connecting to a VirtualCenter server for conversions. This is different than the VMware Converter Enterprise standalone edition, which installs on remote systems instead of the VirtualCenter server. The standalone version remains at version 3.0.3 (build 89816).

More information on VirtualCenter 2.5 Update 3 can be found in the release notes on VMware’s website.

VMware defends its upcoming fault-tolerance feature

During VMworld 2008 in Las Vegas last week, VMware Inc. announced its upcoming fault tolerance feature and gave a demonstration of it during one of the keynote sessions. It looked pretty good and simple to use, but Littleton, Mass.-based Marathon Technologies Corp., a company that specializes in fault tolerance software, had plenty to say otherwise.

In response to Marathon’s blog dissin’ the upcoming feature, Palo Alto, Calif.-based VMware’s Mike DePetrillo, a principal systems engineer, wrote a blog defending VMware Fault Tolerance.

For starters, Marathon complained that VMware does not provide component-level fault tolerance. “The most common failures that result in unplanned downtime are component failures such as storage, NIC [network interface card] or controller failures. Yet VMware Fault Tolerance doesn’t do anything to protect against I/O, storage or network failures.”

DePetrillo noted that VMware already has features to protect again component failure. “If your NIC fails you’ve got NIC teaming built into the system. To set it up simply plug in both NICs to the server, go into the network panel and attach both of them to the same virtual switch. Done. Four clicks. Same thing for storage with the built-in SAN [storage area network] multipathing drivers,” DePetrillo wrote. “I absolutely agree with the author that component failures are the cause of most crashes and that’s why VMware added these features in 2002. VMware FT is not designed for component failure because there’s no sense in moving the VM to another host if you’ve simply lost a NIC uplink. NIC teaming will take care of that with ease and is a LOT cheaper than using CPU and memory resources on another host to overcome the failure.”

Marathon’s second beef: VMware’s fault tolerance is too complex. “In order to use VMware Fault Tolerance, you’ll first have to install both VMware HA [High Availability] and DRS [Distributed Resource Scheduler]. No small feat in and of themselves. Then, because VMware FT requires NIC teaming, you’ll also have to manually install paired NICs. Then you’ll need to manually set up dual storage controllers (with the software to manage them) because it requires multipathing. And to top it all off, you’re required to use an expensive, and often complicated, SAN.”

DePetrillo said the process requires checking off two boxes - HA and DRS. That’s it. “If that’s too hard then please comment and let me know how it could possibly be easier. Even my dog has figured out how to do this now. Granted, it’s a pretty smart dog.”

“As for setting up the dual NICs and dual HBAs [host bus adapters], well, yes, you have to actually plug the physical devices in. After you’ve done that the **built-in** NIC teaming and HBA drivers will take over and configure most everything for you. The NIC teaming does require four extra clicks. The HBA drivers actually figure out the failover paths, match them up, and set up the appropriate form of failover all auto-magically. They’ve been doing this since ESX 1.5 (6 years ago),” DePetrillo blogged.

“Lastly, yes, this requires shared storage. Pretty sure that most environments that want FT (no downtime what-so-ever because our business could lose millions) already have a SAN to take advantage of other things virtualization related such as DRS and VMotion,” he wrote.

Also, VMware FT does not require dual NICs or dual HBAs because, DePetrillo said, “This is something you should have in every virtualization setup that’s running VMs you care anything about, but it’s not a requirement to get VMware FT [Fault Tolerance] running.”

The last point Marathon makes that’s worth spending any time on is that VMware  offers onlylimited CPU fault tolerance. “With VMware FT, you’ll need to set up what VMware refers to as a “record/replay” capability on both a primary and secondary server. If something happens to the primary server, the record is stored on the SAN and then restarted on the secondary server. … The whole thing depends on the quality of the SAN. Second, in the words of the VMware engineer who presented at VMworld, “this can take a couple of seconds.” So what happens to your application state in those couple of seconds?”

DePetrillo’s defense is that “if you’re the type of company that requires absolutely no downtime for an app — if the app is just that critical — then I’m pretty sure you’re going to have a decent SAN. … If you’re having so many problems with your SAN that you don’t trust it for FT, then you have much bigger issues at hand that VMware or Marathon or any of the other virtualization related vendors aren’t going to help you with.”

You can read more of VMware’s comments on DePetrillo’s blog, which gets into some details on how VMware Fault Tolerance will work, and vice versa for Marathon.

But I think it is obvious that Marathon is making VMware’s fault tolerance feature seem worse than it is, and VMware is making its new feature seem simpler than it is.

For the most part, this is a pissing contest between the incumbent fault-tolerance vendor and the “new guy,” but the fact of the matter is, if you use VMware virtualization, you can’t use Marathon Technologies because they don’t support VMware (obviously) and if you use Citrix Systems’ XenServer, you can’t use VMware Fault Tolerance, so these arguments are moot.

VMware helps hospital reduce data center power, increase performance

Palo Alto, Calif.-based VMware, Inc. announced that Rochester General Hospital(RGH) deployed VMware Infrastructure 3 to scale and manage its growing IT environment.

RGH, a community-based teaching hospital, has an IT infrastructure supporting business applications and patient-critical systems as well as massive amounts of data storage that is growing exponentially.

“We started using virtualization to address power and space issues in our main datacenter. We quickly adopted VMWare ESX as our standard platform for new projects and consolidated existing servers,” Tom Gibaud, an IT manager at RGH, said in an email. “It allowed us to continue business as usual and we experienced no delay in completing projects on time. Today we are way below our power threshold and gained about 50% of our floor space even after we doubled the amount of Windows Servers.”

In VMware’s statement, VMware Infrastructure has improved application performance and availability, and strengthened the hospital’s disaster-recovery capabilities. “Before going virtual, our datacenter power supply was maxed out. We couldn’t plug in a toaster. Now, with less hardware, we have capacity to handle whatever comes our way,” Gibaud said.

The hospital now runs 50 virtual machine hosts running 400 Guests with a mix of large and small workloads including terminal services, Gibaud said. In all, RGH has virtualized about 95% of its Windows-based applications, including Exchange, SQL Server, the ClinicalCare portal that physicians and nurses use to access electronic medical records, and RGH’s billing system.

In the initial phase of the virtualization deployment, Gibaud said the hospital used IBM Bladecenter servers (HS20, HS21, LS20). “This allowed us to condense many servers is a small amount of space. With VMware and IBM Bladecenters we were able to consolidate over a 150 Servers into one rack,” he said. “Today we use IBM x3850 and HP DL580 G5 to handle larger server workloads.”

In addtion, the hospital is running 200 Windows XP desktops using VMware’s Virtual Desktop Infrastructure on just two IBM x3850’s.

Recovering servers, virtual machines after power failure

An event like a complete data center power failure is something you never want to experience. Having recently gone through one I thought I would share some lessons learned from it.

This particular data center had a full UPS (uninterruptible power supply) system and backup diesel generator when a routine battery maintenance performed on the UPS shorted some circuits causing power loss to the entire data center. This event made me realize that a little preparedness can go along way in getting servers and virtual machines (VMs) back online after a power failure.

First and foremost, the DNS (Domain Name System) is probably the most important service in your data center. Most servers and workstations use DNS names instead of IP addresses to communicate with each other. Without DNS, servers can’t get to anything by hostname and will effectively be isolated from each other. Most administrators are used to using DNS names, so when DNS is not available they usually do not know the IP addresses of the server and subsequently can’t connect to them. So it is a good idea to have a hard copy of all your servers and their IP addresses somewhere in your data center for you to reference when DNS is not available.

Virtual servers can be even more problematic. If you have all your DNS servers virtualized which cannot be started because of network or shared storage issues, you can run into problems starting other servers and services that rely on DNS. Consider having at least one physical DNS server or having one or two DNS servers running on local storage instead of shared storage.

Another helpful insight: Make sure you know command line procedures for administration on your host servers. You may not be able to connect to your host via a graphic user interface (GUI) until certain systems are up so the command line can be your only way to check the host server health and perform VM operations. Again, it helps to have paper documentation of the host command line utilities and their syntaxes.

Finally you want to make sure you start your servers back up in the proper order due to dependencies that certain servers and applications have. Obviously, with the network unavailable, not much is going to function properly. The storage-area network (SAN) is also critical for your host servers that utilize shared storage for VMs. Windows servers also take a very long time to boot if a DNS server and domain controller are not available when they are starting.

Below is a general order for restarting your servers and applications.

1. Network
2. SAN
3. DNS servers
4. DHCP servers
5. Database servers
6. Application/Web servers

The boy scout motto ‘be prepared’ holds true. A little preparation and planning can go along way to ensuring a smoother recovery.

Six reasons Hyper-V will surpass VMware within five years

Clabby Analytics analyst Joe Clabby is 100% convinced that Microsoft’s Hyper-V will take over VMware in market share over the next three to five years, and makes some strong points for this in his recent report, Six Reasons Why Microsoft’s Hyper-V will Overtake VMware to Become the Major Player in the x86 Server Virtualization Marketplace.

The report came out prior to the shake-up at VMware on July 8, when the company announced that its Board of Directors replaced VMware co-founder and CEO Diane Greene was being replaced, and then lowered its revenue forecast.

VMware had the vision to see the value of virtualization and took the technology to the top unchallenged due to strategy, innovation and sales execution, but that ride is about to come to an end, Clabby said.

“With the introduction of Hyper-V by Microsoft, VMware is about to experience some very serious competition from a vendor with deep pockets, with a massive worldwide marketing and sales organization, with major market penetration across Fortune 500 and small and medium business markets, and with extensive and complementary infrastructure and management product depth,” Clabby reported.

Among the reasons Clabby believes Microsoft will crush VMware are that Microsoft already has an expansive installed base, a mammoth network of direct sales and indirect business partners, and is offering lower prices alternatives to VMware’s hypervisor and related infrastructure/management software products.

Unfortunately, I have to agree. History tends to repeat itself, and this has been Microsoft’s strategy for a very long time: see a great technology, copy it, and outprice the rest of the market.

Vanity Fair’s July issue had a great article that illustrates this, called “How the Web was Won” that looks at the eveolution of the Internet over the past 50 years, including details of how Microsoft took over Netscape Navigator by developing Internet Explorer.

The computer programmer known for founding Netscape Communications, Lou Montulli, told Vanity Fair, “From a scientific point of view none of us really respected Microsoft. There was definitely a sense of: They’ve put out of business three or four major companies, and they did it simply by copying what they did and outpricing or outmaneuvering them in the market. This is a general feeling of computer scientists everywhere, that Microsoft doesn’t tend to innovate as much and really just enters the market late, takes it over, and then stays at the top.”

Pricing aside, Microsoft already has a massive installed base.

“It will leverage this installed base, and price its products to out-function/undercut VMware’s pricing,” Clabby wrote. “The computing industry saw this same situation arise when Citrix built a leadership base for its terminal server products — only to have Microsoft enter the market and claim significant marketshare after Citrix pioneered the terminal server marke umbrella. Almost the exact same situation is about to happen again — this time between VMware and Microsoft.”

Microsoft also has a packaging advantage with its Hyper-V hypervisor, as it can be delivered with every single version of 64-bit Windows Server 2008, and installing Hyper-V is a cake walk, according to Clabby.

“A box simply needs to be checked during installation and Hyper-V becomes active. By not requiring IT buyers to find/acquire/download additional virtualization software, the job of deploying and testing virtualization within a Windows Server 2008 is greatly simplified. VMware cannot counter this packaging advantage,” Clabby wrote.

The most damning problem for VMware, according to Clabby, is product depth.

Though VMware has the advantage of technologies like VMotion, to move live VMs, and all of the handy add-on management and infrastructure software integrated into VMware, Clabby said Microsoft’s management and infrastructure is far deeper.

Microsoft’s Systems Center product portfolio inlcludes systems management tools like Configuration Manager; Operations Manager; Data Protection Manager; Virtual Machine Manager; System Center Essentials; Capacity Planner, and the list goes on, ad nauseum.

Besides all of those points, Microsoft is a $51 billion dollar software company and VMware’s revenue is just over $1 billion.

In short, given its deep pockets, large installed base and virtualization strategy, it is safe to say Microsoft will, once again, be laughing all the way to the bank.

Choosing your next virtualization project

For organizations with an established server virtualization environment, future virtualization projects are looming on the horizon. Whether it is desktop or application virtualization, much deliberating will undoubtly be given to the best product for the new virtualization endeavor — as it should.

The next wave of virtualization projects should always be best of breed for the requirements and functionality you require for your particular environment. For example, say you’re an organization with a successful VMware-based server virtualization environment using VirtualCenter and ESX 3. Does this mean that VMware Virtual Desktop Infrastructure (VDI) is the default selection for a virtualized desktop project? Don’t be fooled into thinking that a single-vendor environment is going to translate into an efficient one.

Identify the best solution, even if you can’t afford it. That also includes your host environment hardware for the next virtualization project. Your next virtualization project may require a decision between blades versus general purpose servers for virtual hosts. Taking the time and effort to identify the best solution after making full comparisons for of potential environments will also prepares you for any unforeseen element in post-implementation inquiry.

Make no mistake, there are plenty of advantages to going with what’s familiar: Price discounts, vendor relationships and non-disclosure access are all strong reasons to select the same vendor, but only after due diligence in your decision process should you make another commitment.

Virtualization of Citrix Presentation Server in VMware calculations

In following with Joe Foran’s recent blog about virtualizing Citrix Presentation Server (PS) systems, I too have had success with this practice. I took the approach that, for certain PS configurations, there can be great virtualization candidates depending on how you use Citrix. A web interface for PS is a great candidate for a virtual system if it is on its own server, but additional criteria determine what can be configured for a virtualized Citrix environment.

Based on my experience, the deciding factor for virtualizing PS systems is how many sessions will be concurrent for your published applications. For published applications that are rarely used or will not have very many sessions, this is a good starting point for virtualized PS systems. An example would be a line of business published applications that would not expect more than four concurrent users. A few of these types of applications on a virtual machine in ESX can work very well.

The biggest question becomes virtual machine provisioning from the memory and processor standpoint. If you have a baseline of your current Citrix usage, that is a good starting point for estimating the concurrent session usage. Take the following observations of a Citrix environment:

• Each PS session takes 16 MB of RAM
• Each published application within that environment requires 11 MB of RAM
• There are 4 published applications on a server, that have not exceeded 5 concurrent sessions

Just under 3.5 GB of RAM is required to meet the same environment requirements from the Citrix session perspective. By adding the base server and Citrix PS memory requirements to this calculated amount, you have identified the provisioning requirements of the Citrix server for the virtual role. From the processor standpoint, I generally provision the frequency limit at the rate of the physical system processor.

The good news is that Citrix is licensed by client connection and not the number of servers. Therefore, distributing virtualized Citrix servers in a VMware environment is well poised to meet performance and availability requirements.

Technosium 2008: Virtualization takeaways for business continuity

I am blogging from the inaugural Technosium Global Conference and Expo at the Santa Clara Convention Center. I’ll be signing on intermittently to provide you with everything I consider beneficial to the virtualization space.

First off is storage business continuity. I had an opportunity to attend a breakout hosted by Eric Herzog, the vice president of operations for Asempra Technologies. While business continuity is a topic we all are familiar with, attendees had the chance to look at the building blocks of a successful strategy for continuity, and how it applies to storage for virtualized systems. What I took away from this breakout was that there needs to be clearly defined goals that the business requirements define the following within an organization’s service level agreement (SLA):

• Ability to measure availability and uptime
• Data loss tolerance for your business needs (financial and operational)
• Disaster recovery time frames made specific to your environment
• Solutions that reduce costs with combination of technologies with reduced complexity
• Ensuring that the SLA leverages the existing infrastructure (hardware, software, networks) as much as possible
• Ensuring that there will be usable data on first recovery

Traditional approaches to data continuity to virtualization systems tend to respond with multiple technologies, various products, limited manageability and control, increased costs and expense, and a cumbersome process that limits successful execution of the SLA. While each storage business continuity strategy has positives and negatives, the right solution will depend on your virtualization availability requirements. Among the newer storage technologies are drive snapshots, data deduplication, and remote replication. Some solutions address actual virtual machines, where some address the shared storage systems that host virtual environments. Remote replication provides the fastest recovery time for a virtualized storage system, but also at the highest expense.

In summary, the business continuity strategy for virtualized systems needs to resolve primarily around the technology behind the storage systems in use. The other challenge to virtualization management is to define the goals of virtualization continuity via an SLA.

SearchServerVirtualization.com Products of the Year - Not without their share of snubs

Fortunately for me, my job never requires me to determine vendor awards. However, Alex Barrett and the SearchServerVirtualization.com staff aren’t so lucky. While it’s great to have the power to name Products of the Year, it also means that you’re stuck hearing complaints from everyone that wasn’t named. In case you missed it, Alex recently published the SearchServerVirtualization 2007 Products of the Year.

I think that Alex and the editorial staff did a great job with selecting products, but thought that I would take a moment to highlight some vendors with excellent products that did not make the list. After all, it’s just as much fun to debate the vendors that were not recognized as it is for those who were.

VMware

Yes, VMware’s on the list, but at the same time they’re not on the list. If you didn’t notice, VMware ESX Server 3.5 is nowhere to be found in the article. The SearchServerVirtualization.com editors informed me that ESX 3.5 missed the cutoff date for award consideration (November 30th), and therefore wasn’t eligible. Editors do need time to work with a released product in order to make a fair judgment, so I understand the reasoning for the cutoff. Still, ESX 3.5 was a significant release from VMware, with features such as Storage VMotion adding significant value to VMware deployments.

Novell

Novell quietly had a great 2007, from a virtualization product perspective. Novell was right behind Citrix/XenSource in achieving Microsoft support for their Xen-based virtualization platform, and was pushing the innovation envelope throughout the year. Novell was the very first virtualization vendor to demonstrate N_Port ID virtualization (NPIV) on their Xen platform. Novell was even showing their work with open virtual machine format (OVF) last September at their booth at VMWorld. When you factor in Novell’s work with their heterogeneous virtualization platform management tool, ZENworks Virtual Machine Manager, you’re left with a pretty nice virtualization package. The vendors mentioned in the virtualization platform category (VMware, Citrix/XenSource, SWsoft) are all worthy of recognition, and I think it’s equally fair to recognize Novell’s work in 2007 as well. Perhaps Novell’s heavy lifting in 2007 will result in recognition in 2008; however, it’s safe to say that Novell is going to have some stiff competition from VMware, Citrix/XenSource, Microsoft, Sun, Parallels, and Virtual Iron.

Symantec

I thinks it’s hard to leave Symantec Veritas NetBackup 6.5 out of the discussion. In fact, amongst backup products, I’d list them as first, right alongside CommVault. Symantec was the first major backup vendor to announce support for Citrix XenServer backup, while all other backup products officially supported one virtualization platform - VMware ESX Server. The NetBackup team was also very innovative with VMware Consolidated Backup (VCB), as NetBackup 6.5 includes the capability to perform file level recoveries of VCB image level backups. Typically, a backup product performs two VCB backup jobs - an image level backup for DR purposes, and a file level backup for day-to-day recovery tasks. NetBackup 6.5 provides the ability to do this in a single pass, which I found to be pretty innovative. Factor in Data-deduplication (extremely valuable considering the high degree of file redundancy on VM host systems), also available in NetBackup 6.5, and it’s hard to see how NetBackup could be ignored.

SteelEye

SteelEye is another vendor in the data protection category that I’m surprised did not make the list. VMware HA by itself will not detect an application failure and initiate a failover job as a result, as it’s primarily designed to monitor and react to hardware failures and some failures within the guest OS. SteelEye LifeKeeper, on the other hand, provides automated VM failover in response to application and service failures (in addition to guest OS and physical server failures). Many failures are software-specific, and products that can automate VM failover or restarts in response to software failures go far to improve the availability of VMs in production.I’m limiting my comments only to the award categories, hence I’m only listing some of the products I’ve worked with in 2007 that fit into one of the SSV categories. I hope that for the 2008 awards, we’ll see a higher number of award categories, so all products in the virtualization ecosystem are represented.

Do you agree with editors’ choice of winners? Which deserving vendors do you feel were left off the list? I’d love to hear your thoughts.

Virtualization and high availability: User ponders products, path

Systems admin Michael Gildersleeve wishes for 100% uptime and wonders whether virtualization will bring him closer or further away from that goal. It seems to him that virtualization options only cover one server at a time. “What if I need to do an OS update or patch, or what if some critical hardware fails?” he asks. In that case, he feels a bit more comfortable with a cluster than with virtualization.

Gildersleeve is evaluating high-availability options for virtual machines. VMware’s High Availability (VMware HA) is on his list, but he’s not sure whether that product will work well with his legacy software. He’s also not sure whether HA is as mature and robust as other products on the market.

I’m answering his call for more information. I hope that you will too, either by commenting on this post or emailing me at  jstafford at techtarget.com.

Gildersleeve works for a company that has a Progress database running on a Unix server. Hundreds of Windows clients and Web applications are attached to that database and server through Progress Brokers via service file ports. “I need to provide 365 by 24 by 7 uptime,” Gildersleeve said. “With our new Web business, East and West Coast facilities, and vendors managing our stock and replenishment, we need to be available all of the time.”

He wants to run his database across at least two servers, in a setup like an Oracle Real Application Cluster. He continued:

This would allow me to upgrade the OS, reboot a server or take a server down for maintenance without affecting the database or the users. So far I have only found solutions that will give me a two- to five-minute downtime between switching from one server to another.

Yes, Gildersleeve has looked a little at server virtualization. He’s evaluating server virtualization options and VMware HA to see whether he can reduce the downtime to nil.

What I have seen so far is that if I upgrade my Progress app to v10 (Progress OpenEdge), and then move to two Integrity servers running High Availability, that if one server fails or if we need to do maintenance on a server, we can manually switch to the second server; but the problem with this is that my users will feel the switch because I will need to bring one server down. They will need to log out and in again to the app, or whatever needs to be done to bring the ready server into production mode.

Gildersleeve is willing to evaluate Sun Microsystems options, if they are truly viable for running Progress. Microsoft operating systems are out of the question.

In his evaluations, Gildersleeve has come up with a lot of questions, and he’s looking for advice from HA experts. Can you provide some advice and share your experiences by commenting on this post or emailing me a  jstafford at techtarget.com?