I’ve written about the vSphere’s new Fault Tolerance (FT) feature several times and wanted to put the information together in one blog, as well as include some new information. We’ve broken this guide into several sections as it’s a bit lengthy, so you can skim the witty titles and decide if a section for you, or if you’d rather keep on truckin’ to the next section. But first, if you’d like to check out my previous posts on FT, they are available here:
I. And VMware said, ‘Let there be Fault Tolerance’
Fault Tolerance was introduced as a new feature in vSphere that provided something that was missing in VMware Infrastructure 3 (VI3), the ability to have continuous availability for a virtual machine in case of a host failure. High Availability (HA) was a feature introduced in VI3 to protect against host failures, but it caused the VM to go down for a short period of time while it was restarted on another host. FT takes that to the next level and guarantees the VM stays operational during a host failure by keeping a secondary copy of it running on another host server. If a host fails, the secondary VM becomes the primary VM and a new secondary is created on another functional host.
The primary VM and secondary VM stay in sync with each other by using a technology called Record/Replay that was first introduced with VMware Workstation. Record/Replay works by recording the computer execution on a VM and saving it as a log file. It can then take that recorded information and replay it on another VM to have a replica copy that is a duplicate of the original VM.
II. Power to the processors
The technology behind the Record/Replay functionality is built in to certain models of Intel and AMD processors. VMware calls it vLockstep. This technology required Intel and AMD to make changes to both the performance counter architecture and virtualization hardware assists (Intel VT and AMD-V) that are inside the physical processors. Because of this, only newer processors support the FT feature. This includes the third-gen AMD Opteron based on the AMD Barcelona, Budapest and Shanghai processor families, and Intel Xeon processors based on the Penryn and Nehalem micro-architectures and their successors. VMware has published a knowledgebase article that provides more details on this.
III. But how does it do that?
FT works by creating a secondary VM on another ESX host that shares the same virtual disk file as the primary VM, and then transferring the CPU and virtual device inputs from the primary VM (record) to the secondary VM (replay) via a FT logging network interface card (NIC) so it is in sync with the primary VM and ready to take over in case of a failure. While both the primary and secondary VMs receive the same inputs, only the primary VM produces output such as disk writes and network transmits. The secondary VM’s output is suppressed by the hypervisor and is not on the network until it becomes a primary VM, so essentially both VMs function as a single VM.
It’s important to note that not everything that happens on the primary VM is copied to the secondary VM. There are certain actions and instructions that are not relevant to the secondary VM, and to record everything would take up a huge amount of disk space and processing power. Instead, only non-deterministic events are recorded, which include inputs to the VM (disk reads, received network traffic, keystrokes, mouse clicks, etc.,) and certain CPU events (RDTSC, interrupts, etc.). Inputs are then fed to the secondary VM at the same execution point so it is in exactly the same state as the primary VM.
The information from the primary VM is copied to the secondary VM using a special logging network that is configured on each host server. This requires a dedicated gigabit NIC for the FT Logging traffic (although not a hard requirement, this is highly recommended). You could use a shared NIC for FT Logging for small or test/dev environments and for testing the feature. The information that is sent over the FT Logging network between the host can be very intensive depending on the operation of the VM.
VMware has a formula that you can use to determine this:
VMware FT logging bandwidth ~= (Avg disk reads (MB/s) x 8 + Avg network input (Mbps)) x 1.2 [20% headroom]
To get the VM statistics needed for this formula you need to use the performance metrics that are supplied in the vSphere client. The 20% headroom is to allow for CPU events that also need to be transmitted and are not included in the formula. Note that disk or network writes are not used by FT as these do not factor in to the state of the virtual machine.
As you can see, disk reads will typically take up the most bandwidth. If you have a VM that does a lot of disk reading you can reduce the amount of disk read traffic across the FT Logging network by using a special VM parameter. By adding a replay.logReadData = checksum parameter to the VMX file of the VM, this will cause the secondary VM to read data directly from the shared disk, instead of having it transmitted over the FT logging network. For more information on this see this knowledgebase article.
IV. Every rose has its thorn
While Fault Tolerance is a useful technology, it does have many requirements and limitations that you should be aware of. Perhaps the biggest is that it currently only supports single vCPU VMs, which is unfortunate as many big enterprise applications that would benefit from FT usually need multiple vCPU’s (vSMP). Don’t let this discourage you from running FT, however, as you may find that some applications will run just fine with one vCPU on some of the newer, faster processors that are available as detailed here. Also, VMware has mentioned that support for vSMP will come in a future release. It’s no easy task trying to keep a single vCPU in lockstep between hosts and VMware developers need more time to develop methods to try and keep multiple vCPUs in lockstep between hosts. Additional requirements for VMs and hosts are as follows:
In addition to these requirements your hosts must also be licensed to use the FT feature, which is only included in the Advanced, Enterprise and Enterprise Plus editions of vSphere.
V. How to use Fault Tolerance in your environment
Now that you know what FT does, you’ll need to decide how you will use it in your environment. Because of high overhead and limitations of FT you will want to use it sparingly. FT could be used in some cases to replace existing Microsoft Cluster Server (MSCS) implementations, but it’s important to note what FT does not do, which is to protect against application failure on a VM. It only protects against a host failure.
If protection for application failure is something you need, then a solution like MSCS would be better for you. FT is only meant to keep a VM running if there is a problem with the underlying host hardware. If protecting against an operating system failure is something you need, than VMware High Availability (HA) is what you want, as it can detect unresponsive VMs and restart them on the same host server.
FT and HA can be used together to provide maximum protection. If both the primary host and secondary host failed at the same time, HA would restart the VM on another operable host and spawn a new secondary VM.
VI. Important notes
One important thing to note: If you experience an OS failure on the primary VM, like a Windows Blue Screen Of Death (BSOD), the secondary VM will also experience the failure as it is an identical copy of the primary. The HA virtual machine monitor will detect this, however, restart the primary VM, and then spawn a new secondary VM.
Another important note: FT does not protect against a storage failure. Since the VMs on both hosts use the same storage and virtual disk file it is a single point of failure. Therefore it’s important to have as much redundancy as possible to prevent this, such as dual storage adapters in your host servers attached to separate switches, known as multi-pathing). If a path to the SAN fails on one host, FT will detect this and switch over to the secondary VM, but this is not a desirable situation. Furthermore if there was a complete SAN failure or problem with the VM’s LUN, the FT feature would not protect against this.
VII. So should you actually use FT? Enter SiteSurvey
Now that you’ve read all this, you might be wondering if you meet the many requirements to use FT in your own environment. VMware provides a utility called SiteSurvey that will look at your infrastructure and see if it is capable of running FT. It is available as either a Windows or Linux download and once you install and run it, you will be prompted to connect to a vCenter Server. Once it connects to the vCenter Server you can choose from your available clusters to generate a SiteSurvery report that shows whether or not your hosts support FT and if the hosts and VMs meet the individual prerequisites to use the feature.
You can also click on links in the report that will give you detailed information about all the prerequisites along with compatible CPU charts. These links go to VMware’s website and display the help document for the SiteSurvey utility, which is full of great information, including some of the following prerequisites for FT.
Below is some sample output from the SiteSurvey utility showing host and VM compatibility with FT and what features and components are compatible or not:
Another method for checking to see if your hosts meet the FT requirements is to use the vCenter Server Profile Compliance tool. To use this method, select your cluster in the left pane of the vSphere Client, then in the right pane select the Profile Compliance tab. Click the Check Compliance Now link and it will begin checking your hosts for compliance including FT as shown below:
VIII. Are we there yet? Turning on Fault Tolerance
Once you meet the requirements, implementing FT is fairly simple. A prerequisite for enabling FT is that your cluster must have HA enabled. You simply select a VM in your cluster, right-click on it, select Fault Tolerance and then select “Turn On Fault Tolerance.”
A secondary VM will then be created on another host. Once it’s complete you will see a new Fault Tolerance section on the Summary tab of the VM that will display information including FT status, secondary VM location (host), CPU and memory in use by the secondary VM, the secondary VM lag time (how far behind the primary it is in seconds) and the bandwidth in use for FT logging.
Once you have enabled FT there are alarms available that you can use to check for specific conditions such as FT state, latency, secondary VM status and more.
VIII. Fault Tolerance tips and tricks
Some additional tips and tidbits that will help you understand and implement FT are listed below.
IX. And there’s more! Additional resources
We’ve provided you with a lot of information on the new FT feature that should help you understand how it works, how to set it up ,and how use it. For even more information on FT you can check out the following resources:
VMware White Papers:
VMware KB Articles:
Here’s a summary of some interesting details from the podcast, but if you haven’t listened to it yet, I recommend that you check out the recording as it provides a lot of valuable technical information.
VMware’s FT is first generation technology and will get better as it matures over time. Future releases of FT may include enhancements such as relaxing the build level requirements, support for vSMP VMs, support for backing up an FT-enabled VM with VMware Consolidated Backup and also support for movement of FT-enabled VMs via DRS.]]>
This utility specifically tests to see if the hosts in your clusters are compatible with the new Fault Tolerance feature. It is available as either a Windows or Linux download and once you install and run it, you will be prompted to connect to a vCenter Server.
Once it connects to the vCenter Server you can choose from your available clusters to generate a SiteSurvery report that shows whether or not your hosts support FT and if the hosts and VMs meet the individual prerequisites to use the feature. You can also click on links in the report that will give you detailed information about all the prerequisites along with compatible CPU charts. These links go to VMware’s website and display the help document for the SiteSurvey utility, which is full of great information including some of the following prerequisites for FT.
Here’s part of the online help guide:
Here’s some of the information from a sample report:
So if you plan on using FT be sure and check out this utility to check out your hosts and also to learn about the many pre-requisites for using this great new feature.]]>
The feature works by creating a secondary VM on another ESX host that shares the same virtual disk file as the primary VM and then transferring the CPU and virtual device inputs from the primary VM (record) to the secondary VM (replay) via a FT logging NIC so it is in sync with the primary and ready to take over in case of a failure. While both the primary and secondary VMs receive the same inputs, only the primary VM produces output such as disk writes and network transmits. The secondary VM’s output is suppressed by the hypervisor and is not on the network until it becomes a primary VM, so essentially both VMs function as a single VM.
FT can be used with any application as the guest operating system and applications running on it are completely unaware of FT. This new feature is only included in the Advanced, Enterprise and Enterprise Plus editions of vSphere. It eliminates the need for VMware customers to use Microsoft’s Cluster Server (MSCS) to provide continuous availability for critical applications. In fact, VMware’s documentation states the following as a use case for FT:
Cases where high availability might be provided through MSCS, but MSCS is too complicated to configure and maintain.
While FT is a very useful feature it does have some limitations and strict usage requirements. On the host side it requires specific, newer processor models from AMD and Intel that support Lockstep technology. You might be wondering what Lockstep technology is. Simply put, Lockstep is a technique used to achieve high reliability in a system by using a second identical processor to monitor and verify the operation of the first processor. Both processors receive the same inputs so the operation state of both processors is identical or operating in “lockstep” and the results are checked for discrepancies. If the operations are not identical and a discrepancy is found, the error is flagged and the system performs additional tests to see if a CPU is failing.
This technology is integrated into certain AMD and Intel CPUs and is what the Fault Tolerance feature relies on to sync the CPU operations of a VM between two hosts so they are in identical states (VMware calls it vLockstep). This includes the AMD Barcelona quad-core processors that were first introduced in September of 2007 and the Intel Harpertown family processors that were first introduced in November of 2007. The vSphere Availability Guide references a KB Article (#1008027) on compatible processors that will presumably be published when vSphere is GA. More information on compatible processor models can be found at Eric Sloof’s NTPRO.NL blog and at Gabrie van Zanten’s blog, Gabe’s Virtual World. Below are the official requirements from VMware’s documentation:
One additional requirement that is not listed is that the CPU clock speeds between the two ESX hosts must be within 400 Mhz of each other. The reason for this is so that one CPU does not lag behind the other so they can keep with each other and stay in sync. You can check to see if the processors in your hosts will support the FT feature by using the CPU Host Info utility that is covered in “VMware vSphere: Got 64-bit hardware?”. You can also read more about this new feature at the following links: