Posted by: Eric Siebert
Eric Siebert, ESX, Lotus Domino, VMware
Although there are several tech notes that suggest otherwise, most Lotus Domino workloads can be successfully virtualized. I thought I’d review some of the tech notes as they seem to be discouraging people from virtualizing Domino servers. Both of the notes I’ll review blame the virtualization layer as the cause of poor performance, but as Domino servers are resource intensive, the issue could very well have been improper architecture or the versions of Domino and vSphere. If you do virtualize Domino, make sure that the underlying architecture is sufficient and make sure to have the latest versions of Domino and vSphere, as the latest versions provide I/O and performance enhancements.
The first tech note, High CPU usage by router and poor performance of Domino under VMware, is about two years old. Here’s a summary of the problem, cause and solution.
Your Lotus Domino server uses considerably more CPU under VMware ESX 2.5 and 3.0 than running directly on the same hardware. With a simple mail workload sending mail to just a few users, CPU spikes to almost 100% for the duration of the test both under Windows and Linux guest virtual machines (VMs).
This issue has been investigated and it was determined it is independent from Domino.
Domino depends on the OS and hardware to perform disk I/O operations. If these operations consume excessive CPU and leave virtually no CPU cycles for Domino to perform any additional computation, the performance of the whole server will suffer and response time for users can increase to unacceptable levels.
In the same way as we see an important decrease of CPU when performing a file copy directly on the hardware vs under VMware ESX, we also verified that the same mail workload performed for Domino showed lower CPU usage and better overall performance and response time when Domino was running directly on the hardware.
As this issue is occurring independently from Domino, it is suggested to evaluate the expected load for the server before migrating it under VMware.
The second tech note, Poor performance of Domino HTTP server on VMware ESX 3.0 and 3.5, is only a few months old.
While VMware ESX is a supported hypervisor for Domino and Domino-based applications, several customers have reported poor performance of Lotus Domino HTTP on Windows 2003 on VMware ESX. The symptoms are flat CPU utilization (regardless of the number of vCPU assigned to the VM, 1, 2 or 4 vCPUs presents identical performance), low memory utilization and very high response time to Web users requests with moderate load (more than a few users hitting the server).
The cause of this issue is currently unknown. IBM and VMware are working to identify the source of this problem. Customers that open a service request with VMware can reference VMware problem report 318726.
IBM Lotus Support has confirmed the problem and verified it is not introduced by a Domino server misconfiguration. In fact, the same settings allow the Domino server to serve HTTP requests in a timely fashion and support heavy load when running on a physical server, while response time increases considerably when the same Domino server runs in a Virtual Machine on VMware ESX 3.0 and 3.5.
For customers running Domino HTTP, it is recommended to either keep the Domino server on a physical server or, if load is known to be low, verify the performance of the application in a VM prior to moving into production.
As mentioned earlier, both tech notes seem to discourage customers from virtualizing Domino servers in certain situations and blame the virtualization layer as the cause of poor performance problems. Domino servers are often times very resource intensive consumers, so when virtualizing them it is critical to properly architect the virtual hosts. Another tech note entitled How to size Domino and Sametime systems for full production loads on VMware ESX Server sums this up nicely:
Virtualization is not merely saving hardware by placing multiple services on a single physical machine. Virtualization requires a carefully designed environment, a clear understanding of how system resources are utilized, sufficient bandwidth and a contingency plan such that future upgrades and tuning are made possible for future changes. These considerations are particularly important for mission critical and resource intensive applications like Lotus Domino-based servers.
In general you should make sure that your VMware ESX Server, SAN, HBA, Fiber Switch and network cards can support the combined load of all virtual machines, when external factors cause a burst of user load onto the system.
Virtualized servers require the same or a higher level of performance than that normally available in a physical server environment. VMware ESX does not reinvent the laws of physics: CPUs and GHz, available RAM, network speed, disk subsystem speed and latency do not disappear. VMware ESX server will not provide an adequate environment for running Lotus products in a production-grade workload if sufficient resources are not allocated to each of the virtual machines.
In addition to having the proper architecture, I recommend that you use the latest versions of both Domino and vSphere as they both provide performance enhancements and I/O optimizations that are beneficial to Domino hosts. Additionally, using newer server hardware that has virtualization-specific CPU technology like Intel’s Nehalem and AMD’s Shanghai processors will help reduce the CPU overhead of the virtualization layer which will result in applications that run faster. By following these recommendations you should be able to successfully virtualize almost any Domino workload.
For more on virtualizing Domino servers, check out Benefits of virtualizing Lotus Domino servers and Ensure maximum performance from virtualized Lotus Domino servers.