VMware ESX 3.5

Why you shouldn’t restore VMware ESX from a backup

A common question that arises on the VMware Communities Forum is how to backup VMware ESX so that you can restore the backup if there is a problem, the theory being that this would be faster than reinstalling the server.

As stated within the VMware KB article 1000761 it is possible to restore ESX to identical hardware; however, you need to reinstall ESX first and restore the data you backed up while making changes to how the system boots, else the Universally Unique Identifier (UUID) written by the installation will not work anymore as you have overwritten the data from your backup.

This method will restore everything effectively to identical hardware, however if you want to use new hardware, perhaps with different PCI devices, then the restoration would fail to properly configure the new devices. It may even fail to properly configure NICs if there are any IRQ differences between the supposed identical hardware.

So in these cases you would have to at least verify the configuration and fix anything that was broken. This could lead to a set of unknowns from a security perspective. You are after all trusting the backup was restored properly and if it was not, then you could end up with security issues. So the verification step would have to be extremely well documented.

It is far easier to reinstall VMware ESX to the hardware and to use a either a installation document,  kickstart, or other type of script to configure all the devices for you using either the Remote CLI or the VMware ESX CLI.

When restoring VMware ESX or VMware ESXi the best tool to have will be very good installation documentation that is easy to follow and has graphics and text for every step of the configuration.  These documents could be reviewed for security concerns, and used to derive the scripts that could do the work for you.

Top VMware security links

Keeping track of Security issues associated with virtualization requires a serious investment in time. To aid in that I have put together the top virtualization security links that will continue to grow over time.

Top Virtualization Security Links

The following links are just a sample of what is at the aforementioned site and should be read in order for those interested in securing your VMware Virtual Infrastructure and unfamiliar with VMware ESX, at the same time these are great references for the experienced administrator.

• VMware Virtual Networking Concepts
• VMware ESX Server 3: 802.1Q VLAN Solutions
• Security Design of the VMware 3 Architecture
• DMZ Virtualization with VMware Infrastructure
• VMware Infrastructure 3 Hardening
• CISecurity VMware ESX Security Benchmark followed by the CISecurity Linux Benchmark
• DISA STIG (ESX STIG depends on the UNIX STIG)
  • VMware Communities thread on ESX_SRRSecure - Script to allow ESX to pass a DISA Security Readiness Review.
  • UNIX STIG
  • ESX STIG
• Proven Practice: VI3 Security Risk Assessment - Xtravirt.com
• Remote Authentication - Full/Partial AD Integration, Secure LDAP, NIS, …

It is recommended to read as each guide or benchmark as each covers things from a slightly different but useful perspective.

VMware VI:OPs has also launched the Top 100 Virtualization Security Questions. This can also be referenced from the Top Virtualization Security Links sites as well as other blogs of interest.

CompareMyVM website to offer VMware ESX virtual machine configuration comparisons

Virtual machine provisioning sometimes requires administrator guesswork which doesn’t always yield the best results. To help address this need, Portsmouth, N.H.-based VKernel has launched a beta-stage community website, CompareMyVM.com, which will post configuration of virtual machines to help virtualization administrators.

At this point on the site, there are hardly enough virtual machines to make a quality sample of how the world is using ESX-based VMs. So, I encourage you to upload to the site, and as it matures the quality of the site will improve with more data. Uploading a VM or appliance’s configuration is easy, simply go to the CompareMyVM website and select to Snapshot My VM via a Java applet that will connect to VirtualCenter or the ESX host, or you can click the Contribute My VM button to enter the configuration manually. Once a virtual machine is loaded into the site, it is shown in the list among the other contributed VMs. An example of a VM I uploaded is shown below:

Uploaded VMs can be given information about what applications they run, such as a Windows domain controller, Apache 2.2 web engine or an Exchange mail server. At first glance, you may think that this is simply a way that virtualization admins can put their data on the CompareMyVM site and have it sold as data that says how administrators configure their VMs.

Not so according to Christian Simko, director of marketing and communications from VKernel who states that “The idea is to keep it open and never charge for it… We may eventually publish a best practices guide with data derived from CompareMyVM, but that will also be a freebie.” This is good news, as the site develops, there will surely be value to determine how other administrators are provisioning VMs with similar application inventories.

As the site matures, look for more comparative performance data on the contributed configurations as well. To date, there is only a limited number of virtual machines posted from virtualization users. I’ve put some VMs up there and hopefully you will as well. As more users contribute data, this free service will hopefully provide some valuable comparative information on how people use virtualization across different market segments. Most importantly, this information will be vendor-neutral, so different server hardware as well as various software configurations will be represented on the community site.

Antivirus software on the VMware ESX Service Console?

This question is often asked, should I install antivirus (AV) software on the VMware ESX Service Console? If you ask VMware and many seasoned ESX Administrators the answer is usually no. According to VMware’s Security Hardening Guide ESX is less susceptible (not immune) to viruses and if you follow proper security best practices installing AV software on the service console is not recommended:

Because it is based on a light-weight, kernel optimized for virtualization, VMware ESX Server is less susceptible to viruses and other problems that affect general-purpose operating systems. However, ESX Server is not impervious to attack, and you should take proper measures to harden it, as well as the VMware VirtualCenter management server, against malicious activity or unintended damage.

Because ESX Server runs a customized, locked-down version of Linux, there is much less likelihood of security exploits than in a standard Linux distribution. If you follow the best practice of isolating the network for the service console, there is no reason to run any antivirus or other such security agents, and their use is not recommended. However, if your environment requires that such agents be used, then use a version designed to run on Red Hat Enterprise Linux 3, Update 6.

The key here is following proper security best practices. This means taking steps like keeping your host patched in a timely manner, isolating Service Console network traffic from the other traffic on the network including VM traffic and following the additional hardening best practices listed in VMware’s Security Hardening Guide and CISecurity’s ESX Security Benchmark.

There are some good reasons for not installing AV software on the Service Console. The first reason is that it can impact on the performance of the ESX host and subsequently all of the virtual machines that reside on it because of the extra CPU, memory and disk resources that antivirus software typically uses. Antivirus software can be particularly resource intensive and can draw resources away from the virtual machines and potentially have a very negative impact on them. Another reason is that there is the possibility that the software may cause other issues on the Service Console as is true with any additional third party software that is installed on the Service Console.

Despite the reasons for not installing it some enterprises mandate that AV software be installed on all systems regardless of how secure they are. You might try and exempt your ESX hosts from this by explaining the following points:

1. The design of ESX does not allow for a VM that is infected by a virus to spread it to the ESX host through the VMKernel. It can only spread through traditional means which is typically over the network by leveraging open ports and OS vulnerabilities. If you isolate your Service Console network then it is protected from any VMs that may be infected.

2. Most viruses are written to exploit Windows systems, there are very few viruses that are written specifically for Linux systems.

3. ESX has a built-in firewall that protects the Service Console and blocks all ports except those few required by ESX and VirtualCenter.

4. ESX has a very good historical track record, I’ve never heard of a virus infecting the ESX Service Console. This doesn’t mean it can’t happen just that the chances are extremely low.

5. Explain the negative performance impact the AV software will have on an ESX host which can affect all of the VMs on the host.

If you are forced to install antivirus software on the Service Console because of security requirements in your environment then you should make sure that you run a version that is designed for the version of Linux that the Service Console uses. Additionally try and configure it as minimal as possible to minimize the impact on the server’s resources. It’s best to exclude all your VMFS volumes from scanning or at a minimum exclude specific virtual machine files like vmdk and vswp files that are frequently written to. Make sure and keep a close eye on the performance of the host to see how much impact the antivirus software is having on it. Also if you have to run on-demand scans make sure they are performed in off-peak hours when activity on the ESX host is low.

Security outside the box

Virtualization security depends on more than securing the virtualization host and hypervisor. It depends on everything that touches your virtualization host, directly or indirectly, also being secure.

Let us take a quick look at the daily operations you may perform within your virtual infrastructure.

• Create/Move/Modify virtual machines (VMs)
• Backup/replicate VMs
• View performance and other data about hosts and VMs
• Install/Reinstall/Update hosts
• Add or remove administrative users from VMs and hosts
• Monitor your VMs and hosts
• etc.

The list is quite endless. Each one of these actions will affect security or be affected by security.

Ease of use, ease of administration is sometimes paramount. This is the age-old debate of security over usability. Do not knock holes in your firewalls, instead, use existing secure protocols to improve the experience. One I use is OpenVPN to access my administrative network workstation using pre-shared keys for encryption. The extra step of starting up the VPN is trivial with the service provided, this gives me secure access to the management network over which the tools can be run.

Affected by Security

There will be at least one more step involved in use of daily tools then you would normally use. Mainly some way to ensure your access to the management tools is secured from sniffing by your non-administrative network. Or the use of different procedures to ensure you have proper auditing in place.

One tool I use is a simple expect script called expectsudo which will allow me to run remote commands on my VMware ESX hosts while retaining logging of all access. To use this tool you must first install the expect RPM onto your host. Then setup sudo to allow access to the appropriate commands.

#!/usr/bin/expect --

set pass [lindex $argv 0]

set timeout 5

spawn /usr/bin/sudo [lrange $argv 1 end]

expect "Password: {send "$passr"; exp_continue

sleep 1

In the above script the first argument will be the password to use. This script can not be run remotely using any form of SSH. For example:
ssh user@hostname expectsudo password esxcfg-vswitch -l
The only drawback to this script is the need to have the password available, instead you can setup pre-shared keys for SSH and setup Sudo to allow the  user to issue certain commands without requiring a password. This form of single sign on relies on the security of the management workstation in use.

Security of your VMware Infrastructure relies on the security of all the management workstations and servers in use. Not just on the hardening of the VMware ESX host.

Secure method to P2V across security zones

A common VMware Communities question is how to P2V or convert a system from within a demilitarized zone (DMZ) to a virtual machine (VM) running within an ESX host that will be part of the DMZ virtual network.

P2V works by imaging the physical host within the DMZ and transferring that image to the administrative/management network attached to the service console (management appliance) of the VMware ESX(i) host. This in essence crosses security zones and could connect the hostile DMZ to the ‘in need of protection’ virtualization management network. Access to this network from the DMZ could be disastrous.

One solution is to perform the P2V migration in stages.

1. Create the DMZ virtual network within your virtual infrastructure.
2. Get your security team to bless a laptop/workstation for work within the DMZ. Ensure this laptop/workstation has enough removable storage to contain the resultant VM or VMs of the physical servers you wish to convert.Use your  P2V tool to convert the VM and store it on the removable media.
3. Disconnect the removable media and bring it to your secure administrative network.
4. Connect the removable media to a workstation within the administrative network. Ensure this connection is read-only for the moment if possible.
5. Virus Scan the removable media, but note a VMDK can give false positives; you are really looking for anything that may be hidden from view.
6. Use VMware Converter to import the VM or VMs into the virtual infrastructure ensuring they are connected to the proper virtual network.
7. Power on the VM with the network disconnected and fix any issues that are caused by the P2V migration, such as the need to remove hardware agents, and fix anything that needs to be fixed.
8. Reboot the VM with the network connected

The P2V migration is now complete and isolated from the network. The key to this is to only power on the VM once you are within a safe environment and to check for viruses and worms that may live within your DMZ.

VMware ESX 3.5 Update 3 released!

The latest version of ESX, Version 3.5 Update 3 has just been released to complement the release of VirtualCenter 2.5 Update 3 that was released a month ago. The new features and supported devices included in this release are listed below. One of the nicest new features is an experimental feature that enables you to recover VMDK files that were deleted. This feature is a new ESX Service Console command (not supported yet with ESXi) and is designed to recover VMDK files if they are deleted or if a VMFS volume is deleted or corrupted. (Almost like having a recycle bin for ESX!). Until now, if you accidentally deleted a VM and its disk file it was gone and almost impossible to recover.

According to the Compatibility Matrix, ESX 3.5 Update 3 will only work with VirtualCenter 2.5 Update 3 and Update 2. It will not work with versions prior to 2.5 Update 1, so upgrade your VirtualCenter first if necessary. You can read the full release notes for this version here and download it here.

New features and supported devices:

• Increase in vCPU per Core Limit — The limit on vCPUs per core has been raised from 8 (or 11 for VDI workloads) to 20. This change only raises the supported limit but does not include any additional performance optimizations. Raising the limit allows users more flexibility to configure systems based on specific workloads and to get the most advantage from increasingly faster processors. The achievable number of vCPUs per core will depend on the workload and specifics of the hardware. It is expected that most deployments will remain within the previous range of 8-11 vCPUs per core. For more information, see VI3 Performance Best Practices and Benchmarking Guidelines.

• HP BL495c support — This release adds support for the HP Blade Server BL495c with all Virtual Connect and IO Options allowing 1 or 10Gb connection to the network (upstream) and 1Gb connections only to the servers (downstream).

• Newly Supported NICs — This release adds support for the following NICs:

• Broadcom 5716 1Gb, Broadcom 57710 10Gb Adapters, Broadcom 57711 10Gb Adapters at 1Gb speed only
Note: iSCSI/TOE hardware offloads available with these adapters are not supported by VMware with ESX 3.5.

• Newly Supported SATA Controllers— This release adds support for the following SATA controllers:

• Broadcom HT1000 (supported in native SATA mode only with SATA hard drives and Solid State Disk devices)

• Intel ICH-7 (supported in IDE/ATA mode only with SATA CD/DVD drives)
Note: Storing VMFS data stores on drives connected to these controllers is not supported

• Newly Supported Guest Operating Systems — Support for the following Guest Operating Systems have been added by VMware during the ESX 3.5 Update 3 release cycle:

• Solaris 10 U5

• Ubuntu 8.04.1

• RHEL 4.7

• Internal SAS networked storage controllers — This release adds experimental support for Intel Modular Server MFSYS25 SAS Storage Control Modules (SCMs). For known issues with this platforms and workaround see SAS Link and Port Failovers with the Intel Modular Server Running Update 3 and Later Versions of ESX 3.5 and ESXi 3.5 (KB 1007394).

• Interrupt Coalescing (IC) for Qlogic 4Gb FC HBAs — Introduced in this release, the feature reduces CPU utilization (and CPU cost per IO) and improves throughput of IO intensive workloads by generating a single interrupt for a burst of Fibre Channel frames, when received in a short period of time, rather than interrupting the CPU each time a frame is received. The feature is enabled by default.

• Experimental Support for the VMDK Recovery Tool — This release adds support for the VMDK Recovery tool, a script intended to help customers to recover VMFS/vmdk data stores from accidental deletion of VMFS/vmdk data store or physical disk corruption. For more information, see VMDK Recovery Tool (ESX 3.5 Update 3) ( KB 1007243).

• Small Footprint CIM Broker — Updated SFCB to version 1.3.0

• IBM SAN Volume Controller — SVC is now supported with Fixed Multipathing Policy as well as MRU Multipathing Policy.