Security zones within VMware ESX are either based on networks employed (Management, VMotion, Storage, DMZ, Production, etc.) or it can be based on functionality such as hypervisor, management tool, backup systems, or virtual machine.

Each of these zones reflects a configuration for the VMware ESX host. It also reflects what is being run within the virtual machines as well as what network upon which the VMs and hosts resides.

Network security zones are relatively easy to understand. Each network would be isolated from one another except through well defined gateways and firewalls. The confusion among some security administrators is that they believe since the VMware ESX or ESXi host has multiple adapters that it would therefore act as an undefined gateway between the disparate networks.

This is simply not the case. Unlike other VMware products where networking is done using bridging technology, the virtual switch within the ESX host is similar to any other layer 2 physical switch. The physical NICs within the VMware ESX or ESXi host act as uplink ports between the physical switch and the virtual switch. Whether a virtual NIC used by virtual machines (VMs), or a vmkernel NIC used by the hypervisor, each virtual device connects to a portgroup within a virtual switch then out the physical NICs to the physical switches unless the communication is VM to VM on the same portgroup.

Since there is no way to layer virtual switches or portgroups, the only way to speak across them is by the use of well-defined gateways or firewalls. VMs can act as firewalls and gateways but only if they have more than one virtual NIC. So the rule of no multi-homed VMs is the best place to start unless it is a well defined virtual firewall or gateway.

The other definition of security zones is based on the roles used within the virtual infrastructure as well. There are three well-defined roles within the virtual infrastructure: hypervisor, virtual machine, and management tool.

The hypervisor controls everything so access to this must be limited to only the service console or management appliance, data stores used as VM storage, or VMware VMotion interfaces. Two of these are through vmkernel NIC used by the hypervisor as described previously: data stores for VM Storage and VMware VMotion Interfaces. Anything that accesses the hypervisor must be protected from those items that should not be able to access the hypervisor.

A virtualization management tool is another role within the system, these tools interact with the service console or management appliance for the virtualization host which will indirectly interact with the hypervisor. This indirect interaction is the key to handling this role. Virtualization management tools such as VMware VirtualCenter, HP SIM VMM plugin, VMware Lab Manager, etc. should be firewalled from other networks as they have indirect access to the hypervisor.

Backup systems act as another role within a virtual environment. These systems can interact directly with the storage devices attached to virtualization hosts through VMware VCB, or through the service console or management appliance just like virtualization management tools. This can also lead to indirect interaction with the hypervisor. Due to the direct and indirect access these systems should be firewalled as well from other networks.

The last role is that of a standard virtual machine. These virtual machines could live on any network but they should not be able to directly access the service console or management appliance of thse virtualization host in addition they should not be able to directly attach to the storage device ued by the virtualization hosts. Virtual machines are considered to be hostile to the virtualization host and should be treated as such.

Security zones depend on the security roles each component of your virtual environment directly or indirectly interacts. These few tips will aid in designing a secure environment.

VMware frequently updates the Guest Operating System Installation Guide (GOSIG), an online book that gives specific information for VMware ESX Server, VMware GSX Server, VMware Server, VMware ACE, VMware Workstation and VMware Fusion guest operating systems. This guide gives very specific configuration matrices for the VMware product and the guest OSes that can be run within the product. Further, there are very handy known issues sections for each guest OS.

Supported VMware Server 2.0 configurations

A specific example that I have found this guide helpful in regards to VMware Server 2.0. According to the GOSIG resource, Windows Server 2003 is only a supported configuration on VMware Server 2.0 with Service Pack 1, Service Pack 2 or the R2 features added to Windows Server 2003. VMware ESX, however, has a fully supported configuration for Windows Server 2003, including the base release without any Service Packs or the R2 features, in all versions from 2.0 through 3.5 Update 3.

Supported Windows Server 2008 configurations

Some configurations are more obvious, such as running Windows Server 2008 as a guest operating system on a hypervisor that predates the release of the guest OS. In the GOSIG guide, Windows Server 2008 64-bit guest OSes are supported only on more current products. Some platforms, such as VMware GSX server and VMware ESX Server 2.x and 3.0x are not a supported configuration for this guest OS. Even with all of this information, and the officially supported configurations – you may find that certain situations are successful even though they are not listed in the GOSIG documentation. A better practice would be to match the hypervisors with the supported configurations in regards to the guest OSes, and this may mean standing up different versions of VMware products to cover the full range of OSes that are required in your environment.

Because it is based on a light-weight, kernel optimized for virtualization, VMware ESX Server is less susceptible to viruses and other problems that affect general-purpose operating systems. However, ESX Server is not impervious to attack, and you should take proper measures to harden it, as well as the VMware VirtualCenter management server, against malicious activity or unintended damage.

Because ESX Server runs a customized, locked-down version of Linux, there is much less likelihood of security exploits than in a standard Linux distribution. If you follow the best practice of isolating the network for the service console, there is no reason to run any antivirus or other such security agents, and their use is not recommended. However, if your environment requires that such agents be used, then use a version designed to run on Red Hat Enterprise Linux 3, Update 6.

The key here is following proper security best practices. This means taking steps like keeping your host patched in a timely manner, isolating Service Console network traffic from the other traffic on the network including VM traffic and following the additional hardening best practices listed in VMware’s Security Hardening Guide and CISecurity’s ESX Security Benchmark.

There are some good reasons for not installing AV software on the Service Console. The first reason is that it can impact on the performance of the ESX host and subsequently all of the virtual machines that reside on it because of the extra CPU, memory and disk resources that antivirus software typically uses. Antivirus software can be particularly resource intensive and can draw resources away from the virtual machines and potentially have a very negative impact on them. Another reason is that there is the possibility that the software may cause other issues on the Service Console as is true with any additional third party software that is installed on the Service Console.

Despite the reasons for not installing it some enterprises mandate that AV software be installed on all systems regardless of how secure they are. You might try and exempt your ESX hosts from this by explaining the following points:

1. The design of ESX does not allow for a VM that is infected by a virus to spread it to the ESX host through the VMKernel. It can only spread through traditional means which is typically over the network by leveraging open ports and OS vulnerabilities. If you isolate your Service Console network then it is protected from any VMs that may be infected.

2. Most viruses are written to exploit Windows systems, there are very few viruses that are written specifically for Linux systems.

3. ESX has a built-in firewall that protects the Service Console and blocks all ports except those few required by ESX and VirtualCenter.

4. ESX has a very good historical track record, I’ve never heard of a virus infecting the ESX Service Console. This doesn’t mean it can’t happen just that the chances are extremely low.

5. Explain the negative performance impact the AV software will have on an ESX host which can affect all of the VMs on the host.

If you are forced to install antivirus software on the Service Console because of security requirements in your environment then you should make sure that you run a version that is designed for the version of Linux that the Service Console uses. Additionally try and configure it as minimal as possible to minimize the impact on the server’s resources. It’s best to exclude all your VMFS volumes from scanning or at a minimum exclude specific virtual machine files like vmdk and vswp files that are frequently written to. Make sure and keep a close eye on the performance of the host to see how much impact the antivirus software is having on it. Also if you have to run on-demand scans make sure they are performed in off-peak hours when activity on the ESX host is low.

Let us take a quick look at the daily operations you may perform within your virtual infrastructure.

• Create/Move/Modify virtual machines (VMs)
• Backup/replicate VMs
• View performance and other data about hosts and VMs
• Install/Reinstall/Update hosts
• Add or remove administrative users from VMs and hosts
• Monitor your VMs and hosts
• etc.

The list is quite endless. Each one of these actions will affect security or be affected by security.

Ease of use, ease of administration is sometimes paramount. This is the age-old debate of security over usability. Do not knock holes in your firewalls, instead, use existing secure protocols to improve the experience. One I use is OpenVPN to access my administrative network workstation using pre-shared keys for encryption. The extra step of starting up the VPN is trivial with the service provided, this gives me secure access to the management network over which the tools can be run.

Affected by Security

There will be at least one more step involved in use of daily tools then you would normally use. Mainly some way to ensure your access to the management tools is secured from sniffing by your non-administrative network. Or the use of different procedures to ensure you have proper auditing in place.

One tool I use is a simple expect script called expectsudo which will allow me to run remote commands on my VMware ESX hosts while retaining logging of all access. To use this tool you must first install the expect RPM onto your host. Then setup sudo to allow access to the appropriate commands.

#!/usr/bin/expect --

set pass [lindex $argv 0]

set timeout 5

spawn /usr/bin/sudo [lrange $argv 1 end]

expect "Password: {send "$passr"; exp_continue

sleep 1

In the above script the first argument will be the password to use. This script can not be run remotely using any form of SSH. For example:
ssh user@hostname expectsudo password esxcfg-vswitch -l
The only drawback to this script is the need to have the password available, instead you can setup pre-shared keys for SSH and setup Sudo to allow the  user to issue certain commands without requiring a password. This form of single sign on relies on the security of the management workstation in use.

Security of your VMware Infrastructure relies on the security of all the management workstations and servers in use. Not just on the hardening of the VMware ESX host.

P2V works by imaging the physical host within the DMZ and transferring that image to the administrative/management network attached to the service console (management appliance) of the VMware ESX(i) host. This in essence crosses security zones and could connect the hostile DMZ to the ‘in need of protection’ virtualization management network. Access to this network from the DMZ could be disastrous.

One solution is to perform the P2V migration in stages.

1. Create the DMZ virtual network within your virtual infrastructure.
2. Get your security team to bless a laptop/workstation for work within the DMZ. Ensure this laptop/workstation has enough removable storage to contain the resultant VM or VMs of the physical servers you wish to convert.Use your  P2V tool to convert the VM and store it on the removable media.
3. Disconnect the removable media and bring it to your secure administrative network.
4. Connect the removable media to a workstation within the administrative network. Ensure this connection is read-only for the moment if possible.
5. Virus Scan the removable media, but note a VMDK can give false positives; you are really looking for anything that may be hidden from view.
6. Use VMware Converter to import the VM or VMs into the virtual infrastructure ensuring they are connected to the proper virtual network.
7. Power on the VM with the network disconnected and fix any issues that are caused by the P2V migration, such as the need to remove hardware agents, and fix anything that needs to be fixed.
8. Reboot the VM with the network connected

The P2V migration is now complete and isolated from the network. The key to this is to only power on the VM once you are within a safe environment and to check for viruses and worms that may live within your DMZ.