This is not a huge issue as long as your VMware ESX service console is properly protected, but you may want to be concerned if it is not. The vulnerability allows any one who can access the VMware ESX host to write anything to these directories and could cause a disk to fill up or something worse to happen.  The remediation is quite simple.

chmod 755 /var/lib/pegasus /var/lib/pegasus/trace

This is a simple oversight that could lead to something possibly dangerous. However it is also easily made right. This is one of the reasons that is you are using a security assessment script that you run after every patch or update.

The other issue is more a historical item that could lead to a possible security issue if a malicious user does have access to the service console once more. This is the ability to run code as root possibly within the hypervisor using the vmkload_app and vmware-vmx commands. These programs are set up with their setuid bits which allows a normal user to run these as the super user. The reason these are set up this way is more an issue of history.

In VMware ESX 2.5.x days, it was possible for a normal user to run VMs. With VMware ESX v3.x this functionality was dropped from any of the management tools, yet the capability was left in. The solution is to remove the setuid bits so that these commands can only be run as the root user, which is their normal method of operation.

chmod u-s /usr/sbin/vmware-authd /usr/lib/vmware/{bin,bin-debug}/{vmkload_app,vmware-vmx}

It is interesting that while the DISA UNIX STIG and CISecurity CIS-CAT for RHEL found these issues, the TripWire ConfigCheck tool did not. This inconsistency has been reported to Tripwire as well, and they will work with VMware for a possible VMware Hardening Guide modification. TripWire ConfigCheck goes so far as to warn you that these files have had their setuid bits removed, when it should check to see if they are missing. It is a contradiction.

Making these changes will add to your security stance but not harden your system 100%. In my case, I never trust just one hardening guide as they often over look things that the others do not. I try to make my systems pass all the guidelines available and if there are inconsistencies between them, I document those decisions as well. Note that there are several inconsistencies in each guide.

These slight changes are also reasons you want to redo security assessments after you patch or update your VMware ESX hosts!

Unfortunately, as it’s a general virtualization roundtable this particular podcast forum cannot get into the intracacies of any one area such as security. While security is brought up from time to time within the roundtable, it is a detailed enough subject to warrant its own round table.

To that end, I would like to announce the first Virtualization security round table podcast to be held on Thursday Jan. 15 at 2:30 PM EST. This will be the first of a series of podcasts that will run every other week.

Roundtable podcasts, Twitter, blogs, and the VMware Communities Forum are some of the best ways to get information and help about virtualization products and resources.

As stated within the VMware KB article 1000761 it is possible to restore ESX to identical hardware; however, you need to reinstall ESX first and restore the data you backed up while making changes to how the system boots, else the Universally Unique Identifier (UUID) written by the installation will not work anymore as you have overwritten the data from your backup.

This method will restore everything effectively to identical hardware, however if you want to use new hardware, perhaps with different PCI devices, then the restoration would fail to properly configure the new devices. It may even fail to properly configure NICs if there are any IRQ differences between the supposed identical hardware.

So in these cases you would have to at least verify the configuration and fix anything that was broken. This could lead to a set of unknowns from a security perspective. You are after all trusting the backup was restored properly and if it was not, then you could end up with security issues. So the verification step would have to be extremely well documented.

It is far easier to reinstall VMware ESX to the hardware and to use a either a installation document,  kickstart, or other type of script to configure all the devices for you using either the Remote CLI or the VMware ESX CLI.

When restoring VMware ESX or VMware ESXi the best tool to have will be very good installation documentation that is easy to follow and has graphics and text for every step of the configuration.  These documents could be reviewed for security concerns, and used to derive the scripts that could do the work for you.

Top Virtualization Security Links

The following links are just a sample of what is at the aforementioned site and should be read in order for those interested in securing your VMware Virtual Infrastructure and unfamiliar with VMware ESX, at the same time these are great references for the experienced administrator.

• VMware Virtual Networking Concepts
• VMware ESX Server 3: 802.1Q VLAN Solutions
• Security Design of the VMware 3 Architecture
• DMZ Virtualization with VMware Infrastructure
• VMware Infrastructure 3 Hardening
• CISecurity VMware ESX Security Benchmark followed by the CISecurity Linux Benchmark
• DISA STIG (ESX STIG depends on the UNIX STIG)
  • VMware Communities thread on ESX_SRRSecure – Script to allow ESX to pass a DISA Security Readiness Review.
  • UNIX STIG
  • ESX STIG
• Proven Practice: VI3 Security Risk Assessment – Xtravirt.com
• Remote Authentication – Full/Partial AD Integration, Secure LDAP, NIS, …

It is recommended to read as each guide or benchmark as each covers things from a slightly different but useful perspective.

VMware VI:OPs has also launched the Top 100 Virtualization Security Questions. This can also be referenced from the Top Virtualization Security Links sites as well as other blogs of interest.

Security zones within VMware ESX are either based on networks employed (Management, VMotion, Storage, DMZ, Production, etc.) or it can be based on functionality such as hypervisor, management tool, backup systems, or virtual machine.

Each of these zones reflects a configuration for the VMware ESX host. It also reflects what is being run within the virtual machines as well as what network upon which the VMs and hosts resides.

Network security zones are relatively easy to understand. Each network would be isolated from one another except through well defined gateways and firewalls. The confusion among some security administrators is that they believe since the VMware ESX or ESXi host has multiple adapters that it would therefore act as an undefined gateway between the disparate networks.

This is simply not the case. Unlike other VMware products where networking is done using bridging technology, the virtual switch within the ESX host is similar to any other layer 2 physical switch. The physical NICs within the VMware ESX or ESXi host act as uplink ports between the physical switch and the virtual switch. Whether a virtual NIC used by virtual machines (VMs), or a vmkernel NIC used by the hypervisor, each virtual device connects to a portgroup within a virtual switch then out the physical NICs to the physical switches unless the communication is VM to VM on the same portgroup.

Since there is no way to layer virtual switches or portgroups, the only way to speak across them is by the use of well-defined gateways or firewalls. VMs can act as firewalls and gateways but only if they have more than one virtual NIC. So the rule of no multi-homed VMs is the best place to start unless it is a well defined virtual firewall or gateway.

The other definition of security zones is based on the roles used within the virtual infrastructure as well. There are three well-defined roles within the virtual infrastructure: hypervisor, virtual machine, and management tool.

The hypervisor controls everything so access to this must be limited to only the service console or management appliance, data stores used as VM storage, or VMware VMotion interfaces. Two of these are through vmkernel NIC used by the hypervisor as described previously: data stores for VM Storage and VMware VMotion Interfaces. Anything that accesses the hypervisor must be protected from those items that should not be able to access the hypervisor.

A virtualization management tool is another role within the system, these tools interact with the service console or management appliance for the virtualization host which will indirectly interact with the hypervisor. This indirect interaction is the key to handling this role. Virtualization management tools such as VMware VirtualCenter, HP SIM VMM plugin, VMware Lab Manager, etc. should be firewalled from other networks as they have indirect access to the hypervisor.

Backup systems act as another role within a virtual environment. These systems can interact directly with the storage devices attached to virtualization hosts through VMware VCB, or through the service console or management appliance just like virtualization management tools. This can also lead to indirect interaction with the hypervisor. Due to the direct and indirect access these systems should be firewalled as well from other networks.

The last role is that of a standard virtual machine. These virtual machines could live on any network but they should not be able to directly access the service console or management appliance of thse virtualization host in addition they should not be able to directly attach to the storage device ued by the virtualization hosts. Virtual machines are considered to be hostile to the virtualization host and should be treated as such.

Security zones depend on the security roles each component of your virtual environment directly or indirectly interacts. These few tips will aid in designing a secure environment.

Because it is based on a light-weight, kernel optimized for virtualization, VMware ESX Server is less susceptible to viruses and other problems that affect general-purpose operating systems. However, ESX Server is not impervious to attack, and you should take proper measures to harden it, as well as the VMware VirtualCenter management server, against malicious activity or unintended damage.

Because ESX Server runs a customized, locked-down version of Linux, there is much less likelihood of security exploits than in a standard Linux distribution. If you follow the best practice of isolating the network for the service console, there is no reason to run any antivirus or other such security agents, and their use is not recommended. However, if your environment requires that such agents be used, then use a version designed to run on Red Hat Enterprise Linux 3, Update 6.

The key here is following proper security best practices. This means taking steps like keeping your host patched in a timely manner, isolating Service Console network traffic from the other traffic on the network including VM traffic and following the additional hardening best practices listed in VMware’s Security Hardening Guide and CISecurity’s ESX Security Benchmark.

There are some good reasons for not installing AV software on the Service Console. The first reason is that it can impact on the performance of the ESX host and subsequently all of the virtual machines that reside on it because of the extra CPU, memory and disk resources that antivirus software typically uses. Antivirus software can be particularly resource intensive and can draw resources away from the virtual machines and potentially have a very negative impact on them. Another reason is that there is the possibility that the software may cause other issues on the Service Console as is true with any additional third party software that is installed on the Service Console.

Despite the reasons for not installing it some enterprises mandate that AV software be installed on all systems regardless of how secure they are. You might try and exempt your ESX hosts from this by explaining the following points:

1. The design of ESX does not allow for a VM that is infected by a virus to spread it to the ESX host through the VMKernel. It can only spread through traditional means which is typically over the network by leveraging open ports and OS vulnerabilities. If you isolate your Service Console network then it is protected from any VMs that may be infected.

2. Most viruses are written to exploit Windows systems, there are very few viruses that are written specifically for Linux systems.

3. ESX has a built-in firewall that protects the Service Console and blocks all ports except those few required by ESX and VirtualCenter.

4. ESX has a very good historical track record, I’ve never heard of a virus infecting the ESX Service Console. This doesn’t mean it can’t happen just that the chances are extremely low.

5. Explain the negative performance impact the AV software will have on an ESX host which can affect all of the VMs on the host.

If you are forced to install antivirus software on the Service Console because of security requirements in your environment then you should make sure that you run a version that is designed for the version of Linux that the Service Console uses. Additionally try and configure it as minimal as possible to minimize the impact on the server’s resources. It’s best to exclude all your VMFS volumes from scanning or at a minimum exclude specific virtual machine files like vmdk and vswp files that are frequently written to. Make sure and keep a close eye on the performance of the host to see how much impact the antivirus software is having on it. Also if you have to run on-demand scans make sure they are performed in off-peak hours when activity on the ESX host is low.

Let us take a quick look at the daily operations you may perform within your virtual infrastructure.

• Create/Move/Modify virtual machines (VMs)
• Backup/replicate VMs
• View performance and other data about hosts and VMs
• Install/Reinstall/Update hosts
• Add or remove administrative users from VMs and hosts
• Monitor your VMs and hosts
• etc.

The list is quite endless. Each one of these actions will affect security or be affected by security.

Ease of use, ease of administration is sometimes paramount. This is the age-old debate of security over usability. Do not knock holes in your firewalls, instead, use existing secure protocols to improve the experience. One I use is OpenVPN to access my administrative network workstation using pre-shared keys for encryption. The extra step of starting up the VPN is trivial with the service provided, this gives me secure access to the management network over which the tools can be run.

Affected by Security

There will be at least one more step involved in use of daily tools then you would normally use. Mainly some way to ensure your access to the management tools is secured from sniffing by your non-administrative network. Or the use of different procedures to ensure you have proper auditing in place.

One tool I use is a simple expect script called expectsudo which will allow me to run remote commands on my VMware ESX hosts while retaining logging of all access. To use this tool you must first install the expect RPM onto your host. Then setup sudo to allow access to the appropriate commands.

#!/usr/bin/expect --

set pass [lindex $argv 0]

set timeout 5

spawn /usr/bin/sudo [lrange $argv 1 end]

expect "Password: {send "$passr"; exp_continue

sleep 1

In the above script the first argument will be the password to use. This script can not be run remotely using any form of SSH. For example:
ssh user@hostname expectsudo password esxcfg-vswitch -l
The only drawback to this script is the need to have the password available, instead you can setup pre-shared keys for SSH and setup Sudo to allow the  user to issue certain commands without requiring a password. This form of single sign on relies on the security of the management workstation in use.

Security of your VMware Infrastructure relies on the security of all the management workstations and servers in use. Not just on the hardening of the VMware ESX host.

The concept of VDC-OS is to better define the various roles and to change how we as administrators view and manage our virtualized data centers. However, with tools like Bluelane the view begins to muddy.

An operating system provides the basic security and fundamentals to run applications and perform tasks as the users dictate. Users do not want to worry about security, they want to have the system just work. Bluelane helps this by allowing VMs to run even if they are not patched yet reap the benefits of some of these patches. Granted not all patching happens by Bluelane, but those patches that are network related will. Less patching means less downtime.

However, are there diminishing returns? Yes, you get protection but at what cost? Higher CPU utilization to handle all the myriad of network related patches that are necessary? Are you protected by zero day attacks? What if Bluelane is attacked directly?

Even with these questions to be answered, VMware’s purchase of Bluelane shows an intriguing picture of a true data center operating system that just works regardless of the application being run; one that has its basic security handled for them. This is one more tool that can be used with the distributed virtual switch that will span the data center.

Picture a ThinApp running as a virtual appliance with Bluelane to handle the network patching required? Where is the operating system in this picture?