Security

VMware ESX Web access in vSphere

With VMware Infrastructure 3, Web access was enabled by default. VMware chose to disable it in vSphere for security purposes. If you access an ESX 4 host with a Web browser you will see the default welcome page but if you click the log-in link you will get a 503 Service Unavailable error message.

This only affects ESX hosts (VMware vCenter Server has this enabled by default; ESXi does not have a Web access user interface (UI) to manage the host and virtual machines).

There is a tech note that describes the process for enabling this feature in ESX 4.0, but before you go ahead and do this you should ask yourself if you really need this feature enabled on all your hosts. VMware disabled this feature for a reason — Web based access methods are inherently insecure and are subject to numerous vulnerabilities that could potentially compromise your VMs and hosts.

The Web admin UI is very limited as you can only administer guest VMs and not host servers. Leaving it disabled removes a potential attack vector for your ESX hosts and makes them more secure. This is also true of other configuration settings that are disabled by default in ESX such as root access using SSH.

The first thing many administrators do is enable this because it is easier than setting up another user account and using su or sudo. So resist the urge to enable web access and utilize the vSphere client instead. If you must use web access for a specific reason only enable it on the hosts that need it. If you are using a vCenter Server use the web access on vCenter instead and use the roles and permissions built into it for additional security. By leaving vSphere web access disabled you are helping to make your ESX hosts and your whole virtual environment more secure.

Preparing for virtualization PCI security standards

This week I went through out annual audit process for the fourth time, and as usual the virtual hosts were mostly ignored. Why? The Payment Card Industry (PCI) security standards have yet to be updated to include the virtualization layer of environments that are audited for PCI compliance. The auditor acknowledged this fact and said at some point in the future this may eventually change so virtual hosts are more closely scrutinized.

The purpose of PCI standards is to ensure that IT environments meet a set of security standards to ensure the protection of card holder data and are a requirement for any companies that take credit cards and have a certain transaction volume.

VMware announced their participation in the PCI Council last year but so far nothing has come out of it. As to how this will affect a new PCI standard, this is anyone’s guess. The auditor I spoke with suggested that new regulations might require segregation of hosts so you do not mix development and test virtual machines (VMs) on the same hosts as productions VMs. Many large environments already separate their test and production VMs, but smaller environments that have a limited number of hosts may find this difficult. New regulations may also require further segregation so hosts that have VMs that are involved in the processing or storage of credit card data are isolated from other hosts.

Whatever comes out of VMware’s participation in the PCI council, we should finally see virtualization covered in the next update. Currently, PCI specification is at version 1.2, and was last updated in October 2008. This participation is critical to the success of the PCI standard, as applying security standards to VMs means nothing if you don’t also apply them to the hosts that the VMs reside on.

The lifecycle process document for the next version of the PCI specification indicates that the next release is not due out until 2010. Since virtualization will be added to the PCI specification in the future it would be wise for an administrator to start getting ready for the upcoming changes today. This would include assessing your virtual environment to identify hosts and VMs that are involved in credit card data, planning on how you might segregate your environment, reading through the published security standards for ESX hosts (i.e. CIS Security Benchmark, VMware’s hecurity hardening guide) and applying them and using virtualization specific security tools to monitor and secure your virtual environment.

VMware has a new Compliance Center on their website that includes white papers and presentations specifically on virtualization and PCI compliance. By doing this now you can be better prepared when the PCI standard is updated and be ahead of the game as often times you will find yourself scrambling to remediate your environment once the new PCI standard is in place and you are audited.

Editor’s note: For more information on how current auditing fails to address virtualized servers, read about how three separate auditing firms failed to address several security vulnerabilities involving virtualized servers in Virtual machine security enters the mainstream.

Useful VMware virtualization roundtable podcasts

The VMware Communities roundtable podcast drew in a large crowd on Wednesday the 7th. In general, the show draws hundreds of listeners that download the recorded sessions available via Talkshoe and iTunes. The host of the podcast is VMware’s John Troyer and the show provides an ongoing forum to discuss current VMware technical issues. The show also provides in-depth information on both older and newer VMware products. I am a panelist in the forum, as are several other SearchVMware bloggers. Join us every Wednesday at 3:00 PM EST.

Unfortunately, as it’s a general virtualization roundtable this particular podcast forum cannot get into the intracacies of any one area such as security. While security is brought up from time to time within the roundtable, it is a detailed enough subject to warrant its own round table.

To that end, I would like to announce the first Virtualization security round table podcast to be held on Thursday Jan. 15 at 2:30 PM EST. This will be the first of a series of podcasts that will run every other week.

Roundtable podcasts, Twitter, blogs, and the VMware Communities Forum are some of the best ways to get information and help about virtualization products and resources.

Solution for VMware Player security vulnerability

The gang at Milw0rm have posted one of the few exploits against VMware’s desktop line of products, specifically VMware Player version 2.5.1. This exploits the vmwarebase.dll file when running VMware Player on Windows systems. There is no chance of being able to run generic code through this exploit. In addition, VMware ESX, VMware ESXi, VMware Server and older versions of VMware Player are unaffected.

The issue occurs when you pass long usernames or passwords into VMware-Authd which will cause the application to crash. While it is restarted, a DoS attack could occur which would keep crashing the application and possibly filling up disk space with crash files as well.

VMware has been very responsive to this vulnerability and a fix is already completed. Check out this VMware thread for some more details. This type of response is quite commendable.

Milw0rm is a great resource for exploits of any kind, but since it is used by hackers it is best to access this site from a system you do not mind rebuilding occasionally; a VM works as does using The Onion Ring (TOR) plug-in for Firefox and Internet Explorer. It is best to be overly cautious when browsing sites that hackers create and visit, though they often have the latest exploits and attacks available.

Another good site is prometric.com, which hosts the 100 top hacker sites. As always be cautious when accessing any of these sites.

A hearty “good job” to VMware for their response to this and all other security issues!

Security outside the box

Virtualization security depends on more than securing the virtualization host and hypervisor. It depends on everything that touches your virtualization host, directly or indirectly, also being secure.

Let us take a quick look at the daily operations you may perform within your virtual infrastructure.

• Create/Move/Modify virtual machines (VMs)
• Backup/replicate VMs
• View performance and other data about hosts and VMs
• Install/Reinstall/Update hosts
• Add or remove administrative users from VMs and hosts
• Monitor your VMs and hosts
• etc.

The list is quite endless. Each one of these actions will affect security or be affected by security.

Ease of use, ease of administration is sometimes paramount. This is the age-old debate of security over usability. Do not knock holes in your firewalls, instead, use existing secure protocols to improve the experience. One I use is OpenVPN to access my administrative network workstation using pre-shared keys for encryption. The extra step of starting up the VPN is trivial with the service provided, this gives me secure access to the management network over which the tools can be run.

Affected by Security

There will be at least one more step involved in use of daily tools then you would normally use. Mainly some way to ensure your access to the management tools is secured from sniffing by your non-administrative network. Or the use of different procedures to ensure you have proper auditing in place.

One tool I use is a simple expect script called expectsudo which will allow me to run remote commands on my VMware ESX hosts while retaining logging of all access. To use this tool you must first install the expect RPM onto your host. Then setup sudo to allow access to the appropriate commands.

#!/usr/bin/expect --

set pass [lindex $argv 0]

set timeout 5

spawn /usr/bin/sudo [lrange $argv 1 end]

expect "Password: {send "$passr"; exp_continue

sleep 1

In the above script the first argument will be the password to use. This script can not be run remotely using any form of SSH. For example:
ssh user@hostname expectsudo password esxcfg-vswitch -l
The only drawback to this script is the need to have the password available, instead you can setup pre-shared keys for SSH and setup Sudo to allow the  user to issue certain commands without requiring a password. This form of single sign on relies on the security of the management workstation in use.

Security of your VMware Infrastructure relies on the security of all the management workstations and servers in use. Not just on the hardening of the VMware ESX host.

VMware ESXi security review: Firewall, please

VMware ESXi may have a smaller footprint than VMware ESX, but the pro-security theory behind the skinny ESX version may be defunct given the lack of ability to create a Defense in Depth strategy around the hypervisor. As is, I suggest you consider ESXi a safe hypervisor only when behind a firewall.

VMware touts ESXi as being more secure by having less of an attack footprint, but it is missing the most important feature of modern operating systems: The ability to build a strategy to protect the system from those reaching it and users gaining access to those things to which they are not authorized, currently known in the IT world as Defense in Depth.

Defense in Depth is more than just the availability of a packet filtering firewall, a la VMware ESX’s iptables-based firewall. It is the ability to control when and from where users can log in, and how they can access or view information on the system as well as audit all actions within the system for later perusal or immediate notification of unauthorized access to data or the system. Defense in Depth often starts with the use of a directory service as a centralized management point for all users.

VMware ESXi v3.x is missing all of these capabilities. Directory services are not supported within the VMware ESXi management appliance, there is no ability to audit actions that take place while on the management appliance, there is no control of when or how a user can access the appliance, and most importantly there is no built in firewall.

All of this begs the question: does ESX’s smaller footprint really offer a more secure hypervisor? From the network facing view, its attack surface is limited but not as much as you would expect. What a user can do or access once on the system is also limited, but also not as much as you would expect.

Network daemons almost the same as ESX, minus default SSH

From the network perspective, all the normal daemons that are available for VMware ESX are also available for VMware ESXi: vmware-hostd daemon, cimserver, time daemon, and webAccess.  What is missing by default is SSH access, which most ESXi users enable immediately. The ability to start most other services on the system is also missing. In other words it has nearly the same network daemons running by default that ESX does.

The major difference is that you can no longer log directly into the ESXi box without degrading your security first, in other words without enabling the dropbear SSH server. This does not need to happen and should not according to VMware.

Management of ESXi is performed by either direct access via Remote Command Line Interface (RCLI), the VMware Infrastructure Client (VI Client) and VMware webAccess or by going through VirtualCenter.  Each of these use SSL in order to encrypt and protect all traffic from the workstation and ESXi host. These are the same tools you can use to manage ESX and, as such, share all the weaknesses and strengths. All administrative access via these tools is performed through the vpxuser account which in turn runs many commands as the root user. This is no different than what ESX does. If you go through VirtualCenter, however, you can gain the benefits and disadvantages of using a directory service, but this is not the case when going direct to ESXi.

Possible split-brain authentication

The largest security difference as discussed above is that there is no Defense in Depth, and that once you break the shell by enabling SSH you now run into possible split-brain authentication and authorization that did not exist before. This implies that unprivileged users can gain access to data which they do not own and should not be able to access or even see.

Lastly, since ESXi has no Defense in Depth, its management appliance belongs behind a firewall of its own. This is a step backward in my opinion, and hopefully will be fixed in future releases!

Google and Microsoft creating virtualized cloud computing data centers

Google and Microsoft are forming mega data centers with low energy costs and serious tax advantages by using renewable energy solutions. Each have negotiated extremely low energy costs direct with the energy providers and have created practically a zero-carbon footprint. The costs for a data center are not all about energy, although energy is a major expense. What would happen if/when a bank or federal government bought into the cloud? Surely security in a cloud computing infrastructure would need to be top of the line, thereby expensive.

The idea is that Microsoft and Google will get their energy at extremely low per-watt prices, and eventually, since they are using renewable energy sources, they may receive a few credits back from the energy providers as they sell off their excess.  They could even get state and federal subsidies and tax breaks by using renewable resources.

But what does this really mean to the virtualization world? Who will actually use these mega data centers?  I imagine part will be for Microsoft and Google themselves, but they plan on selling or renting space within their cloud for applications and services. They may sell quite a bit to the low-hanging fruit that comprises startups and other SMBs who can not afford all the modern equipment, but will they be able to sell to others?

What about security?

The big question in my mind is can you trust either Microsoft and Google from a security perspective. Cloud computing security is still up in the air. Would my bank use these new clouds? Would the federal government?

If my bank uses it, I imagine there will be extremely tight security. How much will I have to pay for this level of security? Would this level of security increase the cost so much that the benefits of low cost energy go by the way side? Security implementations cost, sometimes heavily.

Google and Microsoft may be able to reduce cloud computing energy costs, but at what other costs. If I gain part of a host due to virtualization, how much of my data is comingled on the storage devices and network paths? If a badly configured host is in use, can another company see and gain access to my data? How is this protected within the cloud? In the current virtualization world, this requires dedicated resources which are more expensive than shared resources.

Simply put, we can not ignore security going forward. What is the security and privacy guarantee — not to mention the real cost of use once these concerns have been addressed?

VMware’s Bluelane purchase a move to true VDC-OS

There’s more to VMware’s purchase of Bluelane than meets the eye. Touted as a means to beef up VMware’s security and high availability options within the virtual infrastructure, this purchase is instead more of a move to a full VDC-OS…and not just a concept as presented at VMworld 2008.

The concept of VDC-OS is to better define the various roles and to change how we as administrators view and manage our virtualized data centers. However, with tools like Bluelane the view begins to muddy.

An operating system provides the basic security and fundamentals to run applications and perform tasks as the users dictate. Users do not want to worry about security, they want to have the system just work. Bluelane helps this by allowing VMs to run even if they are not patched yet reap the benefits of some of these patches. Granted not all patching happens by Bluelane, but those patches that are network related will. Less patching means less downtime.

However, are there diminishing returns? Yes, you get protection but at what cost? Higher CPU utilization to handle all the myriad of network related patches that are necessary? Are you protected by zero day attacks? What if Bluelane is attacked directly?

Even with these questions to be answered, VMware’s purchase of Bluelane shows an intriguing picture of a true data center operating system that just works regardless of the application being run; one that has its basic security handled for them. This is one more tool that can be used with the distributed virtual switch that will span the data center.

Picture a ThinApp running as a virtual appliance with Bluelane to handle the network patching required? Where is the operating system in this picture?

Changing the VMware Server 2.0 default permissions

I have been using VMware Server 2.0 (beta 2) on both Windows and Linux platforms for a while now. For Windows systems that are a member of an Active Directory domain, there are inherited permissions that may be assigned from Group Policy. If you want to change that, here are a couple of pointers in changing the security model.

To start looking at the permissions of the server installation, click the server on the left side of the browser view and then click the permissions tab at the top. The default permission is the local Administrators of the Windows system will be in the VMware Server Administrators role as shown below:

Before you make any modifications to the security model, add your desired configuration. That way, you can protect yourself from orphaning your administrative access to the server. Click the New Permission link in the command section, and add the user from either the local accounts of the Windows system or a user from the Active Directory domain or security group from the domain. The figure below shows the addition of the RWVVMwareAdmins group to the role of Administrator within the VMware Server 2.0 web interface:

Once that is added, the new configuration should be tested to ensure the proper access is available. Once the new access is verified, it would be safe to then remove the previous default access (if needed). If you get stuck, you can save off the .VMDK files and reinstall the product if needed.

While the web interface for VMware Server 2.0 takes some getting used to when compared to the thick client for versions 1.0x, features continue to be added to the free virtualization product that can be suited to test and development or live environments.

VMware security coming to the forefront?

Many virtualization analysts punt on the issue of security. But two recent events have brought security into higher relief: the uncovering of VMware’s file-sharing security flaw and VMware’s announcement of VMsafe, a virtual appliance that adds a layer of security to apps running on virtual machines. While VMsafe attempts to address VMware’s file-sharing problems, the flaw has raised questions about VMware security and the security of virtualization technologies in general.

In a recent SearchSecurity.com article, one interviewee said that after testing virtualization, he determined that putting virtualization into production would require reworking tried-and-true centralized security controls. Another interviewee expressed concerns about future problems, particularly a breach involving the hypervisor.

But we aren’t the only ones asking questions about security for virtual environments. At Rational Suvivability, author Christopher Hoff takes a different angle:

Virtualization up until now has quietly marked a tipping point where we see the disruption stretch security architectures and technologies to their breaking point and in many cases make much of our invested security portfolio redundant and irrelevant.

Has virtualization brought a whole new set of security requirements? Has your company explored or purchased virtualization-specific security software? Share your security-in-virtual-environments experience, and we’ll send you a $10 Starbucks gift card. Email me at hdrake@techtarget.com.