Virtualization security

Virtualization security fears grow

As people who deal with virtualization every day — you work with it, I write about it — it’s easy to get caught up in the idea that it’s a ubiquitous technology with infinite use cases and unending appeal.

Well, in the words of Lee Corso:

There are still quite a few people out there who don’t feel totally comfortable virtualizing everything. And they’re not all newbies who don’t know what they’re talking about.

Continued »

Adding virtualization to the PCI standard

Earlier this month, I wrote about how the PCI standard was recently updated but still failed to take virtualization into account. Shortly after, VMware announced its participation in the PCI council to help address virtualization within the PCI data security standards. While this is certainly good news and will help tighten up the security standards around electronic credit card payments, the outcome of this announcement remains to be seen. The following are a few improvements that shouldn’t be too difficult to implement right away:

1) First and foremost, the PCI council needs to recognize virtual hosts and include them in the scope of the standard if any of the virtual machines (VMs) that reside on the virtual hosts fall within the boundaries of the standard. Currently, any server, network or device that has anything to do with cardholder data would be included in the standard and any audits that occur.

Additionally, if any virtual machine is included in the scope, then all of the virtual machines on a host should be considered in the purview of the standard because they all reside on the same physical server. Finally, as virtualization allows for VMs to be easily moved between host servers for failure recovery and load balancing, all of the virtual hosts in a cluster should be included within the boundaries of the standard as well.

2) Clarify the confusing item (2.2.1) that dictates that you can only implement one primary function per server. All they have to do is exclude virtual hosts from this item.

3) Most of the security items that are listed in the standard can be applied to virtual hosts as well. This includes things like audit logging, password policies and applying vendor patches.

4) Address virtual networking. Ensure that the security settings on virtual switches do not allow things like promiscuous code, forged transmits and MAC address spoofing.

By simply addressing these four areas, the Payment Card Industry (PCI) standard would be moving in a better direction. From there the council could delve deeper and address other specific areas on virtual hosts using some of the existing security guidelines. Another distinction it should make is between bare-metal and hosted virtualization products. Hosted virtualization products are typically less secure because the underlying operating system is not optimized for virtualization. As a result, they should be subject to tighter scrutiny and control.

Virtual machine security threat levels; don’t believe the hype

I received an email the other day from Wayne, Pa.-based SunGard Availability Services outlining some “essential” steps for addressing virtualization security challenges. In their email, the company urges users take certain measures, including installing security software, to make sure their virtual machines (VM) are safe from security threats.

There are many virtualization security products on the market today, yet reports of major VM security breaches are nil. In fact, the largest virtualization vendor, VMware Inc., asserts that its software is completely secure - possibly more secure than physical machines.

And even though the majority of VM security breaches I’ve heard about were hypothetical, performed by scientists through demonstrations or at hacker conventions, not in real data centers, I still receive a steady flow of press releases and product announcements addressing VM security issues.

So now, when I see security vendors warning users about un-named threats they need to prepare for, I am reminded of the U.S. Homeland Security Threat Level warning system.

Unfortunately, there are no published criteria for the threat levels of the Homeland Security system, so there is no way to tell whether the current threat level is accurate. And by the way, the threat levels have never been green or blue.

Because of this, the system can be manipulated by government officials. For example, during the Presidential election of 2004 when Republican President George W. Bush was running against Senator John Kerry, the Homeland Security Threat Level was bumped up, prompting some academics to speculate this was done by the Bush administration to scare voters into re-electing him. If so (and we will never know), it worked.

Unfortunately, decisions based on fear are usually not well thought out.

But I haven’t heard of any 9-11-style attacks on virtual infrastructures, and the virtualization users I speak with aren’t convinced they have anything to worry about. The thing that gets people to buy into virtualization security software is that haunting “what if” question that makes everyone default to the”better safe than sorry” mantra. After all, there is no harm in taking proactive steps to protect against the unknowns - just in case.

For instance, according to this article on the security benefits and risks of virtualization, “the [virtualization] drawback is based on fear of threats that aren’t around today but could become serious problems in the future.” Natalie Lambert, a security analyst with Cambridge, Mass.-based Forrester Research, continues in the article:

“One big concern is about what could happen if a flaw were found in a hypervisor, which would give attackers access to thousands of desktops sitting on a virtual server…That’s not a reality today, but it’s certainly a fear for the future.”

And as Sunguard said in its email, “With many organizations focusing on virtualization benefits, they must also examine core risks before it is too late - meaning security needs to be built in from the start.”

It is why we buy life insurance and car insurance and fire insurance for our homes. (Those damn what ifs and their expensive safeguards).

So, for the paranoid among us, check out SunGard’s suggestions for securing your virtual infrastructure here. As they say, better safe than sorry, right?

PCI Data Security Standard updated, but still does not address virtualization

Last week I noticed that the Payment Card Industry’s Data Security Standard (PCI-DSS) was recently updated on October 1, 2008, from version 1.1 to 1.2. PCI-DSS is a security standard set forth by a conglomerate of all the major credit card companies and is designed to protect cardholder data. As a result, any company that accepts credit cards is forced to comply with it.

About six months ago I wrote that the PCI-DSS standard did not specifically address virtual environments, and instead only focused on servers and networks that are directly involved with cardholder data. In other words, the specification dictates what must be done to secure a server that may store or process cardholder data, but if that server happened to be a virtual guest the host server would not be considered in the scope of the specification. Subsequently you could secure a virtual guest all you want, but if you do not properly secure the host server you could easily compromise the virtual guest regardless of how it was secured.

I downloaded the summary of changes document that specified all of the changes that were made from version 1.1 and 1.2, anxious to see if they had finally added parameters for virtual host servers. Out of the 14 pages of changes, there was still no mention of virtualization technologies in the specification. Surprised by this, I searched through the whole version 1.2, 72-page specification document for the word virtual and found only one instance of it for virtual private network.

I am puzzled as to why they would continue to ignore virtualization. After all, isn’t just about every company virtualizing in some fashion these days? Are the people that write the specification parameters just ignorant of what virtualization is, and that it has a direct impact on their regulations? Or are they just trusting that we are all securing our virtual hosts properly and there is no need to address them? If that’s the case then they have misplaced a critical amount of trust as I am sure there are a great many virtual environments that are not properly secured. Likewise, ignoring virtualization completely greatly reduces the effectiveness of their efforts to secure environments that deal with cardholder data. It’s essentially fortifying everything within a castle, but leaving the front gate open.

It wouldn’t require a great deal of effort for them to address virtual hosts. A number of security specifications for virtual hosts already exist, such as cisecurity.org’s for VMware’s ESX. Let’s hope that they wise up and address virtualization in their next update of the specification. Until then their efforts to protect cardholders are not complete. I just hope that my credit card data is not lying on a virtual machine somewhere that resides on an insecure host server that is ripe for the picking. After all, why try and hack a single virtual machine when you can instead hack into a whole host and gain access to all the VMs and their data?

BMC intros slew of virtualization management products

Houston-based BMC Software introduced several new virtualization management products today, including nine new integrated offerings designed to eliminate the risk and operational expenses associated with management of virtualized data centers.

BMC’s new virtualization management products are fully integrated with virtualization products from Microsoft, Sun Microsystems, Inc. and VMware Inc. The new BMC software is based on an automated set of closed-loop change and configuration management (CLCCM) process workflows that reduce the latency, cost and risk associated with change management. All of the new offerings support both virtual and physical infrastructures.

The nine new offerings support goals for performance, compliance and enterprise visibility by addressing the challenges created by virtualization.

Some of the issues addressed include the following:

*Planning a virtualization/consolidation initiative: BMC Virtualization Capacity Management and Planning Service is a packaged services offering that helps customers accelerate their virtualization efforts.

*Simplifying management: BMC Performance Management does complete performance monitoring across virtual infrastructure and applications with enhanced VMware Infrastructure 3 and VMotion support.

*Ensuring availability: BMC Application Performance and Analytics helps IT actively manage service levels in virtual infrastructures.

*Performance: BMC Capacity Management replaces educated guesses with automatic assessment, prioritization of server workloads, and ongoing capacity monitoring. The result is high performance while reducing capital and operational expenses and maximizing server consolidation.

*Server sprawl: Virtualization allows new servers to be created very rapidly, leading to virtual machine (VM) sprawl. BMC Discovery Solution helps customers keep virtualized environments under control by keeping tabs on virtual servers. Support for VMware, Solaris 9/10 containers and zones, AIX LPARS as well as z/VM dependencies on mainframe (z/OS) mean that all types of virtual servers can be discovered and added to BMC Atrium CMDB.

*VM security: BMC BladeLogic Virtualization Module for Servers adds security and strengthens licensing and regulatory compliance. It includes automatic provisioning and configuration of the entire software stack, including virtual infrastructure, guest VMs and applications, and enforces security best practices, including built-in virtual server hardening rules.

*Compliance: BMC BladeLogic Operations Management Suite establishes automated, closed-loop change and configuration governance over entire virtualized environments. BMC’s policy-driven configuration control prohibits noncompliant servers from being deployed or existing beyond the next audit scan. Automated compliance and remediation capabilities detect and correct any compliance violations.

*Administration costs: BMC Run Book Automation Platform and BMC Run Book Automation VMware Adapter exploit BMC’s CLCCM workflows to automate routine change management tasks.

Of course, BMC isn’t the only game in town when it comes to virtual infrastructure management. There are a number of vendors offering management products for various purposes, including Portsmouth, N.H.-based vKernel and San Francisco-based Hyperic, Inc.

In addition, Austin, Texas-based Surgient announced today its Virtual Automation Platform 6.0, which is designed with physical provisioning and Microsoft Windows Server 2008 Hyper-V support to manage virtual resources and eliminate physical server and virtual machine (VM) sprawl.

In addition to third-party VM management products, virtualization providers offer their own; VMware sells a proprietary management and automation suite, as does Microsoft for Hyper-V.

VMware updates its security hardening guide

VMware has just updated their security hardening guide, which provides recommendations for hardening a VI3 environment.

In addition to the updates for virtual machines and the ESX Service Console, they have now added new recommendations for ESXi, VirtualCenter Add-on components (plug-ins) and for Client components.

Here’s a brief overview of the recommendations for VMs and ESX hosts that have been added to the guide. No new recommendations were made for VirtualCenter except for the Plug-in ones.

Virtual machines:

• Disable copy and paste operations between the guest operating system and remote console
• Do not use nonpersistent disks
• Ensure unauthorized devices are not connected
• Prevent unauthorized removal or connection of devices
• Avoid Denial of Service (DoS) caused by virtual disk modification operations
• Specify the guest operating system correctly
• Verify proper file permissions for virtual machine files

ESX Service Console:

• Secure the SNMP configuration
• Protect against the root file system filling up
• Disable automatic mounting of USB devices

There are some general recommendations when using plug-ins and some specific ones when using Update Manager, Converter and Guided Consolidation. The guide recommends that the Update Manager and Converter plug-ins not be installed on the VirtualCenter server but should instead be installed on a separate server or virtual machine.

Also added is a section on client components. The guide recommends against the use of Linux-based clients when using the RCLI, VI Perl Toolkit scripts, VM console access initiated from a web access browser session and programs written using the VI SDK. The reason for this is that communications with Linux clients are vulnerable to man-in-the-middle attacks because the Linux versions of these components do not perform certificate validation. This risk can be partially mitigated by ensuring that the management interfaces (ESX Service Console and VirtualCenter) are on trusted, isolated networks.

The guide suggests that client components are to verify the VI Client integrity because of the VI Client extensibility framework that was introduced into VirtualCenter 2.5 which provides the ability to extend the VI Client. It also recommends that one monitor the usage of the VI Client instances by inspecting log files on client systems. Both of these tasks can be quite difficult to do because there are no native methods for doing this.

Finally a section was added for securing the host-level management in ESXi. Many of the recommendations for ESXi are the same ones that were made for ESX. Some unique recommendations for ESXi include ensuring secure access to CIM (the hardware management api’s). Also, admins may want to audit or disable the special technical support mode which is designed to be used in case of an emergency but is sometimes used by administrators to access specific functions in ESXi.

You can read the updated guide in its entirety here.

Protecting virtual disk files from nosy admins

I recently came across an article revealing that 1 out of 3 IT administrators have used their elevated privileges to snoop on confidential information. It’s always possible to lock out administrators to sensitive data through operating system access controls, however, a virtual environment opens up other avenues for exposing sensitive data.

With physical servers, the task of imaging a server’s hard drive for offline examination is not always easy. An administrator of a virtual environment can easily and stealthily snapshot a virtual machine to temporarily suspend writes to disk file, make a file system copy of the VM’s disk file from the host server while it is running and then take that copy to a workstation where they can mount it and attempt to gain access to information to which they would normally not have access.

Either by mounting the disk file to an existing VM then adding an additional hard drive to access the information on the drive, or creating a new VM and mounting a live CD to utilize hacking utilities to defeat the operating system security, admins can bypass operating system level controls to gain access to the data simply by making a copy of the disk file and mounting it elsewhere .

Virtual servers open up additional attack vectors over physical servers, illustrating why proper security measures must be utilized to ensure that sensitive data is adequately protected in virtual environments. In addition to properly securing host servers, auditing and logging should also be in place to track all logins and activities on host servers. Administrators typically need access to sensitive data to be able to do there jobs but this access should be limited as much as possible to only what they actually need.

Many administrators snoop because they know they can get away with it. By restricting access and logging events, the 2/3rds of IT administrators who set the better example make snooping more difficult for nosey admins.

Tripwire offers free security utility for VMware ESX 3.5 hypervisor

VMware Inc. and Tripwire Inc. have co-developed a free, downloadable utility to address the leading security concern in virtual environments today: misconfiguration of the hypervisor.

Portland, Ore.-based Tripwire ConfigCheck is a free Windows and Linux based utility that assesses the security of VMware ESX 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines, which were released in February.

The Security Hardening guidelines explain in detail the security-related configuration options of the components of VMware Infrastructure 3 and how security affects certain capabilities.

Tripwire ConfigCheck makes sure ESX environments are properly configured according to these guidelines and lends insight into vulnerabilities in virtual environments. It also provides the necessary steps towards full remediation.

Dan Schoenbaum, senior vice president of marketing and business development for Tripwire
said the utility is being offered for free to encourage the proliferation of VMware’s Hardening guidelines and to increase virtual machine (VM) security.

Tripware hopes that by giving a taste of their technology for free, users will become familiar with them and invest in their software products with more security capabilities, Schoenbaum said.

Colorado Springs, Co.-based Configuresoft Inc. also provides a toolkit for compliance with VMware’s security hardening guidelines. The toolkit consists of a set of rule-based templates, reports and dashboards that plug into Configuresoft’s Enterprise Configuration Manager (ECM).

VMware pushes desktop virtualization on management and security benefits

VMware Inc. Senior Director of Enterprise Desktops Gerald Chen visited our office on Tuesday morning to discuss the different types of desktop virtualization and answer common questions about Virtual Desktop Infrastructure (VDI), for example, how it differs from terminal services and cost issues.

Here’s how VDI works: each end user gets a virtual machine (VM) that is deployed from a server in the data center directly to a PC, laptop or thin client computer. Each VM is customizable, so all of the user’s settings are saved and re-booted each time the user signs in, Chen said.

When a user logs off for the day, their VM goes idle, and wakes back up when the user logs into their system again, according to Chen. Chen believes that the advantage of VDI is that sensitive data is not being stored on desktops, which can easily be lost or stolen, and these virtual desktops are easier to manage than physical ones.

“VDI is great for industries like health care that are really concerned about information security and compliance. The real value though, is in management. All of the information is safe in the data center, and centrally managed through Virtual Infrastructure,” Chen said. “For instance, if you have 100 new employees who need desktops, you can deploy a VM for each of them in just minutes, and manage all of them centrally.”

VDI is different from Sever Based Computing (SBC) systems like Citrix Systems Inc.’s XenApp in that VDI is connects a single user to a single operating system (OS), instead of having multiple users share one OS.

“Not every application likes to share an OS, and there is also bad isolation; if one application crashes, everyone sharing that OS crashes as well. Those desktops can’t be customized either. It is a locked environment.”

Chen went on to explain that with VDI, four to ten VMs per server core are supported, so a server with one quad-core processor can, theoretically, house 40 VMs. Of course, that varies depending on things like workload, applications and memory. If the VMs become too heavy for the server to handle, management features in VI3 intervene. VMotion can move live VMs from one server to another when capacity issues arise, as can Dynamic Resource Scheduler, which allocates and balances computing resources as needed using VMotion.

Desktop virtualization case study
As VMware announced customer case studies in February, including one at Huntsville Hospital in Huntsville, Alabama.

The hospital needed to implement a new medical information application throughout its network while protecting HIPAA-related data. Deploying hosted desktops on VMware, the hospital could lock down sensitive patient data and reduce the cost and complexity of desktop management.

They used combinations of thin clients and blade servers to access the centralized virtual desktops, and in turn, reduced power consumption across the hospital by 78%, improved longevity with lower hardware maintenance needs and made wireless thin clients on wheeled carts available to hospital staff. Also, doctors can remotely access their VMs through the Internet using a web browser when necessary.

The downside to desktop virtualization
While the benefits are clear, there are some downsides to desktop virtualization: extra storage and initial cost.

Chen told SearchServerVirtualization.com that VMware is working on reducing image sizes and has designed a way to keep only one copy of files that are identical among many users, like icons and other graphics, to reduce the amount of storage necessary.

The cost of implementing desktop virtualization turns users off. According to Ars Open Forum blogger ‘Bright Wire,’ the cost and the magnitude of system upgrades required is not worth the benefits.

“The cost of deploying virtual desktops is massive,” Bright Wire wrote. “You will need to re-gear your existing desktops to run the virtual or you will need vendor equipment that costs twice as much as a new desktop. Either way, the cost is big in manpower. On top of that, your infrastructure will need serious review.”

According to VMware’s product specifications, local desktop virtualization requires a 500 MHz or faster processor with recommended 256 MB of memory, though Forrester reports that PCs must be faster and have more RAM to work efficiently.

“In addition you need to look into the server infrastructure,” Bright Wire said. “You are talking about needing a lot of iron on the backside to handle the needs of the server to supply two to 16 desktops. All this adds up quickly and can easily swamp a datacenter.”

As for pricing complaints, VMware is used to hearing them and holds firm to the ‘you get what you pay for’ mantra, saying the management benefits are worth the price.

The company charges $150 per concurrent user plus additional costs for support, either Gold or Platinum levels. Both bundles include VMware Infrastructure Enterprise Edition for VDI (which consists of VMware ESX Server 3.5 and VirtualCenter 2.5) and the VMware Virtual Desktop Manager 2. The VMware VDI Starter Edition, which enables 10 virtual desktops, has a list price of $1,500. The VMware VDI Bundle 100 Pack, which enables 100 virtual desktops, has a list price of $15,000.

The market indicates a demand for desktop virtualization, as a number of other vendors also entered the desktop virtualization space including Sun Microsystems Inc., Citrix., Pano Logic Inc. and Symantec. Chen would argue that many customers come for reduction in hardware but stay for the management applications.

“Reducing hardware costs is not a reason to use VDI, it is management. We have customers who have seen 40% to 50% ROI in terms of management costs and the amount of time it frees up.”

Staying vigilant about virtual security

With all the talk about virtual security these days , you would think that people actually are addressing the concerns over security in virtual environments. However, many administrators resist implementing strict and proper security measures in their environments because of administration inconveniences that tighter security usually causes.

For example, the default settings of VMware ESX prevent users from using secure shell (SSH) to log into the server as the root user. Yet, the first thing many users do is to modify the SSH configuration to allow root access via SSH because this is a more convenient way to log into Service Console. The correct and more secure way to do it would be to setup a separate SSH user account and then use the SU – command to gain root privileges. Xtravirt has published a good step by step guide on how to do this here.

When you virtualize servers, additional security measures should be followed in addition to standard ones that you would use for physical servers. Most importantly, the host system must be protected at all costs: If someone gains control of the host server then all of the VMs that run on the host can be compromised. The Center for Internet Security (CIS) has published some security guidelines for ESX and virtual machines that I would recommend you read through and follow to ensure your environment is secure. Xtravirt has a great security assessment template that they’ve put together that you should look at also.

Virtual networking is another critical area for securing virtual hosts. Virtual switches differ from physical ones and must be properly configured to ensure secure host and virtual machine network traffic. Often, simple recommendations like isolating Service Console and vMotion traffic are not followed, which creates unnecessary risk and exposure of your hosts.

Are you willing to risk losing your data? Data breaches can result in negative press exposure, lawsuits and fines. I would encourage everyone to please take security seriously. Security may cause some administration inconveniences and headaches, but they are a small price to pay to ensure that your servers, and more importantly your company’s sensitive data, is well protected and safe.

To help you with this I’ve included a list of some good virtualization security blogs and websites that you should check out:

• SearchServerVirtualization Security blog (various contributors)
• Rational Survivability (Christofer Hoff)
• VMware Security Blog (VMware)
• Vmtn Security Community
• Security Focus
• Security Top 10 List and Security Links