Then, the hacker claiming responsibility for the leak reportedly told Kaspersky Labs’ Threatpost blog that among those files, a terabyte in all, there were 300 megabytes (MB) more VMware source code.

Thus, it was widely anticipated by the VMware community (including this blog) that 300 MB of VMware source code would be released on Saturday.

On May 3, VMware rushed out a bunch of critical patches for ESX, ESXi, Workstation and Player, heightening the anticipation.

The big day has now come and gone, however, and there was nary a whisper of VMware’s name on various Twitter accounts associated with the initial leak. If 300 MB more source code did hit the Internet this weekend, it was done with far less public fanfare than the “sneak preview” received.

Users say the lack of leak doesn’t change much about their outlook on the situation.

“These types of hackers are criminals, and criminals aren’t known for keeping their word,” said Bob Plankers, a virtualization architect at a large Midwestern university. “There are a number of security updates now available for nearly every version of vSphere and its predecessors, so at the least it looks like VMware took the issue seriously on all fronts.”

Trying to guess at what happened means trying to figure out the agenda of a hacker, which is nearly impossible to do, said Edward Haletky, CEO of The Virtualization Practice LLC. It might have been that the wide-ranging publicity the initial leak received was all he was looking for.

“It could’ve been truly just about awareness, saying, ‘hey, you know, this code really isn’t private anymore’,” Haletky said. “There could be a million and one reasons.”

The fact that there was no obvious code release on May 5 shouldn’t make much difference to VMware pros, Haletky said. They should still apply VMware’s new patches and keep up with security best practices. “The answer still is to prepare for such things…do the defense in depth, do the research…if it happened once, it could happen again.”

Affected products include ESX and ESXi versions 3.5, 4.0, 4.1 and 5.0, Workstation and Player. A further description of problems associated with the patches and linked from the security update blog describes remote procedure call (RPC), SCSI driver and network file system (NFS) vulnerabilities which could potentially allow an unauthorized user execute code on a virtualized host.

With the post’s repeated use of the word “critical,” and widespread Tweeting of a link to it by VMware officials, it’s clear the patches are important. In fact, such a security update hasn’t been posted on the VMware Security and Compliance Blog since the announcement of a critical update to ESX 3.5 in 2008.

Though the post referred directly to the leak incident, what’s less clear is the exact relation of these newly announced vulnerabilities and the leaked source code file.

VMware framed the security advisory as the accelerated release of patches the company was working on anyway. “In light of the current circumstances, we have accelerated our most recent security patches and applied them to all affected currently supported products,” the post said.

“I think it is an abundance of caution, but in addition, some pro-active concern,” said security expert Edward Haletky, CEO of The Virtualization Practice LLC. While there is historical evidence that it is possible to crash a VM using paravirtualized drivers and backdoor elements in the past, he added, “the execution of code on the host is intrinsically difficult regardless of how an escape is performed.”

These aren’t the first VMware product patches which raise the spectre of rogue code executed on a host – even in the last few weeks. A security advisory was also issued without nearly as much fanfare April 12, in which three critical patches were released for VMware’s vShield Endpoint security product.

VMware’s Knowledge Base article paired with today’s security advisory also specifically credits an individual, Derek Soeder of Ridgeway Internet Security LLC, with identifying some of the vulnerabilities, rather than specifically linking their discovery back to the leaked file. Soeder, meanwhile, was publicly raising security issues with VMware’s software in a blog posted March 30, before the 2004 source code file was leaked.

Regardless of whether the hacker who threatens to leak megabytes more source code on May 5 acts on that threat, or whether these patches are specifically related to the high-profile leak, VMware customers shouldn’t take any chances, experts say.

“For now, all we can do is what we should always do, keep current on our patching levels,” said Christian Mohn, senior infrastructure consultant at EVRY Consulting in Norway.

Meanwhile, “May 5th might just turn into something more interesting than I had thought a week ago,” he said.

The code, which dates to 2003 or 2004, was apparently stolen from “a variety of compromised Chinese firms,” according to a Threatpost report. The code was confirmed as genuine by the director of VMware’s Security Response Center in a blog post yesterday. Although only a single file has been released publicly, the hacker claims to have another 300 MB of source code and that the rest will be published May 5.

If the rest of the code is of the same vintage, it may not be much of a threat. In fact, providing a more secure hypervisor was a primary goal of the conversion over the last year from ESX to ESXi, a set of code with a much smaller attack surface. So far, no data has been published which indicates the ESXi hypervisor is involved.

But if the remaining code published May 5 is more current, and contains information that could allow hackers to access hosts from guests, it could potentially pose a security threat to enterprises as well as cloud service providers with infrastructures based on vSphere.

The worst-case scenario is that such a “VM escape” is found, but not published, according to Bob Plankers, virtualization architect with a large Midwestern university.

“There’s a lot of money to be made by hacking enterprises,” he said. “So VMware and their customers would be best served by an attitude akin to a race: who can find all the security holes first?”

The risk is probably not very high right now based on what’s been released, according to security expert Edward Haletky, CEO of The Virtualization Practice LLC. But “believe me, on May 5, I’ll be paying attention to what is released,” he said.

So far, escape-the-VM attacks have proven relatively toothless – none has been able to really do much to cross VM boundaries even when they have penetrated the hypervisor in experimental settings, Haletky said. If areas of the code having to do with the virtual machine manager leak out, it could help such an attack do more damage.

For now, it’s much easier to attack virtual machines through the management layers, and therefore much more common, Haletky said. Enterprises can protect themselves by following security best practices such as separating management networks from storage networks, fault tolerance and vMotion networks; limiting the footprint of VMs; effective network monitoring; and using early warning systems. But it’s something he says most enterprises don’t do.

“I think this may push more people to follow best practices because of the increased awareness,” he said.

IT pros shouldn’t expect this to be an isolated incident, according to Haletky. VMware and its competitors have become high-profile enough that their software is a juicy target for potential attackers.

“Years ago…we said we can’t say there won’t be a major incident involving one of the hypervisor vendors, whether it be VMware, Microsoft or even Citrix or Red Hat, and it’s going to be disastrous,” he said. “Does this raise the risk for VMware? Yes. As a company, absolutely.”

The vSphere 5 edition of VMware’s Security Hardening Guide is still in the works, but one blogger brought up a potential conflict between the API, called VIX, and a recommendation against enabling it in the Hardening Guide issued with vSphere 4.1.

“SRM now requires that the VIX API be enabled on all protected virtual machines that will have their IP changed during recovery,” according to the blog post by Michael Webster, a VMware Certified Design Expert and director of IT Solutions 2000 Ltd., a VMware consultancy based in Auckland, New Zealand.

Previously, users had the option of changing IP addresses without using the API, which is slower but considered more secure. “This has already caused me design problems in a number of customer environments,” Webster wrote.

However, enterprises that don’t require the highest security measures may not run into an issue, experts say.

Shannon Snowden, a consulting partner at New Age Technologies in Louisville, Ky., said he has yet to run across the problem despite having done several large-scale SRM deployments over the past few months.

“If it is of concern, we could most likely use a couple of scripts to enable it temporarily during the actual SRM event then disable it as a post-recovery step,” Snowden said. “Obviously, I would prefer to have the old way as an option along with the new faster way, instead of having to put together and coordinate scripts.”

While most companies probably won’t be impacted, the use of the VIX API to change the IP address of virtual machines (VMs) may be a problem for customers in government, research and finance industries, said Bill Hill, infrastructure IT lead for a Portland, Ore.-based logistics company. He doesn’t anticipate it will be a problem in his shop, but he can see where it might be for some.

“Ultimately, VIX allows for significantly more access to a virtual machine outside of just changing the IP address,” Hill said.

Other operations enabled by VIX include the ability to copy files from hosts to guests and guests to hosts, for example.

As an alternative to the API, IT pros may be able to use Dynamic Host Configuration Protocol (DHCP) to assign IP addresses to VMs according to MAC address, suggested VMware principal architect Duncan Epping in Webster’s blog’s comments.

But the environments that are concerned with VIX API may also disallow DHCP, according to Webster. “I think in a lot of environments block this at the switch and insist on static IP addresses.” he wrote.

Some applications for data conversion, PDF generation, and multi-factor authentication in Hill’s environment require static IP definition and therefore wouldn’t be able to use the DHCP workaround, he said.

One financial shop running SRM, South Africa’s Investec Bank, will avoid the VIX issue because its layer 2 domain is stretched, so IP addresses don’t have to be reassigned at all.

“If we do a test we actually isolate the environment completely and our VMs have the same IPs as they would have in production,” wrote Etienne Neethling, who administers SRM for the bank, in an email. “And if we had a real DR [situation], they would [also] stay the same.”

However, this approach comes with its own set of challenges, especially over distance.

A new information supplement from the Payment Card Industry (PCI) Security Standards Council attempts to answer that question. The supplement, produced by a PCI special interest group consisting of more than 30 merchants, vendors and security assessors, identifies the challenges of virtualizing PCI-regulated environments and advises organizations on how to implement virtualization in compliance with the Data Security Standard (DSS).

These virtualization guidelines (PDF) may look good on paper, but many organizations will still encounter roadblocks — surrounding not only PCI compliance, but the virtualization technology itself.

Guidelines focus on hypervisor access

The PCI council’s guidelines say virtualized organizations should constantly monitor the effectiveness of security controls and their ability to respond quickly when a breach occurs. These businesses should also educate personnel on the proper handling of sensitive data and how to recognize security threats, and they should isolate security functions such as network firewalls as well.

The main challenges facing organizations are configuring the hypervisor and deciding who has access to key parts of the infrastructure, said Hemma Prafullchandra, chief technology officer for security vendor HyTrust and a member of the group that created these new guidelines. The hypervisor adds a new attack surface, and with multiple virtual machines (VMs) sitting on it, it provides a single point of access to the entire infrastructure. That complicates access control: The flexibility of VMs and ability to access multiple virtual devices from a single logical location or user makes it difficult to pinpoint roles and access policies.

To overcome these challenges, the guidelines state that “access to the hypervisor be restricted according to least privilege and need to know, and that independent monitoring of all activities be enforced.” They also stress logging, where every instance of attempted infrastructure access (including whether it was granted or denied) should be tracked for security management.

Most importantly, PCI environments must define granular access according to administrators’ specific skills, Prafullchandra said. Network admins, for instance, should only have control of the network. Seasoned virtualization teams that are used to working across multiple technologies may find this separation of duties problematic.

For organizations that handle credit card data but haven’t virtualized yet, the biggest roadblock is the technology itself, Prafullchandra said. Virtualization security technologies are still evolving, and not all organizations are ready to take the plunge, because they’ve seen the implementation and security-assessment challenges that others have had to deal with. The new PCI guidelines, however, should alleviate these fears, Prafullchandra said.

Could PCI ease security?

The new guidelines also bring much-needed standardization. Their appendix of virtualization best practices will help all security assessors use the same testing methods to determine if a company has met PCI DSS requirements, Prafullchandra said. (That has not been the case in the past; as virtualization expert Eric Siebert wrote last year, “the enforcement of PCI DSS requirements is largely open to auditor interpretation.”)

Prafullchandra pointed out that virtualization can even improve security management, because common benefits such as increased server uptime and quick recovery times are especially vital for organizations that handle credit card data. With virtualization, it’s also easier to place compromised VMs in quarantine.

PCI in the cloud

Of course, what everyone wants to know is: What about cloud? The new PCI virtualization guidelines say the risks are still too high to store credit card data in shared hosting or public cloud environments.

“For now, the easiest is not to go cloud,” Prafullchandra said. “If you did, you would be dependent on the card brand and your qualified security assessors.”

Storing cardholder data in a hybrid or public cloud is tricky, because the network layer is shared among multiple merchants, and the organization is dependent on the service provider, Prafullchandra said.

“Typically, that’s where the public cloud starts falling short,” she added. But PCI in the private cloud is doable, she said, because an organization has complete control over the security of its assets.

Traditionally, to protect against malware and viruses, antivirus agents must be placed in each virtual machine (VM). It’s no secret that this model is plagued with problems. Antivirus scans are resource-intensive, and they can cripple host performance if multiple VMs perform scans at the same time. But don’t blame antivirus vendors for this archaic protection method.

“It’s primarily the fault of VMware,” said Eric Siebert, senior systems administrator for Boston Market and regular TechTarget contributor. “It took awhile for VMware to develop a framework that looks inside the VM.”

With the release of vShield Endpoint, VMware tries to combat the resource-utilization problems associated with antivirus software. Endpoint uses a virtual appliance on the host, connected to each VM through a small driver. The driver offloads the scanning and updating processes from the individual VMs to the virtual appliance.

In theory, this arrangement should reduce host-resource utilization problems. Instead of several antivirus agents running full bore, the virtual appliance — acting as a centralized hub — eases the load on the host. (For a closer look at vShield Endpoint, stay tuned for our upcoming series on vShield.)

According to Seibert, Endpoint is great for virtual desktop infrastructure, which already has extra overhead inside each VM. By reducing host-resource demands, Endpoint can also increase VM-to-host ratios, he said.

So far, Trend Micro’s Deep Security is the only antivirus product that taps vShield Endpoint’s capabilities. In a recent study commissioned by the security vendor, Deep Security consistently drew lower CPU, memory and disk I/O, compared to traditional antivirus offerings from McAfee and Symantec. (Granted, it’s a vendor-sponsored survey.)

If anything, these antivirus developments show a promising future for a critically important, but stagnant technology. Perhaps antivirus software for virtualization is coming out of the Dark Ages.

The National Institute of Standards and Technology (NIST) this week released its virtualization security guidelines. The document emphasizes that virtualization involves many moving parts, from the host down to the VM, applications and associated technologies such as storage.

“The security of a full virtualization solution is heavily dependent on the individual security of each of its
components,” the report says.

The NIST virtualization security guidelines focus on these four main areas:

• Hypervisor security: Keep all hypervisors updated and patched per vendors’ recommendations, and restrict access to its management interface. It’s also important to disconnect or disable all unused hardware and services, which can serve as attack vectors.
• Guest OS security: Prompt updates are recommended here as well, as is disconnecting unused virtual hardware. You should also back up virtual drives, following the same policies for physical backups. The guidelines warn, “If a guest OS on a hosted virtualization system is compromised, that guest OS can potentially infect other systems on the same hypervisor.”
• Infrastructure security: Only the guests that use certain storage or networking should have access to that specific hardware.
• Desktop virtualization security: No two desktop virtualization deployments are the same, and determining how to protect virtual desktops depends on their use cases and sensitivity of their workloads.

The NIST virtualization security guidelines go into much more detail in the full report, “Guide to Security for Full Virtualization Technologies” (PDF). For additional resources, check our our server virtualization security best practices guide.

Altor makes Virtual Firewall, which took the bronze medal in the security category of our 2009 Virtualization Products of the Year. Virtual Firewall 4.0, announced in June, added compliance and automation features, as well as deeper integration with VMware’s VMsafe APIs.

The Juniper-Altor acquisition continues the trend of major vendors buying up smaller companies that sell management tools and other point products for virtual infrastructures. It also raises questions about whether security should be a point product for virtualization.

Back in May, CA’s Andi Mann noted that most virtualization security vendors had either broadened their scope or been acquired by larger companies. But he noted: “It seems that only Altor Networks still plays strongly in the pure-play virtualization security space.”

In light of today’s news, Mann asked, “Is pure-play virtualization security over?”

It sure seems like it is. But is that good for customers?

Virtualization may have started off in one particular department, maybe to consolidate a few servers or for a test and dev lab. Nowadays, however, it’s an integral part of many IT infrastructures — which, thanks to the cloud, are no longer constrained to internal data centers. Organizations shouldn’t look at physical security, virtualization security and cloud security separately. So, the thinking goes, products shouldn’t address physical security, virtualization security and cloud security separately.

Juniper hasn’t specifically come out and said what its plans are for the Altor product, but it appears they’re going down this “all-in-one product” road. From Juniper exec Mark Bauhaus: “This acquisition will allow Juniper to extend its market-leading security position by delivering an integrated, highly-scalable security architecture that protects both physical and virtual systems.”

The problem is, this approach brings us back to the whole “one throat to choke” debate. Do some customers like the all-in-one products that vendors provide? Of course. But a lot of other customers like to pick and choose and cobble together their own solutions to meet their needs. These customers’ options, when it comes to the virtualization security component, are dwindling.

Some more reaction to the Juniper-Altor acquisition news:

Jeff Wilson, Infonetics Research: “Dedicated virt sec went a bit quiet…I think Altor and their contemporaries were just a little early…”

Chenxi Wang, Forrester Research: “Altor fetches $95m, a decent exit for them. Good to see virtualization security co. going places.”

Scott Sanchez, ScaleUp: “I like the Juniper acquisition of Altor. Makes strategic sense. Nice when that happens.”

I can also agree. My own conversations since I started covering virtualization have followed a similar path, away from the hypervisor and back toward concerns about the underlying infrastructure. As Saipetch notes, this is particularly true when it comes to data security (as well as virtual backup, which is mostly a separate discussion).  

Survey suggests struggles with virtualization security rise in proportion to percent virtualized

The experiences and discussions @edsai, @Knieriemen and I had are anecdotal, but a recent survey of just under 300 networking pros showed a more scientific correlation between the percentage of virtualization in an environment and the identification of security as a top problem.

“As companies virtualize more of their critical servers and resources, security becomes a greater issue,” wrote a Stephen Brown, product marketing manager for Network Instruments which conducted a survey in an email to The Virtualization Room. “Companies ranking security as their top concern had more than half of their servers (53%) virtualized and one-third of storage (29%) virtualized. This compared to the general virtualized population where 43% had over half of their servers virtualized and 26% had over half of their storage virtualized.”

Standards offer some guidance, but is it enough?

One approach the IT industry is taking to improving virtualization security is the development of standardized and enforced guidelines for how organizations handle sensitive data. . But it’s an arduous task, as demonstrated by the lengthy process that went into the latest PCI DSS 2.0 spec . The latest version of the spec finally acknowledged virtualization as acceptable, after more than two years of development and debate. The spec has yet to be supplemented with virtualization-specific guidance for IT practitioners.

Chris Richter, VP of security products for Savvis, said there are generally two “schools of thought” among security auditors when it comes virtualization. One holds that today’s virtual security controls are adequate for use in closely regulated environments. The other maintains that today’s procedures are not yet adequate for validating e virtual security controls and that there are hypervisor exploits of which we are not yet aware.

Thus, even with the general blessing of virtualization in DSS 2.0, whether virtual environments pass muster is still largely left up to individual auditors, who remain divided on whether virtual environments can truly be secured at this stage of their development.

Meanwhile, Edward Haletky, CEO and a virtualization security, SMB and cloud analyst for the Virtualization Practice, also points out that other regulations, like the Health Insurance Portability and Accountability Act (HIPAA), which focus on keeping sensitive data confidential (i.e., encrypted), and which haven’t been fully brought to bear in the virtual world.

“Right now if you’re a virtualization administator, you can pretty much see all the data. Cloud admins can see data. IT as a service admins control the service catalog and may be able to see data,” Haletky said.

Keeping heavily classified data confidential may require a virtual version of the trusted platform modules (TPMs) that are currently used to authenticate hardware devices by applying cryptographic hashes that ensure the software running on them has not changed.

‘Virtual TPMs’, as well as data encryption that can be applied more granularly at the level of virtual disks and memory, rather than to whole physical disks, would go a long way toward improving enterprise virtual security overall, Haletky says. “We need data confidentiality enforced at the VM level through encryption, and we’re not there yet.”

One of the most important aspects of this new standard for virtualization pros is how specifically the PCI DSS requirements will address server virtualization. Previous versions of the standard have specified that payment card information must be kept separate from general corporate data–but what exactly does “separate” mean?

More importantly, in a converged virtual environment, are virtual security measures like VMware’s VM Safe and vShield products enough to pass muster as separating PCI infrastructure from the rest of the network? Until detailed guidance on server virtualization is made an official part of the standard, it remains up to the auditor — and auditors’ attitudes toward and knowledge of how virtualization works vary widely.

As far as the general standard goes, the following information has been made available publicly by the PCI Security Council in a note to press this morning:

• Virtual technologies have been incorporated into the DSS definition of system components, and also into requirement 2.2.1, which was updated to illustrate how the intent of “one primary function per server” could be applied to a virtual server environment. (Ed. note: Documentation on this will be available on the PCI Council’s website at noon ET today.)
• If you’re working in a virtual environment, there are considerations that people must look into. The standards are all based on system components, so the technical implementation of the specific virtualized environment will determine what needs to be reviewed.
• The new standard aligns physical with virtual: when we say system component, when you have virtualization environment, that language was previously open for interpretation. Now it encompasses virtual environments.
• Future detailed guidance will come from the Virtualization SIG

So, huzzah, virtual servers are now officially “in scope” for PCI. I understand this to mean that the most persnickety auditors will no longer be allowed to just declare any kind of virtualization anaethema for PCI compliance. But that last bullet point is the really important part for anyone trying to actually run a compliant virtualization environment, and, of course, that’s where things get pretty loosey-goosey.

A PCI virtualization special interest group (VirtSIG) created a draft document offering a reference architecture for PCI-compliant virtualization last October, which has been relied upon by some auditors as a guide to evaluating virtualized infrastructure. Another version of this document – which, based on information available at this point, could conceivably remain a non-binding draft, or become officially ratified and part of the standard — is slated to be released…later.

There are indications VMware and some partners will put out a whitepaper with a reference architecture for PCI and virtualization next month, but that whitepaper is not tantamount to the official PCI SIG’s guidance, just the vendors’ interpretations. Sources say last October’s document is the latest draft that’s widely available, but that major changes have been made to more recent versions.

So, to summarize: we now know virtual machines are not inherently contradictory to the idea of separate infrastructure for PCI compliance. But what about VLANs? Virtual switches? Virtual firewalls? And what are the reference configurations for all of the above?

For that, the virtualization market still has to wait.