VI3

VMware offers new searchable compatibility for support resources

In October of this year, I mentioned in a prior blog post that VMware updated their storage and compatibility guides to reflect a split of sorts between ESX 3.0.x and ESX 3.5 and ESXi. This is now available as a searchable by product name or hardware vendor for multiple solutions and provides a central resource for all supported configurations. This tool allows for the following search categories:

Systems - What products are supported for installations for ESX and ESXi platforms
Storage and SAN – Allows searches for partners and their products for ESX and ESXi-based storage devices.
I/O devices – Has brand information for supported HBAs, RAID controllers, and SCSI adapters.
VMware View – Lists supported connecting devices to the new virtual desktop product.

This new hardware compatibility guide search website also has direct links to all of the relevant other configuration information. This includes resources on CPU configuration for VMotion, supported guest operating systems, as well as my trusty PDF documents that are available as a traditional download.

More information on the new tool can be found in the online help section of the hardware compatibility guide website.

Focusing on the consolidation ratio in planning virtual environments

Recently, myself and other bloggers as well as the editorial staff at TechTarget were invited to a briefing on Hyper-V. Microsoft is, on all fronts, trying to get Hyper-V and the Microsoft System Center technologies into the hands of administrators, and to some extent it is working.

During this briefing there was some discussion about the memory overcommit technology of VMware’s ESX, not available in Hyper-V. This feature pretty much answers my question on where I put my virtualization environment. Microsoft said IT managers and administrators won’t make their platform decisions based on one specific technology feature such as memory overcommit. I could not make sense of that answer, so I chewed on it a little and came up with this.

Managers and administrators do make their platform decisions on factors like the consolidation ratio, which is the number of guest virtual machines that are running on one physical host. In the case of VMware versus Hyper-V, there is no question which is going to perform better in achieving a higher density of guests per host. This ratio is made almost entirely possible by the memory overcommit technology that ESX has. To that end, how important is a consolidation ratio? Granted there are many factors that go into the workload and various planning tools that can be used to map out the virtual landscape, but for me, I can’t think of a more important measurable in architecting a virtual environment.

Understanding VMware ESX security zones

There is confusion around VMware ESX with regards to security zones. On one hand VMware ESX is a single multi-homed physical machine. On the other hand, it contains multiple security zones. We need to look within the physical to properly understand security zones within VMware ESX and ESXi.

Security zones within VMware ESX are either based on networks employed (Management, VMotion, Storage, DMZ, Production, etc.) or it can be based on functionality such as hypervisor, management tool, backup systems, or virtual machine.

Each of these zones reflects a configuration for the VMware ESX host. It also reflects what is being run within the virtual machines as well as what network upon which the VMs and hosts resides.

Network security zones are relatively easy to understand. Each network would be isolated from one another except through well defined gateways and firewalls. The confusion among some security administrators is that they believe since the VMware ESX or ESXi host has multiple adapters that it would therefore act as an undefined gateway between the disparate networks.

This is simply not the case. Unlike other VMware products where networking is done using bridging technology, the virtual switch within the ESX host is similar to any other layer 2 physical switch. The physical NICs within the VMware ESX or ESXi host act as uplink ports between the physical switch and the virtual switch. Whether a virtual NIC used by virtual machines (VMs), or a vmkernel NIC used by the hypervisor, each virtual device connects to a portgroup within a virtual switch then out the physical NICs to the physical switches unless the communication is VM to VM on the same portgroup.

Since there is no way to layer virtual switches or portgroups, the only way to speak across them is by the use of well-defined gateways or firewalls. VMs can act as firewalls and gateways but only if they have more than one virtual NIC. So the rule of no multi-homed VMs is the best place to start unless it is a well defined virtual firewall or gateway.

The other definition of security zones is based on the roles used within the virtual infrastructure as well. There are three well-defined roles within the virtual infrastructure: hypervisor, virtual machine, and management tool.

The hypervisor controls everything so access to this must be limited to only the service console or management appliance, data stores used as VM storage, or VMware VMotion interfaces. Two of these are through vmkernel NIC used by the hypervisor as described previously: data stores for VM Storage and VMware VMotion Interfaces. Anything that accesses the hypervisor must be protected from those items that should not be able to access the hypervisor.

A virtualization management tool is another role within the system, these tools interact with the service console or management appliance for the virtualization host which will indirectly interact with the hypervisor. This indirect interaction is the key to handling this role. Virtualization management tools such as VMware VirtualCenter, HP SIM VMM plugin, VMware Lab Manager, etc. should be firewalled from other networks as they have indirect access to the hypervisor.

Backup systems act as another role within a virtual environment. These systems can interact directly with the storage devices attached to virtualization hosts through VMware VCB, or through the service console or management appliance just like virtualization management tools. This can also lead to indirect interaction with the hypervisor. Due to the direct and indirect access these systems should be firewalled as well from other networks.

The last role is that of a standard virtual machine. These virtual machines could live on any network but they should not be able to directly access the service console or management appliance of thse virtualization host in addition they should not be able to directly attach to the storage device ued by the virtualization hosts. Virtual machines are considered to be hostile to the virtualization host and should be treated as such.

Security zones depend on the security roles each component of your virtual environment directly or indirectly interacts. These few tips will aid in designing a secure environment.

Security outside the box

Virtualization security depends on more than securing the virtualization host and hypervisor. It depends on everything that touches your virtualization host, directly or indirectly, also being secure.

Let us take a quick look at the daily operations you may perform within your virtual infrastructure.

• Create/Move/Modify virtual machines (VMs)
• Backup/replicate VMs
• View performance and other data about hosts and VMs
• Install/Reinstall/Update hosts
• Add or remove administrative users from VMs and hosts
• Monitor your VMs and hosts
• etc.

The list is quite endless. Each one of these actions will affect security or be affected by security.

Ease of use, ease of administration is sometimes paramount. This is the age-old debate of security over usability. Do not knock holes in your firewalls, instead, use existing secure protocols to improve the experience. One I use is OpenVPN to access my administrative network workstation using pre-shared keys for encryption. The extra step of starting up the VPN is trivial with the service provided, this gives me secure access to the management network over which the tools can be run.

Affected by Security

There will be at least one more step involved in use of daily tools then you would normally use. Mainly some way to ensure your access to the management tools is secured from sniffing by your non-administrative network. Or the use of different procedures to ensure you have proper auditing in place.

One tool I use is a simple expect script called expectsudo which will allow me to run remote commands on my VMware ESX hosts while retaining logging of all access. To use this tool you must first install the expect RPM onto your host. Then setup sudo to allow access to the appropriate commands.

#!/usr/bin/expect --

set pass [lindex $argv 0]

set timeout 5

spawn /usr/bin/sudo [lrange $argv 1 end]

expect "Password: {send "$passr"; exp_continue

sleep 1

In the above script the first argument will be the password to use. This script can not be run remotely using any form of SSH. For example:
ssh user@hostname expectsudo password esxcfg-vswitch -l
The only drawback to this script is the need to have the password available, instead you can setup pre-shared keys for SSH and setup Sudo to allow the  user to issue certain commands without requiring a password. This form of single sign on relies on the security of the management workstation in use.

Security of your VMware Infrastructure relies on the security of all the management workstations and servers in use. Not just on the hardening of the VMware ESX host.

Secure method to P2V across security zones

A common VMware Communities question is how to P2V or convert a system from within a demilitarized zone (DMZ) to a virtual machine (VM) running within an ESX host that will be part of the DMZ virtual network.

P2V works by imaging the physical host within the DMZ and transferring that image to the administrative/management network attached to the service console (management appliance) of the VMware ESX(i) host. This in essence crosses security zones and could connect the hostile DMZ to the ‘in need of protection’ virtualization management network. Access to this network from the DMZ could be disastrous.

One solution is to perform the P2V migration in stages.

1. Create the DMZ virtual network within your virtual infrastructure.
2. Get your security team to bless a laptop/workstation for work within the DMZ. Ensure this laptop/workstation has enough removable storage to contain the resultant VM or VMs of the physical servers you wish to convert.Use your  P2V tool to convert the VM and store it on the removable media.
3. Disconnect the removable media and bring it to your secure administrative network.
4. Connect the removable media to a workstation within the administrative network. Ensure this connection is read-only for the moment if possible.
5. Virus Scan the removable media, but note a VMDK can give false positives; you are really looking for anything that may be hidden from view.
6. Use VMware Converter to import the VM or VMs into the virtual infrastructure ensuring they are connected to the proper virtual network.
7. Power on the VM with the network disconnected and fix any issues that are caused by the P2V migration, such as the need to remove hardware agents, and fix anything that needs to be fixed.
8. Reboot the VM with the network connected

The P2V migration is now complete and isolated from the network. The key to this is to only power on the VM once you are within a safe environment and to check for viruses and worms that may live within your DMZ.

VMware administrator resources

When it comes to finding technical information on VMware products there are a number of obvious sources of information such as the official documentation and books but there are also some not so obvious sources that contain tons of great technical information.

VMware’s Knowledge Base

First there is VMware’s online knowledge base. It continues to amaze me about the volume and quality of information provided in VMware’s knowledge base. Typically many users access vendor knowledge bases as a tool to try and resolve problems in their environment as many known problems are documented there. VMware’s knowledgebase, however, has much more than that and also contains how-to articles, troubleshooting tips, sample configurations, best practices and so on. Here’s just a sample of some of the great content published there in the last 10 days:

• Configuring the speed and duplex of an ESX Server host network adapter
• Cisco Discovery Protocol (CDP) network information via command line and VirtualCenter on an ESX host
• Sample Configuration - ESX connecting to physical switch via VLAN access mode. External Switch VLAN Tagging (EST Mode)
• Troubleshooting SCSI Reservation failures on Virtual Infrastructure 3 and 3.5
• Advanced Configuration options for VMware High Availability
• Advisory for advanced VMkernel parameter NFS.LockDisabled
• Sample Configuration - Network Load Balancing (NLB) Multicast mode over routed subnet - Cisco Switch Static ARP Configuration
• VLAN Configuration on Virtual Switch, Physical Switch, and Virtual Machines - ESX 3.X
• Diagnosing an ESX Server that is Disconnected or Not Responding in VirtualCenter

They even have a knowledge base article on how to search the knowledge base. I encourage you to frequently check out the knowledge base as new documents are added daily. You can also subscribe to a weekly digest by editing your VMware account preferences and setting up your email subscriptions.

VI:OPS

Next there is a new website VMware launched called VI:OPS that contains many documented proven practices from both VMware itself and customers. The information is organized into several different zones: Strategy, Applications, Security, Management and Availability. Already there are dozens of great documents there that provide some fabulous information.

Another great resource is the fabulous technical and white papers that VMware publishes. There are over 200 of them available on a wide variety of subject matters.

There is a wealth of information available on the VMworld website from the all the great sessions each year. Only attendees can access the current year’s sessions but anyone can register for a free account and access previous years.

VMTN

You may have used the VMTN community forums for posting technical questions but did you know there is a whole separate documents section that users can create technical documents to share their information and tips with other users. There are hundreds of documents already created there with some great tips from both other users and VMware employees.

In addition to all these great VMware resources be sure and check out all the great tips and information on both SearchServerVirtualization.com and SearchVMware.com.

Updating VMware ESX? Don’t forget VMware Tools

While VMware Tools guest OS upgrades are an inconvenience in the life of a VMware ESX and VMware Infrastructure 3 (VI3) administrator, they need to be done. VMware Tools is a set of drivers designed to greatly improve console interaction, but also includes drivers such as disk and network devices for most operating systems. As new versions of ESX are available, the guest operating system will need to be upgraded. Having the VMware Tools running with an old version will give basic functionality, but will need to be upgraded eventually on the virtual machine (VM).

The good news there are a few ways to approach getting tools out to the guest VM. Here are a few from my bag of tricks that you can use:

Install through Windows Group Policy: For 100% virtualized Windows environments within an organizational unit (OU), this is easy for the one-time installation. This can be applied to a computer accounts OU that contain VMware-based virtual machines.

Interactive or automatic install on guest console: This option can allow the tools to be upgraded from the VMware Tools .ISO that’s mounted and run on the guest VM. Note that the automatic upgrade may require an automatic reboot of the VM. The option to install or upgrade tools is shown below:

Run VMware Tools upgrade from a UNC path. If you copy the contents of the mounted .ISO of VMware Tools to a shared path on the network, Windows-based guest VMs can run the install from that central location. This can save the work of adding permissions for junior administrators to have device mount permissions in VMware, and you can control the access entirely in the operating system.

One caveat is that installations or upgrades of VMware Tools usually cause an interruption in network connectivity by updating the network driver. Depending on the mechanism accessed and its configuration for disconnected sessions, the install may be cancelled. This can happen in certain Remote Desktop for Windows systems in particular during installations of VMware Tools.

Lastly, this challenge of updating VMware Tools is not unique to VMware ESX Server. The same challenge exists on VMware Workstation, VMware Server as well as other products, but may be solved in the same fashion.

Replacing a VMware ESX SAN

Replacing or upgrading a SAN is no trivial task. There are a few tried-and-true steps to take when replacing a SAN which I’ll outline in this blog post, including a key step to the process that will ensure a successful switch.

I recently upgraded from an HP MSA 1000 to an IBM DS3400 because I wanted to improve performance and lower my overall energy costs.  One of the reasons I decided to replace my old SAN is because it is much cheaper to have a single 2U device running than the three devices for the old SAN. In addition, I dropped from 42 drives to 12 drives with more storage. Minimally my SAN power costs should drop to just 1/3 the original. I also have gone from 2.5 TB to 3 TBs of storage. Not a huge increase in storage capability.

I will know next month if my energy cost reductions have been realized and will report back then.

The steps for replacing a SAN are not all that tricky, but there was a single gotcha that could be avoided with careful planning. To replace a SAN, follow these steps:

1. Plug in the new SAN to your existing fabric. Luckily I had a pair of unused fibre connections and Gbics available else this would have been another expense and a delay until the cables and Gbics arrived.

2. Find a system on which to install the management console. For the IBM DS3400 I chose my VirtualCenter and VMware Consolidated Backup (VCB) server to be the management console for the SAN. There are two methods to manage the IBM DS3400: in-band or over the fibre channel fabric, or out of band using Ethernet — even a VM would suffice given networking is connected to the SAN. Software exists for both 64-bit Linux and Microsoft Windows.

3. Create the LUNs on the new SAN. This is a good chance to correct any problems you may have with the LUN configuration on the old SAN. I did a one-to-one mapping, except I slightly increased the size of the LUNs.

4. Present the LUNs to your VMware ESX host(s) and VCB server(s).

5. Rescan the storage adapters for new LUNs using the VMware Infrastructure Client (VI Client) for the first VMware ESX host. Once this is completed, you can then add as many Virtual Machine File Systems (VMFSs) as required.

6. Rescan the storage adapters for new LUNs and VMFSs using the VI Client on all the other ESX hosts.

7. Employ Storage VMotion via the VI Client to migrate VMs from one LUN to another LUN. This works if you have the patience to move all the VMs one by one. If not you can employ other measures. If you do this, however, you will end up having to edit the VMX files for each VM migrated to change the location of the virtual disk files. There are scripts to do this for you as well. This second option, however, also requires you to power off all VMs. Use of Storage VMotion does not require any VM downtime. Be sure to move all files from the LUNs in use.

8. For a LUN with an RDM (mine was a Linux file server), use Storage VMotion to move any VMDKs related to the VM. Then map the new RDM to the VM. You will have to reboot the VM to complete. Then create a new filesystem on the new RDM mount the file system. Then you must copy all the files from the old RDM to the new RDM. I used the following command to complete this task to copy all files from /files to /files2.

• cd /files; rsync -ravlpog * /files2

9. Then I modified the mount point for /files within /etc/fstab to be the correct new location. Finally I powered off the VM, deleted the old RDM from the VM and powered on the VM picking up the new data.

Here is the gotcha. I missed it, but it will be extremely useful for you (and me) going forward: Remove the old SAN’s LUNs from each VMware ESX host. If you miss this step when you finally disconnect the old SAN, the ESX hosts will go into a state of constantly attempting to failover the old LUNs. This will spew massive failures into the log files. If this happens there is no recourse but to reboot the VMware ESX hosts.

Now the SAN has been replaced. With the exception of dealing with any RDMs, it is possible to migrate to a new SAN without any downtime.

VMware ESXi security review: Firewall, please

VMware ESXi may have a smaller footprint than VMware ESX, but the pro-security theory behind the skinny ESX version may be defunct given the lack of ability to create a Defense in Depth strategy around the hypervisor. As is, I suggest you consider ESXi a safe hypervisor only when behind a firewall.

VMware touts ESXi as being more secure by having less of an attack footprint, but it is missing the most important feature of modern operating systems: The ability to build a strategy to protect the system from those reaching it and users gaining access to those things to which they are not authorized, currently known in the IT world as Defense in Depth.

Defense in Depth is more than just the availability of a packet filtering firewall, a la VMware ESX’s iptables-based firewall. It is the ability to control when and from where users can log in, and how they can access or view information on the system as well as audit all actions within the system for later perusal or immediate notification of unauthorized access to data or the system. Defense in Depth often starts with the use of a directory service as a centralized management point for all users.

VMware ESXi v3.x is missing all of these capabilities. Directory services are not supported within the VMware ESXi management appliance, there is no ability to audit actions that take place while on the management appliance, there is no control of when or how a user can access the appliance, and most importantly there is no built in firewall.

All of this begs the question: does ESX’s smaller footprint really offer a more secure hypervisor? From the network facing view, its attack surface is limited but not as much as you would expect. What a user can do or access once on the system is also limited, but also not as much as you would expect.

Network daemons almost the same as ESX, minus default SSH

From the network perspective, all the normal daemons that are available for VMware ESX are also available for VMware ESXi: vmware-hostd daemon, cimserver, time daemon, and webAccess.  What is missing by default is SSH access, which most ESXi users enable immediately. The ability to start most other services on the system is also missing. In other words it has nearly the same network daemons running by default that ESX does.

The major difference is that you can no longer log directly into the ESXi box without degrading your security first, in other words without enabling the dropbear SSH server. This does not need to happen and should not according to VMware.

Management of ESXi is performed by either direct access via Remote Command Line Interface (RCLI), the VMware Infrastructure Client (VI Client) and VMware webAccess or by going through VirtualCenter.  Each of these use SSL in order to encrypt and protect all traffic from the workstation and ESXi host. These are the same tools you can use to manage ESX and, as such, share all the weaknesses and strengths. All administrative access via these tools is performed through the vpxuser account which in turn runs many commands as the root user. This is no different than what ESX does. If you go through VirtualCenter, however, you can gain the benefits and disadvantages of using a directory service, but this is not the case when going direct to ESXi.

Possible split-brain authentication

The largest security difference as discussed above is that there is no Defense in Depth, and that once you break the shell by enabling SSH you now run into possible split-brain authentication and authorization that did not exist before. This implies that unprivileged users can gain access to data which they do not own and should not be able to access or even see.

Lastly, since ESXi has no Defense in Depth, its management appliance belongs behind a firewall of its own. This is a step backward in my opinion, and hopefully will be fixed in future releases!

VMware adds another Windows user to its customer list

VMware, Inc. announced last week that another Windows customer is using VMware Infrastructure 3 (VI3) instead of Microsoft Hyper-V to consolidate servers and reduce costs.

Independence Blue Cross (IBC), the largest health insurer in Philadelphia, has grown quickly in recent years, and their  computing demands and costs have grown along with its business. Physical server sprawl and increasing power consumption has plagued the hospital and the cost of acquiring and managing new hardware was growing out of control, VMware reported.

To reverse these issues, IBC turned to virtualization. The company looked at Microsoft Hyper-V, but ultimately chose VMware because “it offered a more complete solution and robust tool set, rather than simply a hypervisor,” VMware’s spokesperson said on behalf of the customer. Another plus for VMware was that is offers VMotion to live migrate virtual machines (VM), which Micrsoft’s Hyper-V product won’t offer until the next version, as well as high availability,  resource pooling, manageability and automation, VMware said.

So far, IBC’s Windows application environment is approximately 70% virtualized, including applications like Active Directory, Exchange, SharePoint and SQL Server, PeopleSoft and Oracle 9i. There are 386 VMs are running on 48 physical hosts, and CPU utilization has increased from 5% to 75%, VMware reported.

Michael Garber, director of distributed infrastructure, at IBC, stated in the release that VI3 paid for itself in less than 16 months and helped IBC avoid more than $1 million in hardware costs.

VMware currently has over 3,000 hospitals on its list of customers, according to VMware.

VMware release lots of customer case studies to show the world how great they are, but when they announce Windows users as customers, Microsoft Hyper-V takes a bullet.  Hyper-V is built right in to Windows Server 2008, so why wouldn’t a Windows user just virtualize with Hyper-V? That’s Microsoft’s argument, and it looks like people aren’t buying it.

There is a ton of speculation on whether Hyper-V will be able to surpass VMware in the virtualization market, but I haven’t seen anything from Microsoft (like Hyper-V customers!) signaling that possibility.