1. Please slow down a bit. Produce a quality bug-free product and not try to rush out new versions, features and functionality until they are ready. Stop with the experimental features and only put them in the finished product unless they are ready and you are going to fully support them. I know it’s almost impossible to produce 100% bug-free code, especially as your product code grows larger and larger in size, but please catch the major ones that can cause outages for your customers. If you can’t slow down, at least hire more QA personnel and do more public Betas so your customers can help you with this. You can’t afford another mishap like Microsoft is currently experiencing with their Zune music players.

2. On the release of VI4 (or vSphere as you now call it): This should be an exciting upgrade and further distance you from your competitors, but please don’t release it before it’s fully done, polished and tested. I can wait an extra month or two if necessary.

3. Please, no more product name changes. Enough is enough with the name changes! You’re just confusing your customers and complicating things. Instead, get your marketing department to do more to attract new customers, keep your current ones and fight all the HyperV vs. ESX misinformation that Microsoft releases. Also please leave ESX named ESX, I know your marketing department is probably itching to change it so something like vHypervisor but resist and leave it as ESX. (For those who don’t know ESX stands for Elastic Sky X which was the name used in the development of the original version.)

4. More competitive pricing. You have lots of competition now and the hypervisor is becoming commoditized. Giving away ESXi for free was a good start. Why not give ESX away for free also and sell all the advanced features as add-ons? You also have plenty of automation and management products that you can sell to complement it. Also, please reduce the price of Workstation. It’s too expensive for many. You’d probably sell a lot more if you reduced the price so it was close to the price of Fusion.

5. On VMworld presentations: Please go back to releasing these to non-attendees after the show ends as you did in previous years. Not everyone can afford to go to it and the information in the sessions would be valuable to both your current and potential customers. It’s to your benefit to educate your customers and provide as much information to them as possible. At the very least, allow people to purchase a subscription to the sessions so they can access them right away after the show ends.

6. Relax the VMware Certified Professional (VCP) certification requirements. I shouldn’t have to take a class to become a VCP, if I have the knowledge and experience to pass the VCP exam that should be enough. Many qualified people can’t afford to take a class just so they can take the test.

Well VMware, I hope 2009 is a very good year for you, I look forward to the release of vSphere and any other great things that you will deliver to us in the upcoming year.

Best,

A VMware aficionado

As stated within the VMware KB article 1000761 it is possible to restore ESX to identical hardware; however, you need to reinstall ESX first and restore the data you backed up while making changes to how the system boots, else the Universally Unique Identifier (UUID) written by the installation will not work anymore as you have overwritten the data from your backup.

This method will restore everything effectively to identical hardware, however if you want to use new hardware, perhaps with different PCI devices, then the restoration would fail to properly configure the new devices. It may even fail to properly configure NICs if there are any IRQ differences between the supposed identical hardware.

So in these cases you would have to at least verify the configuration and fix anything that was broken. This could lead to a set of unknowns from a security perspective. You are after all trusting the backup was restored properly and if it was not, then you could end up with security issues. So the verification step would have to be extremely well documented.

It is far easier to reinstall VMware ESX to the hardware and to use a either a installation document,  kickstart, or other type of script to configure all the devices for you using either the Remote CLI or the VMware ESX CLI.

When restoring VMware ESX or VMware ESXi the best tool to have will be very good installation documentation that is easy to follow and has graphics and text for every step of the configuration.  These documents could be reviewed for security concerns, and used to derive the scripts that could do the work for you.

Systems - What products are supported for installations for ESX and ESXi platforms
Storage and SAN – Allows searches for partners and their products for ESX and ESXi-based storage devices.
I/O devices – Has brand information for supported HBAs, RAID controllers, and SCSI adapters.
VMware View – Lists supported connecting devices to the new virtual desktop product.

This new hardware compatibility guide search website also has direct links to all of the relevant other configuration information. This includes resources on CPU configuration for VMotion, supported guest operating systems, as well as my trusty PDF documents that are available as a traditional download.

More information on the new tool can be found in the online help section of the hardware compatibility guide website.

Top Virtualization Security Links

The following links are just a sample of what is at the aforementioned site and should be read in order for those interested in securing your VMware Virtual Infrastructure and unfamiliar with VMware ESX, at the same time these are great references for the experienced administrator.

• VMware Virtual Networking Concepts
• VMware ESX Server 3: 802.1Q VLAN Solutions
• Security Design of the VMware 3 Architecture
• DMZ Virtualization with VMware Infrastructure
• VMware Infrastructure 3 Hardening
• CISecurity VMware ESX Security Benchmark followed by the CISecurity Linux Benchmark
• DISA STIG (ESX STIG depends on the UNIX STIG)
  • VMware Communities thread on ESX_SRRSecure – Script to allow ESX to pass a DISA Security Readiness Review.
  • UNIX STIG
  • ESX STIG
• Proven Practice: VI3 Security Risk Assessment – Xtravirt.com
• Remote Authentication – Full/Partial AD Integration, Secure LDAP, NIS, …

It is recommended to read as each guide or benchmark as each covers things from a slightly different but useful perspective.

VMware VI:OPs has also launched the Top 100 Virtualization Security Questions. This can also be referenced from the Top Virtualization Security Links sites as well as other blogs of interest.

At this point on the site, there are hardly enough virtual machines to make a quality sample of how the world is using ESX-based VMs. So, I encourage you to upload to the site, and as it matures the quality of the site will improve with more data. Uploading a VM or appliance’s configuration is easy, simply go to the CompareMyVM website and select to Snapshot My VM via a Java applet that will connect to VirtualCenter or the ESX host, or you can click the Contribute My VM button to enter the configuration manually. Once a virtual machine is loaded into the site, it is shown in the list among the other contributed VMs. An example of a VM I uploaded is shown below:

Uploaded VMs can be given information about what applications they run, such as a Windows domain controller, Apache 2.2 web engine or an Exchange mail server. At first glance, you may think that this is simply a way that virtualization admins can put their data on the CompareMyVM site and have it sold as data that says how administrators configure their VMs.

Not so according to Christian Simko, director of marketing and communications from VKernel who states that “The idea is to keep it open and never charge for it… We may eventually publish a best practices guide with data derived from CompareMyVM, but that will also be a freebie.” This is good news, as the site develops, there will surely be value to determine how other administrators are provisioning VMs with similar application inventories.

As the site matures, look for more comparative performance data on the contributed configurations as well. To date, there is only a limited number of virtual machines posted from virtualization users. I’ve put some VMs up there and hopefully you will as well. As more users contribute data, this free service will hopefully provide some valuable comparative information on how people use virtualization across different market segments. Most importantly, this information will be vendor-neutral, so different server hardware as well as various software configurations will be represented on the community site.

Let us take a quick look at the daily operations you may perform within your virtual infrastructure.

• Create/Move/Modify virtual machines (VMs)
• Backup/replicate VMs
• View performance and other data about hosts and VMs
• Install/Reinstall/Update hosts
• Add or remove administrative users from VMs and hosts
• Monitor your VMs and hosts
• etc.

The list is quite endless. Each one of these actions will affect security or be affected by security.

Ease of use, ease of administration is sometimes paramount. This is the age-old debate of security over usability. Do not knock holes in your firewalls, instead, use existing secure protocols to improve the experience. One I use is OpenVPN to access my administrative network workstation using pre-shared keys for encryption. The extra step of starting up the VPN is trivial with the service provided, this gives me secure access to the management network over which the tools can be run.

Affected by Security

There will be at least one more step involved in use of daily tools then you would normally use. Mainly some way to ensure your access to the management tools is secured from sniffing by your non-administrative network. Or the use of different procedures to ensure you have proper auditing in place.

One tool I use is a simple expect script called expectsudo which will allow me to run remote commands on my VMware ESX hosts while retaining logging of all access. To use this tool you must first install the expect RPM onto your host. Then setup sudo to allow access to the appropriate commands.

#!/usr/bin/expect --

set pass [lindex $argv 0]

set timeout 5

spawn /usr/bin/sudo [lrange $argv 1 end]

expect "Password: {send "$passr"; exp_continue

sleep 1

In the above script the first argument will be the password to use. This script can not be run remotely using any form of SSH. For example:
ssh user@hostname expectsudo password esxcfg-vswitch -l
The only drawback to this script is the need to have the password available, instead you can setup pre-shared keys for SSH and setup Sudo to allow the  user to issue certain commands without requiring a password. This form of single sign on relies on the security of the management workstation in use.

Security of your VMware Infrastructure relies on the security of all the management workstations and servers in use. Not just on the hardening of the VMware ESX host.

If you look in these directories the contents of them are exactly the same as the directories that are named the same as your VMFS volumes. So what are these directories? Let’s explain what happens when you create a VMFS volume to find out.

When creating VMFS volumes you are prompted to name them. This name is not what the ESX host uses to reference the volume; it is purely a friendly name to make it easier for the user to identify the volume. The ESX host actually uses a unique identifier called a Universal Unique ID (UUID) to reference the volume. The name you specify when you create a VMFS volume is a user-defined device name which is a symbolic link to the UUID of the VMFS volume. This is done to solve the problem of changing the device name, when you change the volume name you are only changing the user-defined device name and not the UUID of the volume. So when you look in your /vmfs/volumes directory you will see both a UUID, ie. 4404e8b4-bcfd52fc-1e4b-0017a4a91076 and the symbolic link, i.e. ServerA-Local. Changing to the symbolic link name by using the cd command or clicking on it in WinSCP simply takes you to the UUID directory. You can see the relationship between symbolic links and UUID’s by using the ls –l command inside the service console as shown below.

Additionally you can see the UUID of a volume in the VMware Infrastructure Client by selecting a volume in the Configuration, Storage section and then looking in the Details pane at the Location field. It’s definitely a lot easier to remember the volumes friendly name then it’s long UUID which is why symbolic links are used with ESX.

P2V works by imaging the physical host within the DMZ and transferring that image to the administrative/management network attached to the service console (management appliance) of the VMware ESX(i) host. This in essence crosses security zones and could connect the hostile DMZ to the ‘in need of protection’ virtualization management network. Access to this network from the DMZ could be disastrous.

One solution is to perform the P2V migration in stages.

1. Create the DMZ virtual network within your virtual infrastructure.
2. Get your security team to bless a laptop/workstation for work within the DMZ. Ensure this laptop/workstation has enough removable storage to contain the resultant VM or VMs of the physical servers you wish to convert.Use your  P2V tool to convert the VM and store it on the removable media.
3. Disconnect the removable media and bring it to your secure administrative network.
4. Connect the removable media to a workstation within the administrative network. Ensure this connection is read-only for the moment if possible.
5. Virus Scan the removable media, but note a VMDK can give false positives; you are really looking for anything that may be hidden from view.
6. Use VMware Converter to import the VM or VMs into the virtual infrastructure ensuring they are connected to the proper virtual network.
7. Power on the VM with the network disconnected and fix any issues that are caused by the P2V migration, such as the need to remove hardware agents, and fix anything that needs to be fixed.
8. Reboot the VM with the network connected

The P2V migration is now complete and isolated from the network. The key to this is to only power on the VM once you are within a safe environment and to check for viruses and worms that may live within your DMZ.

VMware touts ESXi as being more secure by having less of an attack footprint, but it is missing the most important feature of modern operating systems: The ability to build a strategy to protect the system from those reaching it and users gaining access to those things to which they are not authorized, currently known in the IT world as Defense in Depth.

Defense in Depth is more than just the availability of a packet filtering firewall, a la VMware ESX’s iptables-based firewall. It is the ability to control when and from where users can log in, and how they can access or view information on the system as well as audit all actions within the system for later perusal or immediate notification of unauthorized access to data or the system. Defense in Depth often starts with the use of a directory service as a centralized management point for all users.

VMware ESXi v3.x is missing all of these capabilities. Directory services are not supported within the VMware ESXi management appliance, there is no ability to audit actions that take place while on the management appliance, there is no control of when or how a user can access the appliance, and most importantly there is no built in firewall.

All of this begs the question: does ESX’s smaller footprint really offer a more secure hypervisor? From the network facing view, its attack surface is limited but not as much as you would expect. What a user can do or access once on the system is also limited, but also not as much as you would expect.

Network daemons almost the same as ESX, minus default SSH

From the network perspective, all the normal daemons that are available for VMware ESX are also available for VMware ESXi: vmware-hostd daemon, cimserver, time daemon, and webAccess.  What is missing by default is SSH access, which most ESXi users enable immediately. The ability to start most other services on the system is also missing. In other words it has nearly the same network daemons running by default that ESX does.

The major difference is that you can no longer log directly into the ESXi box without degrading your security first, in other words without enabling the dropbear SSH server. This does not need to happen and should not according to VMware.

Management of ESXi is performed by either direct access via Remote Command Line Interface (RCLI), the VMware Infrastructure Client (VI Client) and VMware webAccess or by going through VirtualCenter.  Each of these use SSL in order to encrypt and protect all traffic from the workstation and ESXi host. These are the same tools you can use to manage ESX and, as such, share all the weaknesses and strengths. All administrative access via these tools is performed through the vpxuser account which in turn runs many commands as the root user. This is no different than what ESX does. If you go through VirtualCenter, however, you can gain the benefits and disadvantages of using a directory service, but this is not the case when going direct to ESXi.

Possible split-brain authentication

The largest security difference as discussed above is that there is no Defense in Depth, and that once you break the shell by enabling SSH you now run into possible split-brain authentication and authorization that did not exist before. This implies that unprivileged users can gain access to data which they do not own and should not be able to access or even see.

Lastly, since ESXi has no Defense in Depth, its management appliance belongs behind a firewall of its own. This is a step backward in my opinion, and hopefully will be fixed in future releases!