Virtualization Pro


February 9, 2009  3:39 PM

Using live CDs with virtual machines

Eric Siebert Eric Siebert Profile: Eric Siebert

Live CDs are bootable CD/DVD images that contain an operating system that you can boot from to perform various tasks on a server. When you boot from a live CD, the operating system from the server does not load. Instead the live CD operating system loads, allowing you access to the server’s file system. There are hundreds of free live CD’s available that you can download and use, and they are particularly useful for troubleshooting or repairing a problem with a server.

Most of these live CDs use one of the many Linux distributions as there is no cost for using them and they are very compact. There are also some Windows versions that use a special version of Windows called Windows PE. While physical servers usually rely on a physical CD/DVD which must be burned and placed in the server’s CD/DVD-ROM drive, virtual machines can make use of ISO images which are image files that can be directly mounted to a VM’s CD-ROM.

Live CDs have a wide range of functionality. It’s handy to have a good selection of them available. So when would you want to use a live CD?

•When you have a server that has been infected with a virus or spyware. Booting from a live CD will allow you to run scanning utilities to remove the infection without the server being active.
• When you have a corrupt operating system that will not boot or that crashes while booting. Booting from a live CD will allow you to repair corrupted files or configuration files.
• When you’re installing an application that renders a system unusable. Booting from a live CD will allow you to disable or uninstall the application.
• When you’ve forgotten the administrator password to log in to your server. Booting from a live CD can allow you to crack or reset it.

Almost all live CD images are available as an ISO download which is ready to use on your virtual machines. You simply edit the settings for the virtual machine and select the CD/DVD-ROM drive and map it to either your client CD or a host data store where you can select the ISO file. It’s a good idea to have an ISO data store available to all your hosts that contains the ISO images so you can use them when needed.

As mentioned, there are many live CD images that you can download and use, so to get you started I’ve compiled a list of some of the best ones that will make a good addition to your ISO library.

Ultimate Boot CD – A Swiss Army knife boot CD which has many diagnostic, hard disk and file system tools as well as antivirus and network tools.
Ultimate Boot CD for Windows – A Windows version of the Ultimate Boot CD with a huge array of tools including many antivirus and antispyware, disk, file, security, password reset, network tools and much more.
Knoppix – A Linux Live CD that contains many tools and utilities.
Backtrack – A huge selection of security tools that you can use to scan for vulnerabilities of your VMs.
Kaspersky Resce CD – A Live CD with anti-virus scanner to clean infected systems.
Gnome Partition Editor – A hard disk partition utility that lets you do all sorts of things with disk partitions and is particularly useful for resizing existing partitions on your VM after you have increased the size with vmkfstools or the VMware Infrastructure Client.
Ophcrack – A utility that will crack forgotten Windows passwords.
System Rescue CD – A multi-purpose rescue CD that includes partition editors, file system utilities, password recovery and much more.

These are just a few of the live CDs available. If you would like to build your own live CD based on Windows PE check out the Bart PE website for more info. You can also check out the live CD list website for a huge list of available live CD’s.

February 5, 2009  7:59 PM

New VMware ESX v3.5 U3 vulnerabilities

Texiwill Edward Haletky Profile: Texiwill

There is a new vulnerability in VMware ESX v3.5 U3 with all the patches that has just come to light. VMware has been made aware of this issue, and will fix it sometime in the near future. This bug relates to world writable directories on the VMware ESX service console.

This is not a huge issue as long as your VMware ESX service console is properly protected, but you may want to be concerned if it is not. The vulnerability allows any one who can access the VMware ESX host to write anything to these directories and could cause a disk to fill up or something worse to happen.  The remediation is quite simple.

chmod 755 /var/lib/pegasus /var/lib/pegasus/trace

This is a simple oversight that could lead to something possibly dangerous. However it is also easily made right. This is one of the reasons that is you are using a security assessment script that you run after every patch or update.

The other issue is more a historical item that could lead to a possible security issue if a malicious user does have access to the service console once more. This is the ability to run code as root possibly within the hypervisor using the vmkload_app and vmware-vmx commands. These programs are set up with their setuid bits which allows a normal user to run these as the super user. The reason these are set up this way is more an issue of history.

In VMware ESX 2.5.x days, it was possible for a normal user to run VMs. With VMware ESX v3.x this functionality was dropped from any of the management tools, yet the capability was left in. The solution is to remove the setuid bits so that these commands can only be run as the root user, which is their normal method of operation.

chmod u-s /usr/sbin/vmware-authd /usr/lib/vmware/{bin,bin-debug}/{vmkload_app,vmware-vmx}

It is interesting that while the DISA UNIX STIG and CISecurity CIS-CAT for RHEL found these issues, the TripWire ConfigCheck tool did not. This inconsistency has been reported to Tripwire as well, and they will work with VMware for a possible VMware Hardening Guide modification. TripWire ConfigCheck goes so far as to warn you that these files have had their setuid bits removed, when it should check to see if they are missing. It is a contradiction.

Making these changes will add to your security stance but not harden your system 100%. In my case, I never trust just one hardening guide as they often over look things that the others do not. I try to make my systems pass all the guidelines available and if there are inconsistencies between them, I document those decisions as well. Note that there are several inconsistencies in each guide.

These slight changes are also reasons you want to redo security assessments after you patch or update your VMware ESX hosts!


February 5, 2009  5:35 PM

Avoid disk space problems; monitor your snapshots

Eric Siebert Eric Siebert Profile: Eric Siebert

In a VMware snapshot series I wrote a few months ago (How VMware snapshots work, Deleting virtual machine snapshots without wasting disk space and Troubleshooting VMware snapshots), I listed a few methods for finding snapshots that are running in your environment. I thought I would expand on those methods and also offer some new methods for finding snapshots. Snapshots are a handy tool but leaving them running for any longer then necessary is not a good idea, (read the snapshot series for an explanation).

VMware’s vCenter Server does not provide a centralized mechanism for managing snapshots so subsequently you must resort to other methods if you want to find them. Let’s review the various methods that you can use to find snapshots. If you know how to find them, you’ll be aware of snapshots that are still running and won’t be (as) surprised later when your volumes run out of disk space.

The first method is fairly simple and relies on the find command that is part of the VMware ESX service console. To use it, log in to the service console, switch to your VMFS volume directory and type the following command. The command will find all files that have ‘delta’ in their file names. Deltas indicate snapshot files.

find –iname “*-delta.vmdk”

Or, to find orphaned snapshots that have not been modified in a number of days, type:

find –iname “*-delta.vmdk” –mtime +7 ls

The second method requires one of the free Perl scripts (SnapHunter or SnapAlert) that were written by VMware users. This method requires use of a small Perl script which can be run inside the ESX service console or with the remote command-line utility (Remote CLI) and will search for snapshots and create a report. (You have the option to email the report.) To use these scripts you copy it to the service console or the workstation/appliance running the Remote CLI, modify the script with your vCenter or ESX server name and log in credentials, and then run them.

I also listed a third method that used a VB script to query the vCenter Server’s VPX_SNAPSHOT table for active snapshots. While this method works, it will only show snapshots that vCenter Server is aware of and may not show orphaned snapshots or those taken outside of vCenter Server. Also, since the database design is subject to change with any vCenter Server upgrades, this may break any existing SQL queries. So I’d like to recommend a different method instead which involves using PowerShell and the VMware Infrastructure Toolkit to query for snapshots. You can use simple PowerShell commands to check for snapshots. There are a few sample scripts that you can use. You can find them at the following URLS:

http://www.peetersonline.nl/index.php/vmware/powershell-oneliner-4/
http://communities.vmware.com/docs/DOC-6980

Lastly, there are a few more methods that you can take advantage of by using a reporting or search application. Check out the freely available RV Tools or vKernel’s SearchMyVM. Both enable you to search and report on your virtual environment. These free tools are a good addition to any VMware environment. RV Tools is a .NET application that installs on a workstation and SearchMyVM is a virtual appliance that you can download and import to an ESX host.

Whichever method you choose, you should make sure to periodically look at the snapshots running in your environment. You can automate many of these queries (i.e. using a cron job or Windows task scheduler) so that they run on a scheduled basis. By staying on top of your snapshots, you ensure optimal performance of your hosts and VMs and also avoid disk space problems.


January 30, 2009  9:31 PM

VMware Workstation vs. Sun VirtualBox: Workstation wins

Texiwill Edward Haletky Profile: Texiwill

Sun VirtualBox is not quite a newcomer to the virtualization arena but it is definitely newer than VMware Workstation, and since it is free it is gaining quite a bit of traction. But is VirtualBox worth using? Is it a replacement for VMware Workstation?

I have used both products, and the bottom line is that Sun VirtualBox is a little rough around the edges. While it loads faster, sound capability is lacking. It has a much simpler interface, but at the same time the interface is a little cryptic. It does, however, load virtual disks from VMware Workstation.

To add virtual machines (VMs) to VirtualBox you must first create or add an existing virtual disk to the virtual disk manager. VirtualBox understands VMDKs from VMware Workstation 6.5 as well as those exported using VMware Converter from VMware ESX hosts. Once you have the virtual disk you can then create the VM and launch the VM.

I used Sun VirtualBox to work around the limitations within VMware Workstation’s USB support. Sun VirtualBox’s implementation of USB is much better and supported the device I need to use: LiveScribe SmartPen. When the SmartPen first came out there was no support for 64-bit Vista implementations, so I had to resort to virtual machines to get the 32-bit drivers to work, but they would not work through VMware Workstation on any version. They did work through VirtualBox. So VirtualBox allowed me to save my notes, but since there was no sound, I could not play them back. Eventually, 64-bit Vista drivers came out, all was well and I removed my VirtualBox implementation.

VirtualBox a good simple product if all you need is a spare system to run USB devices that VMware Workstation doesn’t support. If VirtualBox was given sound support it could rival VMware Workstation. Even so it is a very good tool to include in your virtualization toolbox. Simply put, however, VirtualBox is not as robust as VMware.

VMware Workstation provides many more features than the bare bones Sun VirtualBox. These features include embedded video creation, debugging modes for kernel developers, high speed inter-VM communication via VMCI, solid sound and video support, VM teaming, etc. If you need more than a bare bones, no thrills product then VMware Workstation is for you.


January 29, 2009  10:04 PM

Best practices for running Java virtual machines on VMware ESX

Eric Siebert Eric Siebert Profile: Eric Siebert

VMware recently released a new white paper on best practices for running a Java virtual machine (JVM) on an ESX virtual machine. This includes any product that utilizes a JVM, such as Web application servers Websphere, Weblogic and Tomcat. There are many applications that utilize these types of JVM servers. Often times there may be a JVM running inside your application and you may not  know it as the JVM is often renamed to match the application. Example: On vCenter server there is a Windows service called VMware Infrastructure Web Access. It’s actually a Tomcat application server or JVM.

The white paper mentioned the usual recommendations that deal with memory, CPU and disk I/O, but I was surprised to see a whole section on timekeeping (which we will talk about later). JVMs are often memory hogs depending on how you set your minimum and maximum JVM heap size and they also read and write very often. JVMs also tend to be very multi-threaded and the disk I/O will vary based on the type of applications that they are running. A summary of the best practices for each resource is below:

Memory

• As JVMs are very memory intensive, make sure your JVM has access to physical memory at all times by using memory reservations. If a JVM is forced to use its disk swap file (vswp) for memory on an over-committed host its performance will be affected. Set the memory reservation for a VM running a JVM equal to the amount of memory assigned to the VM. The VM won’t be able to utilize the transparent page sharing (TPS) feature to save memory on your host, but you save memory on a VM running a JVM anyway due to the its nature.
• Make sure to give your VM enough memory based on the maximum heap size of your JVM. JVM’s have a minimum and maximum heap size value and will quickly grow to their maximum size. If you do not have enough memory assigned to it then it will not be able to grow and performance will suffer. Check your max size and allocate another 512 MB for Linux virtual machines (VMs) and an additional 1 GB for Windows VMs.
• Use large memory pages if supported by the JVM and OS. See the following hyperlinked white paper for info on how to do this on the OS and how to enable in JVMs use the option –Xlp for IBM JVMs and –XX:+UselLargePages for Sun JVMs.

CPU

• Many JVMs will run well with one vCPU depending on how many garbage collection (GC) threads are running and may not benefit from using virtual symmetric multiprocessing. GC is the process that reclaims memory inside the JVM for objects that are no longer used. Tuning this can be tricky and relies on specific Java resource monitoring tools to see how often GCs are taking place. Check to see how many GC threads are running on your JVM and either adjust this to match the number of vCPUs in the VM or increase the number of vCPUs to match the number of GC threads. Often times its best to start with on vCPU and see how the application performs and then add another vCPU to see if it improves performance.

Disk

• You want to watch the disk I/O of your application running on the JVM for potential bottleneck issues.  A JVM that is waiting to write to disk won’t perform as well as it could.

Timekeeping

As mentioned previously, timekeeping is very important to a JVM. First you should make sure you sync the clock on your VM using either VMware Tools, W32Time or another network time protocol time source. What’s important here is the affect timer interrupts have on a JVM. Higher resolution timer interrupts cause more work to be done by ESX on behalf of the VM then lower resolution timer interrupts. The guest OS determines the timer interrupt of a VM. Most Linux guests allow you to configure the timer interrupt in the OS but Windows guests must rely on a JVM setting. Due to a weird bug in the JVM the –XX:+ForceTimeHighResolution option in the JVM actually has the opposite effect of lowering the time resolution.

For more information check out VMware’s white paper on Java virtual machines and be sure to check out the documents referenced at the end of it.


January 27, 2009  7:44 PM

Three ways to temporarily move VMs off a VMware ESX host

Eric Siebert Eric Siebert Profile: Eric Siebert

I recently came across an ITKE question from a user that had some virtual machines (VMs) on an ESX host. He wanted to move the VMs off of it so he could reconfigure the server with more storage, reinstall ESX and then move the VMs back on to it. Presumably he only had one ESX host with only local disk. There are several options for accomplishing what he wanted to do.

First, you need to find alternate storage to place the VM’s on. This could be a workstation, a Windows or Linux server or an NFS/iSCSI storage device with enough free space to temporarily hold the VMs. Once you find your temporary home, there are several methods that you can use to move the VMs.

Method 1: Create a network-based data store

The first method is the simplest and involves creating a network-based data store on the ESX host. You can do this with a network file system (NFS) or by using a software iSCSI initiator. If you have a Windows or Linux server that supports NFS, you can create a NFS share and then map your ESX host to it to use as a data store by creating a new data store. You can also setup one of the open-source iSCSI appliances like OpenFiler to use as a data store for your ESX host. Once you have a new data store you simply shut down the VMs and use the service console command-line tools to move the VM to the new data store. The vmkfstools –i command will allow you to clone the VMs disk to another VMFS datastore. Then, after you copy the VMs disk to the new data store, you simply add the data store back to the newly rebuilt ESX host and use vmkfstools –i to copy the disk back to it and create a new VM, and tell it to use an existing disk.

Method 2: Use VMware Converter

Another method is to use VMware Converter to migrate the VM to a local workstation or network drive. To do this just install Converter on the VM, choose to convert the local server, and for the destination choose “Other Virtual Machine.” You can then select a network path to store the new VM. It’s a good idea to resize the VM so there is less data to transfer; you can always resize it back to its original size when you move it back to the ESX host.

Method 3: Use a secure copy (SCP) file transfer utility

Finally, the last method is to use one of the secury copy (SCP) file transfer utilities to copy the VM’s files to a local workstation or network drive. FastSCP works best for this. It copies data the fastest because it does not encrypt the data being copied. Then you simply copy the data back to the host after it is rebuilt and either re-register the VM or create a new one and tell it to use an existing disk.

Once you copy virtual disks to an ESX host it’s always a good idea to clone them before using them using vmkfstools as it allocates all the disk space at once which results in a disk with less fragmentation. Tools like FastSCP only allocate space as the data is copied which can result in more fragmentation of the virtual disk file. To clone a disk with vmkfstools just type vmkfstools –i . Once the cloning process completes you can remove the existing disk from the VM and add a new one and browse to the newly created virtual disk.

Whichever method you choose, make sure you have a good backup of the VM before starting.


January 22, 2009  10:07 PM

Manage virtual machines with an iPhone

HannahDrake Hannah Drake Profile: HannahDrake

Tired of running back to your desk to solve the “Help! This virtual machine is acting funny” emails you read on your iPhone or BlackBerry?

Andrew Kutz, developer of the Storage VMotion graphical user interface (GUI) plug-in, recently launched Virtualization Manager Mobile (VMM), another convenience tool to ease virtual machine (VM) management tasks. But this time, there’s a catch: If you want the convenience, you’ll have to pay for it.

The Storage VMotion plug-in, which Kutz gave away for free as an open source tool, has been out for 12 months. As the plug-in boasts 60,000 downloads to date, Kutz thought he’d try to capitalize on his idea this time around.

But will people buy it?

“We’ll see,” Kutz said. Storage VMotion has 60,000 downloads to date, and VMM is off to a good start with 422 downloads to date, 317 since this morning’s beta update and 105 since last Friday’s debut, according to Kutz.

VMM can manage VMware and Citrix virtual machines from their phones. Kutz said that with the beta 2 release, Hyper-V support will come, which will be the final release before the first commercial release.

The idea wasn’t entirely his. Kutz said he got the idea from Slicehost, a company that offers virtual private servers (VPSes). One of the management tools Slicehost offers is an application for that lets customers manage their VPSes via mobile phones. “I thought I’d create it for VMware, and after working with it for a week I realized I could make it work for any hypervisor,” Kutz said.

VMM shows CPU and Memory stats for each VM, and allows you to stop, start, pause and reset.

The first commercially available version should be released by the end of February at the latest, according to Kutz, with beta 2 being the final beta before first full release.

Here are a few quick facts about VMM:

  • VMM is a Web application that is optimized for mobile devices.
  • VMM runs on any mobile device, not just a smartphone or an iPhone. After receiving feedback, Kutz created a “lite” version that is supported by phones that don’t have fully functioning browsers, like BlackBerrys and Windows Mobile devices.
  • The cost for a licensed version: $50 for 25 virtual machines, $100 for 100 virtual machines, $200 for 300 VMs.
  • The latest beta feature is Accidental Touch Protection, which prevents you from accidentally shutting down a VM.
  • VMM will install on any OS.
  • The beta versions will expire on March 3, 2333, and allow you to manage an unlimited number of servers with an unlimited number of VMs.
  • The final version will include a snapshot manager.

So now you don’t only have to worry about losing your laptop. Better keep tabs on your cell phone, too. (Even though VMM requires you to log-in again after a certain amount of time.)


January 21, 2009  7:43 PM

Using vCenter Converter to create VMware ESXi virtual machines

Rick Vanover Rick Vanover Profile: Rick Vanover

Administrators considering VMware ESXi may wonder if vCenter Converter will workfor VMware ESXi conversions. The answer is yes, but there’s a gotcha.

Converting a physical machine into an ESXi virtual machine works best when you use the latest version of VMware Converter, version 3.0.3. While prior releases, such as 3.0.2 had support for ESXi, it has improved greatly with the updated 3.0.3 release. ESXi has a more updated version as well, version 3.5 Update 3. Likewise, if you are using PlateSpin’s PowerConvert or Vizioncore’s vConverter, you should make sure the version you are using supports ESXi as a target.

For most situations, converting machines to an ESXi host is not a big deal. The only practice issue that you may encounter would be the destination log-in selection of the converted virtual machine. Like many administrators, I have frequently used vCenter as the destination. If vCenter is not present in ESXi, ESXI will use the host as the destination log-in. The figure below shows the log-in:

Destination login to ESXi

Performing P2V conversions directly to ESX or ESXi hosts (no vCenter credentials) will use the local password to authenticate, and root is the default credential for ESXi. Beyond that, vCenter Converter converts machines to ESXi host nicely. Guest conversions can be placed in resource pools if set up on the host, as well as selected storage on VMFS volumes that are iSCSI, local,or fibre channel SAN. More information on vCenter Converter can be found on the VMware website.


January 21, 2009  7:04 PM

Getting started with iSCSI and VMware ESX

Rick Vanover Rick Vanover Profile: Rick Vanover

Many VMware ESX administrators are quite comfortable with fibre channel storage but have not ventured into the iSCSI world. I recently set up my first iSCSI configuration for a small VMware Infrastructure 3 installation and it was quite successful. Here are some takeaways:

iSCSI is quite easy to configure. ESX’s iSCSI support is fully available in the form of a software initiator that uses a VMkernel interface. “That easy?” you ask? Yes, it is really that easy.

Using Ethernet is convenient. Until this point, I have exclusively used fibre channel storage for virtual machine file system (VMFS) volumes. With the ESX iSCSI software initiator, I simply dedicated some gigabit network interface cards to the VMkernel interface and was ready to configure the iSCSI adapter. There is experimental support for a hardware initiator with the QLogic 4010 interface.

There is a minimal configuration for the storage adapter. ESX has an iSCSI software adapter listed in the storage adapters section of the VMware Infrastructure Client. Once you configure this interface, the system is ready to receive a LUN. The figure below shows the configuration of the software iSCSI interface:

iSCSI Storage configuration in the VI Client

After those pointers, I was quickly running with a LUN provided from the storage system. Once the LUN is presented to the host, it is indistinguishable from other VMFS volumes. Full VMotion, Distributed Resource Scheduler and other VMware tools are available on these volumes, including the esxcfg- series of commands.

If you are getting started with iSCSI, be sure to go through the drills related to configuration steps on ESX. Also, visit your system architecture plan and make sure that the iSCSI interfaces are provisioned well by not also holding other traffic, and be sure to check out VMware’s iSCSI configuration document available for download from the VMware website.


January 21, 2009  3:36 PM

VMware videos

Eric Siebert Eric Siebert Profile: Eric Siebert

At a recent company meeting we saw a very cool and creative video of a commercial made by our new advertising agency. It was originally part of their portfolio that they submitted to us during the selection process. Watching their video gave me the idea of looking around and seeing what videos I could find that were related to VMware and virtualization.

The first video I found was on VMware’s VMworld.com website. It’s a cute little video that has various kids describing what they think a virtual machine is (free registration required to view it). I also found several other good videos posted there, including one that was a spoof on the PC versus Mac commercials entitled VMware vs. production servers, another called Are you hip to virtualization and one on The future of the cloud with Paul Maritz. Additionally they have all the video confessionals from VMworld 2008 where people sat in a booth and were filmed talking about VMware and virtualization. Also check out the Best of VMworld award presentation if you haven’t already seen it.

Doing some further digging around on YouTube I found the infamous Falconstor giveaway gone wrong video from VMworld 2008, a VMware Infrastructure 3 demo video and a Site Recovery Manager (SRM) demonstration video with Richard Garsthagen from VMware. I also found a cool distributed power management video with an interesting soundtrack, another interesting video called Live it, love it, VMware it from a company that I just found out is right down the street from me.

SearchVMware.com contributor and blogger David Davis also has many useful VMware training videos on his website that you can check out. And of course I couldn’t possibly talk about videos and VMware without mentioning Eric Sloof who is the Steven Spielberg of virtualization videos. Eric always seems to have a video camera with him and posted tons of videos during his travels to VMworld 2008. He’ll also be at VMworld Europe next month so watch for lots of great video coverage from him also.

Got a good VMware or virtualization related video to share? Let us know in the comments below.


Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: