If VMware eliminates the ESX service console, VMware would basically have to retire ESX as ESX without the service console is essentially ESXi. Before I go any further, however, I’ll explain what the service console is and the architecture differences between ESX and ESXi.

At their cores, ESX and ESXi are identical: the VMkernel (which is the hypervisor layer) is the exact same source code and file size for both versions. ESX and ESXi also have a management layer that operates outside of the VMkernel. With ESXi it’s simply called the management interface/console. This management component runs as a privileged virtual machine that has its own operating system and additional VMware management commands.

On ESX hosts, this management component is the service console and runs a modified version of Red Hat Linux taking up several GBs of disk space. On ESXi hosts, this management component is much smaller and is based on an implementation of the Posix variant of Unix within a Busybox framework, which reduces the size of ESXi to approximately 32 MB.

Because of the differences in the management layer some of the administration tasks are different between ESX and ESXi. For example, patching without using Update Manager is different. ESX makes use of a command-line utility called esxupdate, and ESXi relies on a separate update utility that installs with the VMware Infrastructure and vSphere Clients. Web access for VM administration on ESXi hosts is not supported because ESXi does not run a Web application server like ESX does.

Because ESX is running a full OS in the service console, administrators have more flexibility with configuration, scripting, administration and so on. Software and agents (i.e. backup, hardware) can be installed on the service console, which can add additional capabilities to it. This was especially important for monitoring hardware health, but recently VMware has added this capability into both ESX and ESXi so these agents are no longer required to monitor the host hardware.

Essentially, the service console provides administrators with more power and flexibility, which is why many administrators do not want to see it go away. This added flexibility , however, comes with a cost: increased complexity and increased security and operational risks.

VMware has taken steps to get away from the service console by introducing remote management tools such as the remote command-line interface (Remote CLI) , VMware Infrastructure Management Assistant (VIMA) and vSphere Management Assistant (vMA) to take the place of some of the scripting and administration tasks that can be performed inside the service console. But it is still easier in many cases to use the service console than to use the remote management tools. Troubleshooting is much easier when you’re using the service console because of the additional commands that are available.

So when will VMware finally eliminate the ESX service console and fully transition to ESXi? As I mentioned earlier, it is doubtful that this will happen soon as VMware will need to make additional modifications to ESXi to replace some of the functionality that will be missing without the service console, which includes further developing management tools if necessary. My estimate is that VMware may retire the service console with its next major release (vSphere 5?). The one thing I know for sure, though, is that when it does happen there will be many unhappy administrators who will miss their beloved service console.

Take one morning’s statistics from this past week:

• 3 pages of new VMware vSphere Forum Posts vs 1.5 pages of VMware ESX 3.5 Forum Posts.
• VMware ESX 3.5 Forum posts used to be around 3 pages
• The majority of VMware vSphere Forum Posts that dealt with ESX 4 vs ESXi 4 was in favor of ESXi 4 by a wide margin (I did not actually count posts but noticed there were more ESXi posts than ESX posts in those 3 pages)

So using this as a rough measurement and in a completely unscientific way, we do see that VMware vSphere is being investigated for use at least by those activie within the VMware community forums. The same thing happened when ESX v3.5 was released, and over time ESX 3.0 community posts dropped to less than a dozen per day. (I say per day because I review the vSphere, ESX 3.5, and ESX 3.0 communities once about every 24 hours give or take an hour or so.)

Given the types of questions, it looks like two things are happening:

• VMware vSphere is being investigated
• VMware ESXi 4 is the packaging of choice

The second item could be because many people believe that when the next version of ESX comes out, it won’t have a service console, and these users want to get a head start on the adjustment.

What we do not know from this type of adoption measurement is whether these are adoptions for use within production, enthusiasts, or testing within lab environments.

What we do know is that the increase in volume and the drop in ESX v3.5 forum posts is that vSphere is definitely gaining traction. This is not surprising, but what is to me is that ESXi 4 has a constant flow of posts while ESX 4 does not. This will shift the security model people employ to protect virtualization hosts as well.

The methods for ESXi are very similar to that of ESX, but the execution is different as ESXi doesn’t have a service console like ESX’s. The methods below are listed in order of usage preference, beginning with using normal VM commands and ending with a brute force method.

Method 1: Use the vmware-cmd command in the vSphere command-line interface (CLI)

Note: The vSphere CLI is formerly known as the Remote CLI and is not to be confused with the vSphere PowerCLI. The vSphere CLI is the CLI equivalent of using the vSphere Client. Because ESXi does not have a service console like ESX’s, you need to use the remote vSphere CLI to run the vmware-cmd command with ESXi. The vSphere CLI can be downloaded and installed on any Linux or Windows system and can be used to run specific commands remotely on any ESX/ESXi host, and consists of a collection of Perl scripts for each specific ESX/ESXi command. To use this method, follow the steps below.

1. Run the vSphere CLI on the system that you installed it on. You’ll need to switch to the \bin subdirectory where the Perl scripts are located to run the commands.
2. The vmware-cmd command uses the configuration file name (.vmx) of the VM to specify the VM on which it’s going to perform an operation. You can type vmware-cmd.pl -H <ESXi host name> -l to get a list of all VMs on the host and the path and name of their configuration file. The path uses the Universally Unique Identifier (UUID) or long name of the data store; alternatively, you can use the friendly name instead. You’ll be prompted for a log in to the ESXi host before the command will execute. Here you have the option to specify a vCenter Server with -H and you use -T to specify the ESXi host that the vCenter Server manages. Note: You can avoid entering log-in information every time you run a command by using a configuration file or Windows authentication passthrough using Security Support Provider Interface (SSPI). See the vSphere Command-Line Interface
   Installation and Reference Guide documentation for more info.
3. You can optionally check the power state of the VM by typing vmware-cmd.pl -H <ESXi host name> <VM config file path & name> getstate.
4. To forcibly shut down a VM, type vmware-cmd.pl -H <ESXi host name> <VM config file path & name> stop hard.
5. You can check the state again to see if it worked; if it did the state should now be off.

Method 2: Use the vm-support command to shut down the VM

When you use the vm-support command to shut down a VM, you must first find the virtual machine ID (VMID) and then use the vm-support command to forcibly terminate it. This method does more then shut down the VM – it also produces debug information that you can use to troubleshoot an unresponsive VM. On ESXi hosts the vm-support command can be using the special tech support mode which provides access to its Busybox, Posix-based management console.

1. On the ESXi console, press Alt-F1.
2. Type the word unsupported (text will not be displayed while typing) and press Enter. A password prompt will appear, enter the root password for the ESXi host and you will be at a # prompt in the root partition.
3. The vm-support command is a multi-purpose command that is mainly used to troubleshoot host and VM problems. You can use the -X parameter to forcibly shut down a VM and also produce a file with debug information. As with ESX hosts, running this command will create a .tgz file but it will not be located in the directory that you run the command in. Instead it will be created in the /var/tmp directory which points to the 4 GB Virtual File Allocation Table (VFAT) system swap partition. You can also set a Virtual Machine File System (VMFS) volume as your working directory for the .tgz file. First, type vm-support -x to get a list of VMIDs of your running VMs.
4. To forcibly shut down the VM and generate core dumps and log files, type vm-support -X <VMID>. If you wish to specify an alternate directory for the .tgz file that is created also add the -w <vmfs volume path> parameter. You will receive prompts asking if you want to take a screenshot of the VM. This can be useful if you want to see if there are any error messages. You will also be prompted about whether you wish to send an non-maskable interrupt (NMI) and an ABORT to the VM, which can further aid in debugging. You must say yes to the ABORT prompt for the VM to be forcibly stopped. Once the process completes, which can take 10-15 minutes, a .tgz file will be created in the /var/tmp directory that you can use for troubleshooting purposes.
5. You can check the state of the VM again by typing vm-support -x. You should not see the VM listed at this point. Be sure and delete the .tgz file that is created when you are done to avoid filling up your host disk.
6. You can leave tech support mode by typing ‘exit’ and pressing Alt-F2 to return to the normal console mode.

Method 3: Find the VM’s process identifier and forcibly terminate it

This method also relies on using the tech support mode console that is used in method 2 to run the commands.

1. On the ESXi console, press Alt-F1.
2. Type the word unsupported (text will not be displayed while typing) and press Enter. A password prompt will appear. Enter the root password for the ESXi host and you will be at a # prompt in the root partition.
3. The process status (ps) command shows the currently-running processes on a server, and the grep command finds the specified text in the output of the ps command. Type ps -g | grep <virtualmachinename> which will return the WID (first column), CID (second column) and process group ID (PGID) (fourth column) of the running processes of the VM. You will have several entries returned; the number in the fourth column of the entries is the PGID of the VM.
4. The kill command sends a signal to terminate a process using its ID number. The ‘-9′ parameter forces the process to quit immediately and cannot be ignored like the more graceful ‘-15′ parameter can sometimes be. Type kill -9 <PGID> which will forcibly terminate the process for the specified VM.
5. You can check the state of the VM again by typing vm-support -x; you should no longer see the VM listed.
6. You can leave tech support mode by typing ‘exit’ and press Alt-F2 to return to the normal console mode.

All three of these methods work identically on ESXi hosts in both VMware Infrasture 3 and vSphere.

“Several stories have emerged that made it look like VMware has learned from Microsoft and is now practicing the same strategy. Maybe Paul Maritz’s tricks for Microsoft are now reused against VMware competitors,” van Zanten wrote. “First, there was a change in their VMworld policy reported by Brian Madden which according to VMware in an official response was just to prevent competitors from trashing VMware like Microsoft did at VMworld 2008. Although that is a viable explanation, the text now is in the legal documents and can be used as VMware pleases.”

Here’s another dirty move; VMware asked Veeam to remove its Backup & Replication support for VMware ESXi free edition in its latest Essentials Bundle, Acceleration Kits for VMware vSphere 4. Alex Barrett also wrote a great story on that, and how this kind of move might push ESXi users – which are mostly small and medium sized businesses – right into Microsoft’s arms.

These tactics raise the question of whether VMware plans to block more ESXi tools, forcing people to upgrade to a paid version of their software. As van Zanten wrote in his blog, “It is obvious they want ESXi Free to be “unmanageable”, since it is difficult to manage an ESXi free host with a read only remote administration kit. But why? Does VMware think that a small company will now switch to VMware’s vSphere Essentials edition just to be able to manage ESXi? Is VMware afraid that customers start building large clusters of ESXi Free hosts and use third party products to manage them?”

These concerns and others have yet to be answered. In the meantime, I’ll leave you with a couple text book characteristics of bullies:

Those who bully have personalities that are authoritarian, combined with a strong need to control or dominate… If aggressive behavior is not challenged in childhood, there is a danger that it may become habitual.

And the result of “habitual” bullying behavior, VMware, is alienation. In this case, of your customers.

Get your license keys:

1. Log in to your VMware Account by going to http://www.vmware.com, clicking on Account, then clicking on Manage Product Licenses.
2. Select your product, ‘VMware Infrastructure 3′ from the drop down menu for Server & Datacenter Products.
3. Once you go to vSphere licenses, your VI3 licenses no longer exist, so be sure you get the proper number of VI3 licenses that you need for the time being.
4. In the upper right will be a ‘vSphere Licenses’ button to click. Click on that button or clock on Go To: ‘VMware vSphere’. Either will work.
5. Here is where you can divide your licenses so you have the proper number of CPUs per host. If for example you have 12 licenses and 6 2 CPU hosts, you can create 6 2 CPU licenses.

If you make too many vSphere licenses and need more VI3 licenses you can also use the ‘Downgrade Licenses’ link to do so.

That is all there is to it. VMware vSphere uses a license key instead of a license file. Once you have migrated your licenses to vSphere and divided them appropriately you can record the license keys for future use. No need to download a license file or anything like that.

Get the software:

Getting vSphere 4 is much easier than getting the licensing but there is still a gotcha. The steps are follows:

1. Go to the VMware download page.
2. Click ‘Download’ for VMware vSphere 4.
3. Expand the level of your license. I.e. if your level is Standard you will expand the ‘Download VMware vSphere 4 Standard’ link. If your level is Enterprise you will expand the ‘Download VMware vSphere 4 Enterprise’ link.
4. Click on the link for your choice of install, ESX 4 or ESXi 4 (I usually get both).
5. Read the EULA carefully and accept it as you desire.
6. Download pretty much everything from the resultant download page. VMware has added a Download Manager that I found very useful indeed.
7. Go back and click on the link for VMware vCenter Server 4 as well as for VMware Consolidated Backup and repeat the process you used for ESX 4 and/or ESXi 4.

I usually download all four ESX, ESXi, VMware vCenter Server 4, and VMware Consolidated Backup. Once that is done you are now ready to reinstall.

Upgrade vs. Reinstall

I leave that to you. Hopefully these steps will aid you in upgrading your vSphere licenses and getting the software. Many people are doing this at this moment so occassionally the VMware site will either not display everything properly or be temporarily out of service. Be patient as it will come back very soon.

Administration Tools

• VMware‘s VI Toolkit w/Microsoft PowerShell
• VMware’s VI Perl Toolkit
• VMware’s Remote CLI
• VMware Update Manager Plug-in for VMware Infrastructure Client
• Andrew Kutz’s Storage VMotion VMware Infrastructure Client Plug-in
• Hyper9‘s Search Toolbar VMware Infrastructure Client Plug-in
• Tripwire Opscheck (a must for solving VMware VMotion issues)
• PuTTY (a must for accessing host via secure shell (SSH) from Window’s systems)
• PowerGUI (GUI for PowerShell and VI Toolkit)
• Snaphunter (hunt for and destroy snapshots)
• Snapalert (find and commit snapshots with Remote CLI support)
• Adobe Acrobat (for reading VMware whitepapers, hardware compatibility lists and other documents)
• Twhirl (for Twitter access)
• Firefox for access to the VMware Communities Forums and Google Search

Performance/Monitoring Tools

• VMware Infrastructure Client for real time performance graphs
• VMware’s esxtop for details on server utilization
• vmktree
• Vizioncore‘s vFoglight
• Nagios for service monitoring

Reporting

• VMware Virtual Infrastructure Client Mapping capability
• Veeam Reporter
• Alan Renouf’s (http://www.virtu-al.net) vDiagram

Backup Tools

• VMware Consolidated Backup
• VMware Converter
• If only ESX, then one of either Vizioncore vRangerPro, PHD Technologies’ esXpress, or Veeam Backup
• If only ESXi, then Veeam Backup 3.0
• VMware’s vcbMounter

P2V Tools

• VMware Capacity Planner
• VMware Converter 4.0

Security Tools

• Ultimate Boot CD for Darik’s Boot and Nuke utility (in ISO form)
• Tripwire ConfigCheck
• My own Virtual Security Management Suite (still in pre-beta with a hope to get it out soon!)
• Hardening Script from VMware Virtual Infrastructure Security: Securing ESX and the Virtual Environment, available first half of 2009.
• DISA STIG for ESX SRR
• Nessus
• Nmap
• Backtrack for Penetration Testing (in ISO form)
• Helix for Forensics (in ISO form)
• Nuclues Kernel Linux for Data Recovery
• Virtual Firewall (smoothwall, IPcop, etc.)

General Tool

• Ulli Hankeln’s MOA

There you have it: the contents of my virtualization toolbox. I am sure there are many more that I have missed as I do not use them very much or have never used them. Which of these tools will prove to be the most useful in the future? I believe it will be the VI Toolkit but I will keep all these tools within the toolbox, cleaned, and ready for use.

Editor’s note: Readers, any suggestions for other helpful tools? Comment and add to the list.

A recent VMware Knowledgebase article has a great matrix for collecting diagnostic information for many VMware products. One product that was missing from the lineup, however, was VMware ESXi, so I thought I would cover the methods for collecting diagnostic information from an ESXi host in addition to covering the collection methods for VMware ESX and vCenter Server.

Typically when you open a support case with VMware they will want you to generate diagnostic bundles that they can use to diagnose your issue. Collecting this diagnostic information from ESX/ESXi hosts and vCenter Servers consists of packaging together various log and configuration files as well as command output and performance data that can be used for troubleshooting purposes. There are two ways to do this; you can use the vm-support script that is run from the ESX service console/ESXi management console or you can use the Export Diagnostic Data option in the VMware Infrastructure Client (VI Client).

The vm-support script can be run in the ESX service console, but you can also run it in the ESXi hidden technical support management console. (For information on accessing the ESXi hidden console see How to access the VMware ESXi hidden console.)

When you run the vm-support command without any options, the command creates a single .tgz file containing thousands of files that can be extracted and viewed to troubleshoot problems. The size of this file will vary depending on various factors including how many VMs are on your host and the typical size is around 20 MB. Before running the vm-support script you should switch to the /tmp directory as the .tgz file is created in the directory where you run the script. Once the file is created you can copy it to your workstation using a program like WinSCP and extract it using either the Linux tar command or an application like WinZip. These files are very useful to VMware support for troubleshooting your problem. You can also specify parameters when running vm-support to collect specific information or do certain tasks.

Below are some of the most commonly used parameters:

• -n – causes no core files to be included in the tar file
• -s – take performance snapshots in addition to other data
• -S – take only performance snapshots
• -x – lists world ids (wid) for running VMs
• -X <wid> – grab debug info for a hung VM
• -w <dir> – set working directory for output files
• -h – displays help for command usage and available options

If you do not want to use the command line to collect this information you can use the VI Client. You can connect to either a single ESX or ESXi host, or you can generate diagnostic bundles for multiple hosts by connecting to a vCenter Server. Once you connect, select File from the top menu, Export, and finally the Export Diagnostic Data option. Optionally, you can click the Administration button, select the System Logs tab and click the Generate Diagnostic Data button. Or, to collect vCenter Server only diagnostics, you can use the Generate VirtualCenter Server log bundle – FULL option while directly on the vCenter Server (Start > All Programs > VMware > Generate VirtualCenter Server log bundle – FULL).

When connected to a standalone host you can only opt to select a directory on your workstation to store the file. When connected to a vCenter Server you can collect diagnostic data from multiple hosts as well as the vCenter server. Once you select the hosts, a task will be created in the VI Client, the vm-support script will run on the hosts you selected, and the resulting .tgz files will be downloaded to your workstation. If you also chose to include the vCenter Server it will create a separate diagnostic bundle for it in a zip file format.

As these diagnostic bundles also contain configuration files it is a good idea to run them periodically even if you do not have a problem so you have a backup of your host configuration. Just be careful that you clean up the .tgz files on your hosts afterwards to avoid filling up your disk partitions.

A VM with a raw disk or raw disk map (RDM) allows the guest OS to write directly to the LUN or disk partition within a LUN assigned to it, utilizing a pass-through mechanism. Using a raw disk or RDP won’t result in a noticeable performance gain, but there is often an improvement in management. In general, when a VMDK grows past a certain size set by the administrator, the administrator will opt to use a raw disk or RDM.

With VMware ESX v2.5 and ESX v3.0 it took a few simple VM configuration file edits to enable a raw disk when using local storage. Unfortunately, those methods no longer work on VMware ESX v3.5. There is, however, a solution. It is not very elegant, but it will work.

The solution is to use vmkfstools to import or copy a virtual machine disk file to the LUN or partition to use for the raw device. Here is an example that I just tested and seems to work. First, I needed to find out which device held the LUN I was going to assign to my VM.

1. Run fdisk -l to find the LUN.

# fdisk -l

Disk /dev/cciss/c0d0: 146.8 GB, 146807930880 bytes
255 heads, 63 sectors/track, 17848 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot    Start       End    Blocks   Id  System
/dev/cciss/c0d0p1   *         1        13    104391   83  Linux
/dev/cciss/c0d0p2            14       650   5116702+  83  Linux
/dev/cciss/c0d0p3           651      1287   5116702+  83  Linux
/dev/cciss/c0d0p4          1288     17848 133026232+   f  Win95 Ext'd (LBA)
/dev/cciss/c0d0p5          1288      1924   5116671   83  Linux
/dev/cciss/c0d0p6          1925      2561   5116671   83  Linux
/dev/cciss/c0d0p7          2562      3198   5116671   83  Linux
/dev/cciss/c0d0p8          3199      3453   2048256   82  Linux swap
/dev/cciss/c0d0p9          3454     17835 115523383+  fb  Unknown
/dev/cciss/c0d0p10        17836     17848    104391   fc  Unknown

Disk /dev/cciss/c0d1: 440.4 GB, 440430842880 bytes
255 heads, 63 sectors/track, 53546 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/cciss/c0d1 doesn't contain a valid partition table

2. Run esxcfg-vmhbadevs to find the vmhba device associated with the LUN.

# esxcfg-vmhbadevs
vmhba0:0:0     /dev/cciss/c0d0
vmhba0:1:0     /dev/cciss/c0d1

3. Create the VM with a standard virtual disk (VMDK).

4. Use vmkfstools from the command-line interface to import the VMDK into the vmhba device associated with the LUN using a disk target of RAW. Kill the import after you hit 1%.

# vmkfstools -i OpenFiler.vmdk -d raw:/vmfs/devices/disks/vmhba0:1:0:0 OpenFiler_1.vmdk
Destination disk format: raw disk out of '/vmfs/devices/disks/vmhba0:1:0:0'
Cloning disk 'OpenFiler.vmdk'...
Clone: 1% done.<ctrl-C>

5. Review the resultant VMDK metadata file. Note that the size of the LUN is references. For size take 860216490 and multiply by 512 for 410 GBs.

# cat OpenFiler_1.vmdk
# Disk DescriptorFile
version=1
CID=c0699ec2
parentCID=ffffffff
createType="vmfsRaw"

# Extent description
RW 860216490 VMFSRAW "/vmfs/devices/disks/vmhba0:1:0:0"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.uuid = "60 00 C2 97 5c a6 e9 59-f3 de ba f6 83 ed 15 73"
ddb.geometry.cylinders = "1044"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

5. Using the VMware Infrastructure Client (VIC), add an existing disk into the VM or add the following lines to the VMX file. (Note that you want to use a virtual SCSI controller not already in use. Also, the VM you make these changes to should be powered off and perhaps even unregistered if you are doing this from the CLI. If you use the VIC add the device following the red arrow path in the diagram below.)

scsi1.present = "true"
scsi1.sharedBus = "none"
scsi1.virtualDev = "lsilogic"
scsi1:0.present = "true"
scsi1:0.fileName = "OpenFiler_1.vmdk"
scsi1:0.deviceType = "scsi-hardDisk"

Now you have a VM that can directly access the local LUN on your VMware ESX host. I imagine this will work the same way for ESXi.

Converting a physical machine into an ESXi virtual machine works best when you use the latest version of VMware Converter, version 3.0.3. While prior releases, such as 3.0.2 had support for ESXi, it has improved greatly with the updated 3.0.3 release. ESXi has a more updated version as well, version 3.5 Update 3. Likewise, if you are using PlateSpin’s PowerConvert or Vizioncore’s vConverter, you should make sure the version you are using supports ESXi as a target.

For most situations, converting machines to an ESXi host is not a big deal. The only practice issue that you may encounter would be the destination log-in selection of the converted virtual machine. Like many administrators, I have frequently used vCenter as the destination. If vCenter is not present in ESXi, ESXI will use the host as the destination log-in. The figure below shows the log-in:

Performing P2V conversions directly to ESX or ESXi hosts (no vCenter credentials) will use the local password to authenticate, and root is the default credential for ESXi. Beyond that, vCenter Converter converts machines to ESXi host nicely. Guest conversions can be placed in resource pools if set up on the host, as well as selected storage on VMFS volumes that are iSCSI, local,or fibre channel SAN. More information on vCenter Converter can be found on the VMware website.