ESXi

When will VMware retire the ESX service console?

VMware has been stating that the ESX Service Console (COS) will go away ever since it released ESXi, but so far this has not happened and it doesn’t seem likely to happen any time soon. Despite the session at VMworld this year entitled “How to survive in a COS-less world”, VMware doesn’t seem to be moving that fast to retire the service console. Doing so would be a big change and is only likely to happen with a major new release; VMware had the chance to do this with vSphere but chose not to.

If VMware eliminates the ESX service console, VMware would basically have to retire ESX as ESX without the service console is essentially ESXi. Before I go any further, however, I’ll explain what the service console is and the architecture differences between ESX and ESXi.

Continued »

Monitoring vSphere adoption: A forum moderator’s perspective

As an active moderator and VMware Communities Guru, I’m in a unique position to see the level of vSphere adoption from an interesting vantage point — topic activity in the forums.

Take one morning’s statistics from this past week:

• 3 pages of new VMware vSphere Forum Posts vs 1.5 pages of VMware ESX 3.5 Forum Posts.
• VMware ESX 3.5 Forum posts used to be around 3 pages
• The majority of VMware vSphere Forum Posts that dealt with ESX 4 vs ESXi 4 was in favor of ESXi 4 by a wide margin (I did not actually count posts but noticed there were more ESXi posts than ESX posts in those 3 pages)

So using this as a rough measurement and in a completely unscientific way, we do see that VMware vSphere is being investigated for use at least by those activie within the VMware community forums. The same thing happened when ESX v3.5 was released, and over time ESX 3.0 community posts dropped to less than a dozen per day. (I say per day because I review the vSphere, ESX 3.5, and ESX 3.0 communities once about every 24 hours give or take an hour or so.)

Given the types of questions, it looks like two things are happening:

• VMware vSphere is being investigated
• VMware ESXi 4 is the packaging of choice

The second item could be because many people believe that when the next version of ESX comes out, it won’t have a service console, and these users want to get a head start on the adjustment.

What we do not know from this type of adoption measurement is whether these are adoptions for use within production, enthusiasts, or testing within lab environments.

What we do know is that the increase in volume and the drop in ESX v3.5 forum posts is that vSphere is definitely gaining traction. This is not surprising, but what is to me is that ESXi 4 has a constant flow of posts while ESX 4 does not. This will shift the security model people employ to protect virtualization hosts as well.

Three ways to kill a frozen vSphere ESXi host virtual machine

At some point, you may need to know how to kill a stuck or frozen VMware vSphere 4.0 ESXi host virtual machine when the traditional power controls do not work. As with VMware ESX, there are several methods, which I covered in a previous post, killing a virtual machine (VM) on a VMware ESX host in vSphere.

The methods for ESXi are very similar to that of ESX, but the execution is different as ESXi doesn’t have a service console like ESX’s. The methods below are listed in order of usage preference, beginning with using normal VM commands and ending with a brute force method.

Method 1: Use the vmware-cmd command in the vSphere command-line interface (CLI) Continued »

VMware: the new bully on the block?

VMware architect and virtualization expert Gabe van Zanten wrote an interesting post on his blog, “Gabe’s Virtual World” pointing out that VMware appears to be following in Microsoft’s footsteps by bullying partners and customers.

“Several stories have emerged that made it look like VMware has learned from Microsoft and is now practicing the same strategy. Maybe Paul Maritz’s tricks for Microsoft are now reused against VMware competitors,” van Zanten wrote. “First, there was a change in their VMworld policy reported by Brian Madden which according to VMware in an official response was just to prevent competitors from trashing VMware like Microsoft did at VMworld 2008. Although that is a viable explanation, the text now is in the legal documents and can be used as VMware pleases.”

Here’s another dirty move; VMware asked Veeam to remove its Backup & Replication support for VMware ESXi free edition in its latest Essentials Bundle, Acceleration Kits for VMware vSphere 4. Alex Barrett also wrote a great story on that, and how this kind of move might push ESXi users - which are mostly small and medium sized businesses - right into Microsoft’s arms.

These tactics raise the question of whether VMware plans to block more ESXi tools, forcing people to upgrade to a paid version of their software. As van Zanten wrote in his blog, “It is obvious they want ESXi Free to be “unmanageable”, since it is difficult to manage an ESXi free host with a read only remote administration kit. But why? Does VMware think that a small company will now switch to VMware’s vSphere Essentials edition just to be able to manage ESXi? Is VMware afraid that customers start building large clusters of ESXi Free hosts and use third party products to manage them?”

These concerns and others have yet to be answered. In the meantime, I’ll leave you with a couple text book characteristics of bullies:

Those who bully have personalities that are authoritarian, combined with a strong need to control or dominate… If aggressive behavior is not challenged in childhood, there is a danger that it may become habitual.

And the result of “habitual” bullying behavior, VMware, is alienation. In this case, of your customers.

VMware vSphere is available: Now what do you do?

So now that vSphere is available, how do you get your own copy and upgrade existing licenses? Hopefully, you have already checked if your Service and Support has been properly upgraded or existed prior to May 21, 2009. If so then you are in good shape.

Get your license keys:

1. Log in to your VMware Account by going to http://www.vmware.com, clicking on Account, then clicking on Manage Product Licenses. Continued »

My virtualization toolbox

What comprises a VMware vExpert’s virtualization toolbox? I actually think there are several different layers of tools depending on the particular VMware virtualization administrators job role. Mine, for example, has a number of security tools that most people would not normally need, but I do. So below is a list of what comprises my virtualization toolbox.

Administration Tools

• VMware’s VI Toolkit w/Microsoft PowerShell
• VMware’s VI Perl Toolkit
• VMware’s Remote CLI
• VMware Update Manager Plug-in for VMware Infrastructure Client
• Andrew Kutz’s Storage VMotion VMware Infrastructure Client Plug-in
• Hyper9’s Search Toolbar VMware Infrastructure Client Plug-in
• Tripwire Opscheck (a must for solving VMware VMotion issues)
• PuTTY (a must for accessing host via secure shell (SSH) from Window’s systems)
• PowerGUI (GUI for PowerShell and VI Toolkit)
• Snaphunter (hunt for and destroy snapshots)
• Snapalert (find and commit snapshots with Remote CLI support)
• Adobe Acrobat (for reading VMware whitepapers, hardware compatibility lists and other documents)
• Twhirl (for Twitter access)
• Firefox for access to the VMware Communities Forums and Google Search

Performance/Monitoring Tools

• VMware Infrastructure Client for real time performance graphs
• VMware’s esxtop for details on server utilization
• vmktree
• Vizioncore’s vFoglight
• Nagios for service monitoring

Reporting

• VMware Virtual Infrastructure Client Mapping capability
• Veeam Reporter
• Alan Renouf’s (http://www.virtu-al.net) vDiagram

Backup Tools

• VMware Consolidated Backup
• VMware Converter
• If only ESX, then one of either Vizioncore vRangerPro, PHD Technologies’ esXpress, or Veeam Backup
• If only ESXi, then Veeam Backup 3.0
• VMware’s vcbMounter

P2V Tools

• VMware Capacity Planner
• VMware Converter 4.0

Security Tools

• Ultimate Boot CD for Darik’s Boot and Nuke utility (in ISO form)
• Tripwire ConfigCheck
• My own Virtual Security Management Suite (still in pre-beta with a hope to get it out soon!)
• Hardening Script from VMware Virtual Infrastructure Security: Securing ESX and the Virtual Environment, available first half of 2009.
• DISA STIG for ESX SRR
• Nessus
• Nmap
• Backtrack for Penetration Testing (in ISO form)
• Helix for Forensics (in ISO form)
• Nuclues Kernel Linux for Data Recovery
• Virtual Firewall (smoothwall, IPcop, etc.)

General Tool

• Ulli Hankeln’s MOA

There you have it: the contents of my virtualization toolbox. I am sure there are many more that I have missed as I do not use them very much or have never used them. Which of these tools will prove to be the most useful in the future? I believe it will be the VI Toolkit but I will keep all these tools within the toolbox, cleaned, and ready for use.

Editor’s note: Readers, any suggestions for other helpful tools? Comment and add to the list.

Collecting diagnostic data from hosts and vCenter Server

A recent VMware Knowledgebase article has a great matrix for collecting diagnostic information for many VMware products. One product that was missing from the lineup, however, was VMware ESXi, so I thought I would cover the methods for collecting diagnostic information from an ESXi host in addition to covering the collection methods for VMware ESX and vCenter Server.

Typically when you open a support case with VMware they will want you to generate diagnostic bundles that they can use to diagnose your issue. Collecting this diagnostic information from ESX/ESXi hosts and vCenter Servers consists of packaging together various log and configuration files as well as command output and performance data that can be used for troubleshooting purposes. There are two ways to do this; you can use the vm-support script that is run from the ESX service console/ESXi management console or you can use the Export Diagnostic Data option in the VMware Infrastructure Client (VI Client).

The vm-support script can be run in the ESX service console, but you can also run it in the ESXi hidden technical support management console. (For information on accessing the ESXi hidden console see How to access the VMware ESXi hidden console.)

When you run the vm-support command without any options, the command creates a single .tgz file containing thousands of files that can be extracted and viewed to troubleshoot problems. The size of this file will vary depending on various factors including how many VMs are on your host and the typical size is around 20 MB. Before running the vm-support script you should switch to the /tmp directory as the .tgz file is created in the directory where you run the script. Once the file is created you can copy it to your workstation using a program like WinSCP and extract it using either the Linux tar command or an application like WinZip. These files are very useful to VMware support for troubleshooting your problem. You can also specify parameters when running vm-support to collect specific information or do certain tasks.

Below are some of the most commonly used parameters:

• -n - causes no core files to be included in the tar file
• -s - take performance snapshots in addition to other data
• -S - take only performance snapshots
• -x - lists world ids (wid) for running VMs
• -X <wid> - grab debug info for a hung VM
• -w <dir> - set working directory for output files
• -h - displays help for command usage and available options

If you do not want to use the command line to collect this information you can use the VI Client. You can connect to either a single ESX or ESXi host, or you can generate diagnostic bundles for multiple hosts by connecting to a vCenter Server. Once you connect, select File from the top menu, Export, and finally the Export Diagnostic Data option. Optionally, you can click the Administration button, select the System Logs tab and click the Generate Diagnostic Data button. Or, to collect vCenter Server only diagnostics, you can use the Generate VirtualCenter Server log bundle - FULL option while directly on the vCenter Server (Start > All Programs > VMware > Generate VirtualCenter Server log bundle - FULL).

When connected to a standalone host you can only opt to select a directory on your workstation to store the file. When connected to a vCenter Server you can collect diagnostic data from multiple hosts as well as the vCenter server. Once you select the hosts, a task will be created in the VI Client, the vm-support script will run on the hosts you selected, and the resulting .tgz files will be downloaded to your workstation. If you also chose to include the vCenter Server it will create a separate diagnostic bundle for it in a zip file format.

As these diagnostic bundles also contain configuration files it is a good idea to run them periodically even if you do not have a problem so you have a backup of your host configuration. Just be careful that you clean up the .tgz files on your hosts afterwards to avoid filling up your disk partitions.

VMware ESX storage: How to get local storage to act as a raw disk for VMs

VMTN Communities forum users have recently been asking how to make use of a LUN or partition on your local host within a virtual machine (VM) the same way you would if you had a SAN available. This is a more difficult task than some, and not every RAID controller allows this when using the VMware Infrastructure Client. We have to resort to command-line methods to make this happen.

A VM with a raw disk or raw disk map (RDM) allows the guest OS to write directly to the LUN or disk partition within a LUN assigned to it, utilizing a pass-through mechanism. Using a raw disk or RDP won’t result in a noticeable performance gain, but there is often an improvement in management. In general, when a VMDK grows past a certain size set by the administrator, the administrator will opt to use a raw disk or RDM.

With VMware ESX v2.5 and ESX v3.0 it took a few simple VM configuration file edits to enable a raw disk when using local storage. Unfortunately, those methods no longer work on VMware ESX v3.5. There is, however, a solution. It is not very elegant, but it will work.

The solution is to use vmkfstools to import or copy a virtual machine disk file to the LUN or partition to use for the raw device. Here is an example that I just tested and seems to work. First, I needed to find out which device held the LUN I was going to assign to my VM.

1. Run fdisk -l to find the LUN.

# fdisk -l

Disk /dev/cciss/c0d0: 146.8 GB, 146807930880 bytes
255 heads, 63 sectors/track, 17848 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot    Start       End    Blocks   Id  System
/dev/cciss/c0d0p1   *         1        13    104391   83  Linux
/dev/cciss/c0d0p2            14       650   5116702+  83  Linux
/dev/cciss/c0d0p3           651      1287   5116702+  83  Linux
/dev/cciss/c0d0p4          1288     17848 133026232+   f  Win95 Ext'd (LBA)
/dev/cciss/c0d0p5          1288      1924   5116671   83  Linux
/dev/cciss/c0d0p6          1925      2561   5116671   83  Linux
/dev/cciss/c0d0p7          2562      3198   5116671   83  Linux
/dev/cciss/c0d0p8          3199      3453   2048256   82  Linux swap
/dev/cciss/c0d0p9          3454     17835 115523383+  fb  Unknown
/dev/cciss/c0d0p10        17836     17848    104391   fc  Unknown

Disk /dev/cciss/c0d1: 440.4 GB, 440430842880 bytes
255 heads, 63 sectors/track, 53546 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/cciss/c0d1 doesn't contain a valid partition table

2. Run esxcfg-vmhbadevs to find the vmhba device associated with the LUN.

# esxcfg-vmhbadevs
vmhba0:0:0     /dev/cciss/c0d0
vmhba0:1:0     /dev/cciss/c0d1

3. Create the VM with a standard virtual disk (VMDK).

4. Use vmkfstools from the command-line interface to import the VMDK into the vmhba device associated with the LUN using a disk target of RAW. Kill the import after you hit 1%.

# vmkfstools -i OpenFiler.vmdk -d raw:/vmfs/devices/disks/vmhba0:1:0:0 OpenFiler_1.vmdk
Destination disk format: raw disk out of '/vmfs/devices/disks/vmhba0:1:0:0'
Cloning disk 'OpenFiler.vmdk'...
Clone: 1% done.<ctrl-C>

5. Review the resultant VMDK metadata file. Note that the size of the LUN is references. For size take 860216490 and multiply by 512 for 410 GBs.

# cat OpenFiler_1.vmdk
# Disk DescriptorFile
version=1
CID=c0699ec2
parentCID=ffffffff
createType="vmfsRaw"

# Extent description
RW 860216490 VMFSRAW "/vmfs/devices/disks/vmhba0:1:0:0"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.uuid = "60 00 C2 97 5c a6 e9 59-f3 de ba f6 83 ed 15 73"
ddb.geometry.cylinders = "1044"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

5. Using the VMware Infrastructure Client (VIC), add an existing disk into the VM or add the following lines to the VMX file. (Note that you want to use a virtual SCSI controller not already in use. Also, the VM you make these changes to should be powered off and perhaps even unregistered if you are doing this from the CLI. If you use the VIC add the device following the red arrow path in the diagram below.)

scsi1.present = "true"
scsi1.sharedBus = "none"
scsi1.virtualDev = "lsilogic"
scsi1:0.present = "true"
scsi1:0.fileName = "OpenFiler_1.vmdk"
scsi1:0.deviceType = "scsi-hardDisk"

Now you have a VM that can directly access the local LUN on your VMware ESX host. I imagine this will work the same way for ESXi.

Using vCenter Converter to create VMware ESXi virtual machines

Administrators considering VMware ESXi may wonder if vCenter Converter will workfor VMware ESXi conversions. The answer is yes, but there’s a gotcha.

Converting a physical machine into an ESXi virtual machine works best when you use the latest version of VMware Converter, version 3.0.3. While prior releases, such as 3.0.2 had support for ESXi, it has improved greatly with the updated 3.0.3 release. ESXi has a more updated version as well, version 3.5 Update 3. Likewise, if you are using PlateSpin’s PowerConvert or Vizioncore’s vConverter, you should make sure the version you are using supports ESXi as a target.

For most situations, converting machines to an ESXi host is not a big deal. The only practice issue that you may encounter would be the destination log-in selection of the converted virtual machine. Like many administrators, I have frequently used vCenter as the destination. If vCenter is not present in ESXi, ESXI will use the host as the destination log-in. The figure below shows the log-in:

Performing P2V conversions directly to ESX or ESXi hosts (no vCenter credentials) will use the local password to authenticate, and root is the default credential for ESXi. Beyond that, vCenter Converter converts machines to ESXi host nicely. Guest conversions can be placed in resource pools if set up on the host, as well as selected storage on VMFS volumes that are iSCSI, local,or fibre channel SAN. More information on vCenter Converter can be found on the VMware website.