vCenter Server

VMware vSphere is available: Now what do you do?

So now that vSphere is available, how do you get your own copy and upgrade existing licenses? Hopefully, you have already checked if your Service and Support has been properly upgraded or existed prior to May 21, 2009. If so then you are in good shape.

Get your license keys:

1. Log in to your VMware Account by going to http://www.vmware.com, clicking on Account, then clicking on Manage Product Licenses. Continued »

Enhanced alarm reporting in VMware vCenter Server 4.0

The alarm settings in VMware vCenter Server 4.0 have been vastly improved in comparison to what’s available in VMware Infrastructure 3 (VI3). The new alarms let you configure many custom alarm triggers and corresponding actions. VI3 alarms are very limited; you can only configure host and virtual machine alarm types and there are very few triggers that you can select from, as shown in the chart below.

In vCenter Server 4.0, alarm types are split into two categories, conditions/state and events, and each category has different triggers.

Continued »

Memory usage in vCenter Server 4.0

If you are using vCenter Server and plan on upgrading to vSphere you should be aware that vCenter Server 4.0 uses much more memory than vCenter Server 2.5 does. According to the official documentation for both 2.5 and 4.0, the system requirements are very similar:

• 2.0 GHz or faster Intel or AMD x86 processor. Processor requirements can be more if your database runs on the same hardware.
• 2 GB RAM. You might need more RAM if your database runs on the same hardware.

But VMware has a note in the vSphere documentation that states:

vCenter Server includes a service called VMware vCenter Management Webservices. This service requires128MB to 1.5GB of additional memory. The vCenter Management Webservices process allocates the required memory at startup.

Continued »

New vCenter Server features in vSphere

Although vSphere’s vCenter Server offers many useful new features, there are three small ones in particular that were sorely needed in VMware Infrastructure 3 that I’m glad to see in this release.

The first deals with the problem of too much data in the vCenter Server database. The majority of the data in the vCenter Server database is from both guest OS and host historical performance statistics and also Task and Event data. The statistic data is archived per your interval settings so there are limits to its growth, but Task and Event data is retained in the database forever, even for guest OSes and hosts that have been removed from vCenter Server’s inventory.

Continued »

Log file locations in VMware Infrastructure 3

Whenever I am troubleshooting a problem with my VMware ESX hosts I find myself frequently having to look up the location of various VMware Infrastructure 3 (VI3) log files. To try and make this easier for myself and others I’ve decided to put together a list of all the log files that you might use that reside on your ESX host servers and vCenter Servers.

VMware ESX host logs:

• /var/log/vmkernel - VMkernel log, records activities related to the virtual machines and ESX hosts. Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension
• /var/log/vmkwarning - VMkernel Warnings log, records activities with the virtual machines, a subset of the vmkernel log and uses the same rotation scheme.
• /var/log/vmksummary - VMkernel Summary log, used to determine uptime and availability statistics for ESX hosts human-readable summary found in /var/log/vmksummary.txt.
• /var/log/vmware/hostd.log - ESX Server host agent log, contains information on the agent that manages and configures the ESX host and its virtual machines. (Search the file date/time stamps to find the log file it is currently outputting to, or open hostd.log which is linked to the current log file.)
• /var/log/vmware/esxcfg-firewall.log - ESX Firewall log, logs all firewall rule events. Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.
• /var/log/vmware/esxcfg-boot.log - ESX Boot log, logs all ESX boot events. Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.
• /var/log/vmware/esxupdate.log - ESX Update log, logs all updates done through the esxupdate tool.
• /var/log/messages - Service console log, contains all general log messages used to troubleshoot virtual machines or an ESX host.
• /var/log/vmware/webAccess - Web Access log, records information on Web-based access to an ESX host.
• /var/log/vmware/aam - High Availability (HA) log, logs information related to the High Availability service. In ESX 3.0.x these logs were in /opt/LGTOaam512/ instead.
• /var/log/secure - Authentication log, contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.
• /var/log/vmware/vpxa.log - Vpxa log, contains information on the agent that communicates with vCenter Server. Search the file date/time stamps to find the log file it is currently outputting to or open vpxa.log file which is linked to the current log file.

Virtual machine logs:

There is only one log file for virtual machines on a host and it is located in the working directory on the VMFS volume for the VM (i.e /vmfs/volumes/storage1/VM1/). The current log file is always vmware.log and older log files are incremented numerically (i.e. vmware-5.log).

vCenter Server logs:

The primary log file used by vCenter Server is the vpxd-#.log file (the # sign represents a number) which is located in the %allusersprofile%\Application Data\VMware\VMware VirtualCenter\Logs directory on the vCenter Servers version 2.5. The %allusersprofile% variable on most systems is the C:\Document and Settings\All Users directory. On older VirtualCenter 2.0 servers this file was located in the C:\Windows\Temp directory. New log files are automatically created when the logs grow to a certain size and the number in their filename is incremented. You can see which one is currently being used by sorting by modified date or by checking the vpxd-index file, which lists the number of the log file that is currently in use.

In addition to the vCenter Server log there are also logs for some of the other components, including Update Manager, Capacity Planner and Converter Enterprise. The logs are located in the \logs subdirectory under the various product directories which are located in %allusersprofile%\Application Data\VMware\. Here’s the list of log files that you will typically find on vCenter Servers:

• C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs - vCenter Server main log files
• C:\Program Files\VMware\Infrastructure\VirtualCenter Server\tomcat\logs - Tomcat web server log files for the web access component of vCenter Server
• C:\Documents and Settings\All Users\Application Data\VMware\VMware Capacity Planner\Logs - Capacity Planner log files
• C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Logs - Update Manager log files
• C:\Documents and Settings\All Users\Application Data\VMware\VMware Converter Enterprise\Logs - Converter Enterprise log files

The License Manager Server log file is typically named lmgrd.log file and is located in the Windows temp directory (i.e. C:\Windows\Temp). You can verify the location of this file by opening the VMware License Server Tools application and click on the Config Services tab and checking the path to the debug of file field.

I’d recommend printing out this blog post or bookmarking it so the next time you are troubleshooting a problem in your virtual environment you can quickly figure out which log to look at to help figure out the cause.

Troubleshooting mysterious VM resets

Have you ever had VMs mysteriously restart and didn’t know why? Even after checking the VMs vmware.log file and the event log in vCenter Server, you could find no evidence of the restart or any potential problems. Well, the Virtual Machine Monitor (VMM) feature could be the culprit. VMM extends the High Availability (HA) feature to be able to detect guest OS failures by monitoring a heartbeat provided by VMware Tools. In case a guest OS failure is detected (i.e. Windows blue screen of death), the heartbeat would stop and the VM would be restarted on the same host. While this is a great feature to have, it can also be troublesome and cause VMs to restart even when there is not a problem with the guest OS.

I recently experienced this problem first hand when VMs were mysteriously resetting at times even though they were not experiencing problems. The culprit? VMM. As mentioned, when this happens it can be difficult to determine as the vCenter event log will not have an entry that VMM caused the VM to reset, virtual machine state alarms will not be triggered, and the vmware.log will have no mention that the event occurred. The only evidence that a VMM event took place will be in the /var/log/vmware/hostd.log file on the ESX host and will look similar to what follows.

[2009-03-20 04:44:35.252 'TaskManager' 3076453280 info] Task Created : haTask-512-vim.VirtualMachine.reset-47992
[2009-03-20 04:44:35.323 'ha-eventmgr' 3076453280 info] Event 8420 : Win2003-1 on esx1.xyz.com in ha-datacenter is reset
[2009-03-20 04:44:35.323 'vm:/vmfs/volumes/48331160-05c64c5c-edf0-001e0bd8c708/Win2003-1/Win2003-1.vmx' 3076453280 info] State Transition (VM_STATE_ON -> VM_STATE_RESETTING)

Since this feature relies on monitoring heartbeats through VMware tools there is the possibility of certain events happening that cause the heartbeat to stop longer than the configured failure interval, thereby triggering a false positive and resetting the VM. One example of these types of events is an upgrade to VMware Tools on a VM which causes heartbeats to temporarily stop while the VM is being upgraded. For this reason VMware changed the VMM feature in vCenter Server 2.5 Update 4 to also monitor the VM’s disk and network activity. Therefore even if no heartbeats are received within the failure interval, the VM does not reset unless no disk or network activity is detected for a predetermined I/O stats interval. A VM guest OS that is truly locked up will typically not have any disk or network activity in addition to the loss of heartbeat. This added level of monitoring will help eliminate false positives and make this feature even better. Additionally you can change the failure detection interval to higher than the default of 30 seconds. This setting is located in the HA advanced options section as shown below.

So if you ever have a VM mysteriously reset itself and you have VMM enabled, be sure and check the /var/log/vmware/hostd.log file on the ESX host to see if it was caused by VMM. It would be very helpful if VMware could make it so that these types of events are also logged in the vCenter Server events view.

Using custom attributes with vCenter Server

In vCenter Server you can display certain views that will show listings of your hosts and virtual machines (VMs). These views are useful for displaying the configuration and status of all your hosts and VMs, and they are also customizable so you can either display or hide predefined columns to control what is displayed. But what if you want more information besides what you can see with the predefined columns? You can use custom attributes. This feature is unique to vCenter Server. You can display additional information in the columns or annotations section of the host/VM summary page. For example, you could add columns that display information about the OS or function. You can also sort by the custom attribute columns so you can group information together.

As mentioned this is only a feature of vCenter Server, so if you connect to a host directly using the VMware Infrastructure Client (VI Client) you will not see these attributes. They are stored inside in the vCenter Server database in two different tables. The attributes themselves are stored in the VPX_FIELD_DEF table and the attribute values are stored in the VPX_FIELD_VAL table. Because this data is custom if something happens to your database these attributes will be lost as they do not repopulate like some of the other tables do when hosts are added to vCenter Server.

You can create these attributes in two different ways; either be selecting Administration then Custom Attributes from the top menu, or by selecting a host or VM in the left pane and in the right pane selecting the summary tab and, under Annotations, clicking the edit link. VMs will have a notes field built in to the vCenter Server database — this field is not considered a custom attribute. The note field is useful for documenting information about the VM, but custom attributes provide more fields so you can split information up into multiple fields rather than putting it all together in the notes field.

Now let’s walk through how to add custom attributes.

You can add or remove attributes once you have the Custom Attributes window open:

When you add a new attribute, you have a choice of three different types; Host, Virtual Machine or Global:

You may or may not see Host/Virtual Machine If you select the machines individually and click the Edit link instead of using the top menu option. Host and Virtual Machine attributes only apply to those objects and will only be displayed for each one. Global attributes are displayed for either object type.

Once you add the attribute it will be displayed in the Annotations section for the host/VM and will also be available as a column to be displayed in the Host/VM views.

Now that you have your custom attributes defined you can start setting the values for each VM or host. To do this, first select the host or VM, then click the edit link on the summary tab in the annotations section. You can also set this in the view screens by clicking under the column that you want to update on the host or VM that you want to update, as shown below. This method is faster than setting them individually on host or VM.

If you want to further customize the columns, simply right-click on the column heading and you can check or uncheck columns that are displayed, including the predefined columnsn and your custom columns.

Using these custom attributes is a great way of documenting your hosts and VMs inside the VI Client so you know more about them. Often times VM names are not descriptive, so having these extra fields can provide more information to anyone who uses the VI Client. This can be useful as it can prevent certain things from happening, such as accidentally rebooting the wrong VM.

Collecting diagnostic data from hosts and vCenter Server

A recent VMware Knowledgebase article has a great matrix for collecting diagnostic information for many VMware products. One product that was missing from the lineup, however, was VMware ESXi, so I thought I would cover the methods for collecting diagnostic information from an ESXi host in addition to covering the collection methods for VMware ESX and vCenter Server.

Typically when you open a support case with VMware they will want you to generate diagnostic bundles that they can use to diagnose your issue. Collecting this diagnostic information from ESX/ESXi hosts and vCenter Servers consists of packaging together various log and configuration files as well as command output and performance data that can be used for troubleshooting purposes. There are two ways to do this; you can use the vm-support script that is run from the ESX service console/ESXi management console or you can use the Export Diagnostic Data option in the VMware Infrastructure Client (VI Client).

The vm-support script can be run in the ESX service console, but you can also run it in the ESXi hidden technical support management console. (For information on accessing the ESXi hidden console see How to access the VMware ESXi hidden console.)

When you run the vm-support command without any options, the command creates a single .tgz file containing thousands of files that can be extracted and viewed to troubleshoot problems. The size of this file will vary depending on various factors including how many VMs are on your host and the typical size is around 20 MB. Before running the vm-support script you should switch to the /tmp directory as the .tgz file is created in the directory where you run the script. Once the file is created you can copy it to your workstation using a program like WinSCP and extract it using either the Linux tar command or an application like WinZip. These files are very useful to VMware support for troubleshooting your problem. You can also specify parameters when running vm-support to collect specific information or do certain tasks.

Below are some of the most commonly used parameters:

• -n - causes no core files to be included in the tar file
• -s - take performance snapshots in addition to other data
• -S - take only performance snapshots
• -x - lists world ids (wid) for running VMs
• -X <wid> - grab debug info for a hung VM
• -w <dir> - set working directory for output files
• -h - displays help for command usage and available options

If you do not want to use the command line to collect this information you can use the VI Client. You can connect to either a single ESX or ESXi host, or you can generate diagnostic bundles for multiple hosts by connecting to a vCenter Server. Once you connect, select File from the top menu, Export, and finally the Export Diagnostic Data option. Optionally, you can click the Administration button, select the System Logs tab and click the Generate Diagnostic Data button. Or, to collect vCenter Server only diagnostics, you can use the Generate VirtualCenter Server log bundle - FULL option while directly on the vCenter Server (Start > All Programs > VMware > Generate VirtualCenter Server log bundle - FULL).

When connected to a standalone host you can only opt to select a directory on your workstation to store the file. When connected to a vCenter Server you can collect diagnostic data from multiple hosts as well as the vCenter server. Once you select the hosts, a task will be created in the VI Client, the vm-support script will run on the hosts you selected, and the resulting .tgz files will be downloaded to your workstation. If you also chose to include the vCenter Server it will create a separate diagnostic bundle for it in a zip file format.

As these diagnostic bundles also contain configuration files it is a good idea to run them periodically even if you do not have a problem so you have a backup of your host configuration. Just be careful that you clean up the .tgz files on your hosts afterwards to avoid filling up your disk partitions.

vCenter Server 2.5 Update 4 release brings Performance Overview Charts

Many people were hoping that VMware would use VMworld Europe to announce the release date of the next generation VI3 software called vSphere 4.0. While this didn’t happen, VMware did release the next maintenance version of vCenter Server 2.5, which is Update 4. This new release brings the usual bug fixes but also adds a new feature, performance overview charts.

Performance Overview Charts aggregate the normal VM and host performance charts into a single view of key performance metrics for CPU, memory, disk and network usage. This eliminates the need to have to look at each factor individually and helps to show trends or resource issues.

This new feature is a plug-in component to vCenter Server and uses Java, so to use it you’ll need to download the Java SE Development Kit 6u11 and install Java Development Kit 1.6. If you are using Oracle or SQL Express there are some additional steps that you need to complete. Below are the KB articles you should read if you want to use this new plug-in:

• KB Article on how to install the Performance Overview plug-in
• KB Article on using the plug-in with Oracle database
• KB Article on using the plug-in with SQL Express

One other interesting new feature is an update to the Virtual Machine Monitoring feature that is part of HA (High Availability) and will restart VMs on the same host if there is an guest operating system failure and the OS stops responding.  This feature previously relied on a “heartbeat” provided by the VMware Tools application, which is installed on the guest OS. If that heartbeat were to stop, the guest OS would be considered defunct and automatically restarted.

The enhancement allows vCenter Server to not only monitor the heartbeat but also the disk and network activity on the VM. Together, these features add more failure triggers to better indicate a failed OS. One situation where heartbeats may stop for a short period of time is when the VMware Tools application is being upgraded. This would previously cause the VM to restart, as the short stop time would indicate a failure. The added requirement vCenter to report dead network and disk activity for a predetermined I/O stats interval ensures that the VM guest OS is truly in a failed state (i.e. Blue Screen of Death) before it’s restarted.

As usual, it’s a best practice to read the release notes before you install the upgrade to vCenter Server. This upgrade includes updated plug-ins for both Converter and Update Manager, so you should read the release notes for them also. Finally, once you install the upgrade you will have to manually update your plug-ins in the VMware Infrastructure Client on your workstations that use them.

VMworld Europe kicks off with Partner Day: Cloud interoperability, VDI, vSphere

After much anticipation, VMworld 2009 in Cannes has finally begun. After talking about it for days, seeing lots of blogs from people planning to visit VMworld, we’re finally on the go.

Andy Hunt, Vice President for the EMEA Partner Organization from VMware kicked off partner day today, which traditionally is the opening day for VMworld. He welcomed us all and thanked us for coming to VMworld despite that poor economic climate. He was very happy to see about 1,500 visitors at partner day, and was happy to announce that there should be about 5,000 visitors at VMworld in total.

Next to take the stage was CEO of VMware Paul Maritz. Maritz explained that VMware has a budget of $515 billion dollars for research and development (R&D) and that the team of engineers in VMware’s R&D department is larger than any team he has ever worked with while working for Microsoft.

This may sound like a typical blanket statement, but keep in mind that Maritz has worked with Microsoft for 14 years where, amongst other functions, he has been Vice President of the Platform Strategy and Developer Group where he oversaw the development and marketing of System Software Products (including Windows 95, Windows NT and Windows 2000).

Maritz also showed us that in today’s world, IT departments use 70% of their budget to just keep the lights on –  a mere 30% is used for research and competitive development. VMware sees potential to move that 70% around, and is focusing on ways to reduce typical running costs so that more money is available for R&D.

The three biggest VMware initiatives at the moment are:

1. Foundation for the cloud. This is an area where the Virtual Data Center OS (VDC-OS) will play a big role. VDC-OS will be about service and policy (namely availability, security and scalability) on one side and about aggregation (with products like vCompute, vStorage, vNetwork) on the other side. VDC-OS should become the new software mainframe.
2. Choice and cloud federation. This is all about vCloud working on and with standards to provide high interoperability between in-house and external clouds.
3. Desktop as a service. The key is to provision to users, not to provision devices. Depending on the type of worker, users could get a virtual desktop or a client-side hypervisor.

Maritz also announced the new name for VDC-OS. It’s officially vSphere. The name didn’t really come as a surprise because rumors have been humming for a few weeks now, but finally all those people under a non-disclosure agreement can shout it out loud:  VSphere is here! (But not entirely, the name may be official but the product isn’t here just yet.)

Three more announcements Maritz made:

1. Collaboration with Teradici. On the desktop side, VMware has announced the collaboration with Teradici to improve the VDI protocol. Tomorrow, VMware will announce the VMware Client Hypervisor.
2. VCenter Server Heartbeat. Maritz announced a new vCenter Server add on called vCenter Heartbeat. Through vCenter Heartbeat it will be possible to make your vCenter fully redundant. VCenter Heartbeat is to be expected in mid March.
3. Additional security. VMware vShield zones will make it possible to deliver security and compliance to the internal cloud.

As a consultant, these are all products that I expect to see at customer’s sites very soon, and I welcome the product announcements.

These releases should give VMware a new boost in the battle for the data center. Bring it on!