VirtualCenter is an eager beaver. It really wants to start when your server does. The problem is that it starts too soon if it is installed on the same server as Microsoft SQL. The Windows event log will record two types of errors: the first will be in the Application log and detail how Microsoft SQL server is not ready to accept connections:
Event Type: Error Event Source: MSSQLSERVER Event Category: (4) Event ID: 17187 Date: 1/31/2008 Time: 12:10:26 AM User: N/A Computer: PUKKLE Description: SQL Server is not ready to accept new client connections; the connection has been closed. Wait a few minutes before trying again. If you have access to the error log, look for the informational message that indicates that SQL Server is ready before trying to connect again. [CLIENT: 192.168.0.47] For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 23 43 00 00 10 00 00 00 #C...... 0008: 07 00 00 00 50 00 55 00 ....P.U. 0010: 4b 00 4b 00 4c 00 45 00 K.K.L.E. 0018: 00 00 00 00 00 00 ......
The second error is in the System log and notifies the server administrator that the VirtualCenter service (vpxd) failed to start.
Event Type: Error Event Source: Service Control Manager Event Category: None Event ID: 7024 Date: 1/31/2008 Time: 12:11:02 AM User: N/A Computer: PUKKLE Description: The VMware VirtualCenter Server service terminated with service-specific error 2 (0x2). For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
It would be incredibly annoying to have to manually start the VirtualCenter service each time your server boots. Luckily there is an easy fix — make the VirtualCenter service depend on the Microsoft SQL service. Microsoft has posted instructions for creating service dependencies on their site.
In short find the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vpxd and edit the DependsOnService property. Add two entries to this REG_SZ_MULTI value:
Save the changes. Now the next time the server boots the VirtualCenter service will not attempt to start until the Microsoft SQL service and SQL Server Agent service come online, sparing you of those annoying errors in the event log and having to restart the service manually.
Hope this helps!
I am blogging from the Technosium Global Conference and Expo in Santa Clara, California. While here, I will be providing you important VMware info from the show.
VMware Update Manager is now shipping with the ability to patch offline virtual machines. This is done by using a special patching network for suspended or offline systems. Any organization implementing any network access control or that would be concerned by a system at risk by presenting itself to the main network.
Offline patching uses the Shavlik patch management technology to interact with Update Manager and the virtual machine. The patching capabilities currently include Windows virtual machines for Microsoft, Adobe, and Mozilla updates and a scanning only capability for Linux systems.
VMware Infrastructure 3.5 (VI3.5) saw major updates last fall with the release of ESX 3.5, VirtualCenter 2.5, and ESX 3i. Just what do these new products mean for IT administrators? What functionality have these three SKUs added to the VI3 bundle? Mike Laverick from RTFM-Education fame produced a great 88 page PDF that details the updated features at on his blog. Eighty-eight pages is a long read, so I decided to collapse the material into the most important points that readers need to take from Mike’s awesome document. So here they are, the 18 changes you need to know about:
– The default mode for port groups created via Kickstart is now “Active/Standby” instead of route-based-on-virtual-port-ID, despite the assertion by the installer that the latter option is the default.
– Enabling the iSCSI initiator via Kickstart no longer opens the iSCSI port (3260).
– You can now load-balance HBAs via a new method, albiet experimental, Round-Robin.
– ESX now supports a boot-from-SATA option.
– Using N_Port ID Virtualization (NPIV) you can now assign WWNs to VMs directly.
– The “Create a VM” wizard let’s you set a VM’s MAC address to a static value.
– You can finally set a boot delay (in MS) for a VM, giving you time to hit the ESC key in order to boot from a CD-ROM (or ISO image).
– You can turn on paravirtualization support for VMs that have guests (Linux only right now) that support it. Expect to see some performance increases for those guest OSs that do.
– There is an option to turn on MMU virtualization, allowing a VM to manage its own page tables. The host system must support the Nested Page Table or Extended Page Table processor extensions to be able to use this option.
– Yipee! You can finally resize a VM’s disks via the VI client (the guest OS must be able to claim the new space).
– Distributed Power Management (DPM) finds its way into ESX clusters allowing you to set rules similar to DRS that will put ESX servers into a power-saving stand-by mode if they are underutilized and could transition (re: VMotion) their VMs to other ESX servers in the cluster.
– Where once there was 16, there are now 32. That’s 32 ESX hosts per ESX cluster.
– The HA algorithm goes to school and actually searches for the ESX server with the most available resources to place VMs on instead of just looking down the alphabet.
– You can configure HA per VM failure by monitoring the VM with a heartbeat alarm.
– Storage VMotion allows you to move a VM to a new LUN while it is running. It is not that big of a deal, even though everyone seems to think it is (How often are you re-carving your SAN where there isn’t expected downtime?)
– VMware Update Manager (VUM) allows easy patching of ESX and VirtualCenter servers, as well as patching of some Windows and Linux guests.
– An exciting new feature is the ability to create plugins for the VirtualCenter client, such as the Guided Consolidation Manager or VUM.
– Jumbo frames!
For a full list of features I recommend you read Mike’s report as well as check out VMware’s changelog.
Today VMware announced the general availability of a new addition to their Virtual Desktop Infrastructure (VDI) — VMware Virtual Desktop Manager 2 (VDM2). Some of the new features of VMD2 include:
– SSL secured connections
– Second-factor authentication with RSA SecureID
– Integration with ActiveDirectory for authentication and authorization
– Create persistent and non-persistent desktop pools for groups of users
– Increased clustering support for VDM2
The release of VDM2 continues VMware’s dominance of the virtual desktop market. With broad support from independent hardware vendors (IHVs), VDI is finally beginning to evolve into a product not only worth using, but one that businesses deploying numerous desktops should not be without.
For more information check out the VMware VDM2 product site.
VLAN trunking, or tagging, is a powerful feature that allows ESX virtual machines to connect to multiple networks. However, your network team may not want to enable that. In fact, I have come across many situations where trunking is not done to servers, but only to switches for VLAN availability. To break that mold, we have to start with education on the VMware networking technologies.
Most of us are not network admins as well as virtualization admins, so we have to start with some good resources to “get what we want.” Without VLAN trunking, each VLAN you would want to connect to would require its own physical interface. That clearly is a growth and agility inhibitor as well as incredibly expensive for additional cable runs and interfaces on your ESX servers. Quoting Andrew Kutz, “that makes ESX practically worthless.” So I am on a mission to turn the tide. There is a good resource online from VMworld 2006 that explains the ESX networking so that we can inform ourselves, and better integrate with our network staff to ensure all standards are met.
With networking, configuration consistency among ESX servers becomes absolutely critical (as is with other areas of VI3). It is important to deliver the ESX configuration as planned, as if there is a variation and an outage or other networking issue, there may be a swift change of heart from your networking staff.
Readers, I invite you to share your experiences with making the case for trunking below with a comment.
Besides being an IT Knowledge Exchange Blogger and SearchVMware.com author with VCP and CCIE certifications, I also create video training courses. Recently, I created a video training course covering VMware ESX Server, VirtualCenter and the VMware Infrastructure Suite. This is a 15-hour video training course covering every aspect of VMware ESX Server – from design, to installation, to virtual center configuration, SAN configuration, P2V conversions, backups, performance tuning, and troubleshooting.
With VMware training being very costly and there being so much to know about virtualization and VMware ESX Server, I believe that this video training course is an excellent method anyone wanting to learn about ESX Server. Really, you don’t have to learn ESX Server “the hard way” — by reading the documentation & trial and error — this video training course makes learning ESX Server easy.
I’m eager to hear your feedback, so please let me hear from you in comments on this post.
Have you ever wondered why VMware’s VI client is Windows only? The most obvious reason is that it is written in C# using the .NET framework. However, the client could easily be written in Python or Java or some other OS agnostic language. So why Windows only? There are in fact two reasons the VI client is Windows only, let’s take a look at them.
“I’ll be your Huckleberry”
VMware apparently uses a C library called “TomSawyer” to handle the task of graphing performance output. Digging into a disassembled VI client, you can see where the Interop assembly is used and what it is used for. Using the Microsoft OLE/COM viewer to peer directly into TomSawyer’s library certain functions and strings like “GridColumnObject” and “TSEFlowChartType” give away the library’s purpose. I cannot think of a reason to marshal a library that graphs datasets unless VMware thought .NET was not up to the task of plotting data. However, the performance graphs displayed in a VI client take so long to initialize anyway that I do not think anyone one have noticed. As Barney on HIMYM says, “JIT up!”
Sorry, I did not have a witty title for this section The other reason that the VI client is Windows only appears to be the library that it uses for the console view. The MKS (mouse, keyboard, sound) library is what the VI client marshals when it creates the console connection directly to the VM. Notice the absence of a V? I do too. Although the library’s name does not explicitly state that it is responsible for video, the functions inside of it, FullScreen, SetFullScreen, SetGuestSize, etc., seem to indicate that it *is* the responsible library.
TomSawyer can certainly be ported to run natively in .NET or another language (it does already in fact have a Java version), but QuickMKS may be a bit more difficult. It almost certainly relies on native OS calls that are most likely quite difficult to get working on other systems. However, if VMware would be so kind to send the source code my way, I’d sure like to try!
Rick Vanover recently wrote a terrific blog on how to monitor the VI database for authentication attempts. While you can certainly monitor the database directly, VMware does not support this option. A supported alternative would be using the SDK to collect historical authentication events and monitor the VI servers in real-time for new ones.
Collecting historical authentication events is quite easy. I whipped up a quick Perl script that shows you how to do just that. printloginhistory.pl will print out a VI server’s authentication events with the ability to filter on a specific user. The script is extremely short and easy to modify.
In order to collect authentication events in real-time, you need to monitor the latest events as they occur. Luckily I also have an arsenal in my handy-dandy utility belt that does just this – Monét.
As Rick’s and my methods show, there is almost always more than one way to accomplish a task. Just pick the one that you’re most comfortable with and forge ahead.
I recently upgraded to ESX 3.5 on a test system, and had an issue that was really stressing me out. The issue was that each time I would perform the “Reconfigure for HA” task, I had errors causing the task to fail and the new ESX host sits there with a red triangle like a car broken down on the side of the highway. The log message that runs in the Virtual Infrastructure Client was largely useless, so I jump into the VMware ESX database. For this situation, I have looked into the VPX_EVENT table and saw the following event:
This message gave me a starting point and I found that in my service console network configuration the DNS suffix order was not the same for all hosts. Specifically, I forgot one DNS suffix in the order and that name made sense to me:
Therfore, the takeaway is when building ESX servers, ensure the configuration for all hosts in a cluster (or datacenter) within Virtual Center have the correct configuration where relevant for DNS, subnet mask, interface naming, and storage configuration.
Because VMware ESX and Virtual Center (VC) have great magnitude in the datacenter, I determined it would be a good idea to have an audit trail of authentication attempts in to the Virtual Infrastructure Client and SSH on the host. In my recently upgraded VC 2.5 environment, I made a quick trip to the database to query some of the authentication events. Here are some queries of the VC database running on Microsoft SQL Server that may be useful.
Failed authentication attempts
This query in Query Analyzer will show the failed authentication attempts into VC or the ESX host:
[Using VMware Database]
select EVENT_TYPE, USERNAME, CREATE_TIME, HOST_NAME
Where EVENT_TYPE = 'vim.event.BadUserNameSessionEvent'
This will show you the failed authentication attempts. What you want to look for is perpetual attempts, or attempts from usernames that you are not expecting to log into VC. If you want to run the same query with all fields, replace the EVENT_TYPE, CREATE_TIME, HOST_NAME with a ‘*’ or add additional criteria with time conditioning. You may consider putting in SQL monitors or alerts for this condition – or simply making a daily report for the failed authentication attempts that is accessible for audit purposes. Should you also have authentication attempts to the ESX host (SSH), those attempts would be failed and in this query result.
Successful authentication attempts
Just as it is important to monitor failed authentication attempts, you may have a need to have an audit trail of successful connections. Within the VC database, this query would run showing successful logon events within the Virtual Infrastructure Client or directly to the ESX host:
select EVENT_TYPE, USERNAME, CREATE_TIME, HOST_NAME
Where EVENT_TYPE = 'vim.event.UserLoginSessionEvent'
This will show the successful results within the Virtual Infrastructure Client and any logon attempts via SSH to the ESX host. This can provide a solid audit trail with some SQL jobs or other reporting that you can do against the SQL database.
Database general rule of thumb for safety
Whenever you get into the database, use extreme caution. A good safe practice would be to back up the database and restore it new somewhere else, and practice all queries, jobs or reporting you want to do against the VC database. This way, once you have your monitoring elements safe and clearly defined you can roll them into your live environment confidently.