Posted by: Rick Vanover
Rick Vanover, SQL Server, VI3, Virtualization, VMware ESX
I had a peculiar situation come up recently in my VirtualCenter 2.5 environment with ESX 3.02 hosts in this particular environment. I posted the situation on the VMware Communities forum, but have yet to get a clear answer on what happened. Here is my account of what happened so you can either help me out or at least protect yourself against this same situation:
-Server maintenance for a separate task was being performed on all of my ESX hosts (seven systems). One system was put into maintenance mode at a time.
-Once I had system #2 out of maintenance mode, I cloned a particular guest virtual machine. This clone was put locally as it was intended only as a test/failback guest operating system.
-After the clone was created, VirtualCenter migrated the virtual machine from one host to another. (Remember this virtual machine has local storage now)
-After that migration, the virtual machine could not power on because the virtual disk did not migrate with the virtual machine. The virtual disk was also removed from the inventory of the guest machine.
Once this occurred, I looked for the virtual disk file, and was intending to add it back to the inventory. Further, if I needed to move the virtual machine back to the original host, that was an option as well that did not correct the situation. I could not find the virtual disk anywhere on any of the shared storage resources or local storage resources. I ran the following SQL query on the Virtual Center database:
from VPX_EVENT where
CREATE_TIME > '2008-3-3 00:45:00.000' and
CREATE_TIME CREATE_TIME < '2008-3-3 02:00:00.000'
It goes without saying that you should not proceed into the SQL database without the proper cautions. This query takes a look into the events leading up to the error that occurred when I was attempting to power on the virtual machine. Here are the result that let to the demise of my virtual machine that I placed in a spreadsheet:
I have stripped out the entries that were not pertinent to this virtual machine, and you can see that the vim.event.VmBeingRelocatedEvent occurred after my cloning, and remember this guest had its virtual disk on the local storage of esx-2.intra.net after I cloned it from esx-4.intra.net. Note that the first two yellow entries do not have my admin username being used – thus implying VirtualCenter or DRS is the culprit! You may wonder about the timing of the last three events, there are about three seconds between the vim.event.VmRelocatedEvent and vim.event.VmStartingEvent entries. The scrolling log of the VMware infrastructure client would have completed migration by the vim.event.VmRelocated message on the migration before the virtual power was applied.
Has anyone had this scenario in your virtual environments? As a short term measure I am not putting anything important (even backups as in this case) on local storage. When I get to the bottom of this behavior, I will surely follow up here with what happened.