SQL Server

VMware adds another Windows user to its customer list

VMware, Inc. announced last week that another Windows customer is using VMware Infrastructure 3 (VI3) instead of Microsoft Hyper-V to consolidate servers and reduce costs.

Independence Blue Cross (IBC), the largest health insurer in Philadelphia, has grown quickly in recent years, and their  computing demands and costs have grown along with its business. Physical server sprawl and increasing power consumption has plagued the hospital and the cost of acquiring and managing new hardware was growing out of control, VMware reported.

To reverse these issues, IBC turned to virtualization. The company looked at Microsoft Hyper-V, but ultimately chose VMware because “it offered a more complete solution and robust tool set, rather than simply a hypervisor,” VMware’s spokesperson said on behalf of the customer. Another plus for VMware was that is offers VMotion to live migrate virtual machines (VM), which Micrsoft’s Hyper-V product won’t offer until the next version, as well as high availability,  resource pooling, manageability and automation, VMware said.

So far, IBC’s Windows application environment is approximately 70% virtualized, including applications like Active Directory, Exchange, SharePoint and SQL Server, PeopleSoft and Oracle 9i. There are 386 VMs are running on 48 physical hosts, and CPU utilization has increased from 5% to 75%, VMware reported.

Michael Garber, director of distributed infrastructure, at IBC, stated in the release that VI3 paid for itself in less than 16 months and helped IBC avoid more than $1 million in hardware costs.

VMware currently has over 3,000 hospitals on its list of customers, according to VMware.

VMware release lots of customer case studies to show the world how great they are, but when they announce Windows users as customers, Microsoft Hyper-V takes a bullet.  Hyper-V is built right in to Windows Server 2008, so why wouldn’t a Windows user just virtualize with Hyper-V? That’s Microsoft’s argument, and it looks like people aren’t buying it.

There is a ton of speculation on whether Hyper-V will be able to surpass VMware in the virtualization market, but I haven’t seen anything from Microsoft (like Hyper-V customers!) signaling that possibility.

VirtualCenter 2.5 database issue during upgrade

Last week, I wrote about about the database index defragmentation. I discussed reading the scan index and that if the percentage is poor, a defragmentation or index rebuild would be in line. There is more to the database and VirtualCenter 2.5 upgrade story, as I have found out, regarding special permissions. Today I’ll share those my findings with anyone else who may have upgraded to VirtualCenter 2.5 quickly.

The good news is that there is now a VMware KB article about this topic in the known issues section of the release notes. When I upgraded in December of 2007, this was not available. My issue was that although I had correct permissions with the username and password in SQL authentication to the VirtualCenter database, this account did not have the correct permissions to create the SQL jobs. VirtualCenter 2.5 creates three default SQL Agent jobs to manage statistics:

-Past Day stats rollup
-Past Week stats rollup
-Past Month stats rollup

These jobs move data from the VPX_HIST_STAT1 system to the VPX_HIST_STAT2, VPX_HIST_STAT3, VPX_HIST_STAT4 and eventually out of the system. If you upgraded your VirtualCenter with an account that did not have the ability to create these jobs, they likely are not running. The easy indicator is the VPX_HIST_STAT1 table will have millions of records, and the other VPX_HIST_STATx tables will have no records.

I had to call VMware support to confirm this, and once we noticed that the jobs were not present the solution was clear. However, the job took a long time to catch up on the statistics management.

The unfortunate situation is that the VirtualCenter install does not give an error if these jobs cannot be created.

VirtualCenter 2.5 database index defragmentation

I, like many virtualization administrators, have worked very hard to get the VMware virtual environment set up and running as expected. Now, one of my main tasks is to make sure that we do not do anything to adversely effect server performance. A good place to start in this regard is the VirtualCenter (VC) database. That being said, the VC database is critically important to a successful ESX implementation, so do not do anything that is not advised by VMware documentation or support services. Let’s discuss index defragmentation in particular here when using Microsoft SQL server 2000 for the VC database.

Index defragmentation on statistics

I will save you some work in what to look for in determining which tables will need index defragmentation - statistics. While we all like the statistics and graphing options available in the VMware Infrastructure Client and virtual appliances that may use the table, there can be a great amount of data in that table and it can quickly become fragmented. A fragmented index in a database is similar to a fragmented file system where the ordering of an index is not in the order of the index.

In my VC 2.5 environment, the VPX_HIST_STAT1 table is the heavy hitter. For this database maintenance, I’m going to start with the white paper entitled “VirtualCenter Database Maintenance” available from the VMware website. Here there is a command to check your current fragmentation level:

USE
GO
DBCC SHOWCONTIG (VPX_HIST_STAT1)
GO

I have modified the command to use the table name, as the white paper is VC 2.0 based on the table name, whereas this example is on VC 2.5. The result will look something like the following:

DBCC SHOWCONTIG scanning ‘VPX_HIST_STAT1′ table…
Table: ‘VPX_HIST_STAT1′ (800721905); index ID: 1, database ID: 16
TABLE level scan performed.
- Pages Scanned…………………………..: 505458
- Extents Scanned…………………………: 78097
- Extent Switches…………………………: 457307
- Avg. Pages per Extent……………………: 7.4
- Scan Density [Best Count:Actual Count]…….: 22.34% [113183:905308]
- Logical Scan Fragmentation ………………: 3.81%
- Extent Scan Fragmentation ……………….: 0.47%
- Avg. Bytes Free per Page…………………: 187.2
- Avg. Page Density (full)…………………: 97.69%
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

The key takeaway is the Scan Density percentage. The white paper advises that a number close to 100% is good, meaning that the table index in the example above is quite fragmented. The white paper goes on to identify two correction levels for improving the index. Index defragmentation and, more aggressively, rebuild are the standard options to address the index. If the scan density after a index defragmentation does not do enough to improve the index, database admins will have to begin the rebuild process. A caveat: rebuilding requires VC downtime to perform the database maintenance.

By comparison, here the same command on the VPX_EVENT table. This table is busy, but not near as much as the statistics table:

DBCC SHOWCONTIG scanning ‘VPX_EVENT’ table…
Table: ‘VPX_EVENT’ (36195179); index ID: 1, database ID: 16
TABLE level scan performed.
- Pages Scanned…………………………..: 594
- Extents Scanned…………………………: 86
- Extent Switches…………………………: 132
- Avg. Pages per Extent……………………: 6.9
- Scan Density [Best Count:Actual Count]…….: 66.39% [75:133]
- Logical Scan Fragmentation ………………: 6.73%
- Extent Scan Fragmentation ……………….: 98.84%
- Avg. Bytes Free per Page…………………: 150.5
- Avg. Page Density (full)…………………: 98.14%
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

This table is in better shape, but is much smaller than the statistics table.

Configure statistics logging level

As virtualization administrators, one safeguard we can perform is to limit the logging levels within the VMware Infrastructure Client. To access the logging levels select Administration menu, then VirtualCenter Management Server Configuration, then Statistics. Here you want to limit the number of high level logging to keep the VPX_HIST_STATx tables in check:

In selecting which level works best for your environment, be sure to identify any monitoring tools or virtual appliances that may read the selected tables. Also be sure to benchmark your database size and index fragmentation to see if you gain any improvements. Identifying the parts of the entire VMware Infrastructure environment that you can keep in maintenance mode will make your job as a virtualization administrator much easier.

Did VMware DRS destroy my virtual disk?

I had a peculiar situation come up recently in my VirtualCenter 2.5 environment with ESX 3.02 hosts in this particular environment. I posted the situation on the VMware Communities forum, but have yet to get a clear answer on what happened. Here is my account of what happened so you can either help me out or at least protect yourself against this same situation:

-Server maintenance for a separate task was being performed on all of my ESX hosts (seven systems). One system was put into maintenance mode at a time.
-Once I had system #2 out of maintenance mode, I cloned a particular guest virtual machine. This clone was put locally as it was intended only as a test/failback guest operating system.
-After the clone was created, VirtualCenter migrated the virtual machine from one host to another. (Remember this virtual machine has local storage now)
-After that migration, the virtual machine could not power on because the virtual disk did not migrate with the virtual machine. The virtual disk was also removed from the inventory of the guest machine.

Once this occurred, I looked for the virtual disk file, and was intending to add it back to the inventory. Further, if I needed to move the virtual machine back to the original host, that was an option as well that did not correct the situation. I could not find the virtual disk anywhere on any of the shared storage resources or local storage resources. I ran the following SQL query on the Virtual Center database:

select EVENT_TYPE,CREATE_TIME,USERNAME,VM_NAME,HOST_NAME
from VPX_EVENT where
CREATE_TIME > ‘2008-3-3 00:45:00.000′ and
CREATE_TIME CREATE_TIME < ‘2008-3-3 02:00:00.000′

It goes without saying that you should not proceed into the SQL database without the proper cautions. This query takes a look into the events leading up to the error that occurred when I was attempting to power on the virtual machine. Here are the result that let to the demise of my virtual machine that I placed in a spreadsheet:

I have stripped out the entries that were not pertinent to this virtual machine, and you can see that the vim.event.VmBeingRelocatedEvent occurred after my cloning, and remember this guest had its virtual disk on the local storage of esx-2.intra.net after I cloned it from esx-4.intra.net. Note that the first two yellow entries do not have my admin username being used - thus implying VirtualCenter or DRS is the culprit! You may wonder about the timing of the last three events, there are about three seconds between the vim.event.VmRelocatedEvent and vim.event.VmStartingEvent entries. The scrolling log of the VMware infrastructure client would have completed migration by the vim.event.VmRelocated message on the migration before the virtual power was applied.

Has anyone had this scenario in your virtual environments? As a short term measure I am not putting anything important (even backups as in this case) on local storage. When I get to the bottom of this behavior, I will surely follow up here with what happened.