March, 2008

VMware security coming to the forefront?

Many virtualization analysts punt on the issue of security. But two recent events have brought security into higher relief: the uncovering of VMware’s file-sharing security flaw and VMware’s announcement of VMsafe, a virtual appliance that adds a layer of security to apps running on virtual machines. While VMsafe attempts to address VMware’s file-sharing problems, the flaw has raised questions about VMware security and the security of virtualization technologies in general.

In a recent SearchSecurity.com article, one interviewee said that after testing virtualization, he determined that putting virtualization into production would require reworking tried-and-true centralized security controls. Another interviewee expressed concerns about future problems, particularly a breach involving the hypervisor.

But we aren’t the only ones asking questions about security for virtual environments. At Rational Suvivability, author Christopher Hoff takes a different angle:

Virtualization up until now has quietly marked a tipping point where we see the disruption stretch security architectures and technologies to their breaking point and in many cases make much of our invested security portfolio redundant and irrelevant.

Has virtualization brought a whole new set of security requirements? Has your company explored or purchased virtualization-specific security software? Share your security-in-virtual-environments experience, and we’ll send you a $10 Starbucks gift card. Email me at hdrake@techtarget.com.

VirtualCenter 2.5 database issue during upgrade

Last week, I wrote about about the database index defragmentation. I discussed reading the scan index and that if the percentage is poor, a defragmentation or index rebuild would be in line. There is more to the database and VirtualCenter 2.5 upgrade story, as I have found out, regarding special permissions. Today I’ll share those my findings with anyone else who may have upgraded to VirtualCenter 2.5 quickly.

The good news is that there is now a VMware KB article about this topic in the known issues section of the release notes. When I upgraded in December of 2007, this was not available. My issue was that although I had correct permissions with the username and password in SQL authentication to the VirtualCenter database, this account did not have the correct permissions to create the SQL jobs. VirtualCenter 2.5 creates three default SQL Agent jobs to manage statistics:

-Past Day stats rollup
-Past Week stats rollup
-Past Month stats rollup

These jobs move data from the VPX_HIST_STAT1 system to the VPX_HIST_STAT2, VPX_HIST_STAT3, VPX_HIST_STAT4 and eventually out of the system. If you upgraded your VirtualCenter with an account that did not have the ability to create these jobs, they likely are not running. The easy indicator is the VPX_HIST_STAT1 table will have millions of records, and the other VPX_HIST_STATx tables will have no records.

I had to call VMware support to confirm this, and once we noticed that the jobs were not present the solution was clear. However, the job took a long time to catch up on the statistics management.

The unfortunate situation is that the VirtualCenter install does not give an error if these jobs cannot be created.

The Clone Wars

I started to call this post “Attack of the Clones”, but I was a bit worried that George Lucas might get upset at my use of his movie title. So, while sticking to the Star Wars theme, I settled on the current title.

The idea of leveraging the cloning functionality that’s present in many storage arrays on the market today is not a new or unusual one. In particular, on my personal blog I wrote a number of articles about the processes around using clones, the advantages of using clones, and some of the disadvantages of using clones. While my articles were primarily focused around storage systems from Network Appliance (now just NetApp), the basic principles are very similar for other storage arrays as well.

In How to Provision VMs Using NetApp FlexClones, I discussed the processes and procedures around the use of hardware-based clones. In particular, new functionality within ESX Server 3.x required administrators to enable resignaturing in order to see cloned VMFS datastores and be able to use the virtual machines stored in those datastores. At the time, there was no automated way of registering the VMs stored in the cloned datastores as well, and I believe that is still true even today.

Having discussed the “how” of using clones, I moved on to a couple of articles discussing the advantages and disadvantages of using storage system clones. In NetApp FlexClones with VMware, Part I, I presented the advantages of using clones:

• Reduced storage usage
• Reduced time to create VMs

In NetApp FlexClones with VMware, Part 2, I moved on to some of the disadvantages of using storage system-based clones:

• Scalability issues with regards to a maximum number of LUNs supported within VirtualCenter
• Lack of integration with VMware’s graphical tools
• Potential blurring of responsibilities across functional IT teams

Clearly, it’s up to each customer to determine whether the advantages and disadvantages are truly applicable to their organization. For some organizations, the “potential blurring of responsibilities” may be a non-issue, and the reduced storage requirements are a major issue.

Of course, it’s also possible to use hardware-based clones for purposes other than just provisioning. In VM File-Level Recovery with NetApp Snapshots and Full VM Recovery with NetApp Snapshots, I discussed ways to leverage NetApp FlexClones and LUN clones–which are based on Snapshots–to facilitate VM recovery scenarios. So this technology can be used for a variety of purposes in VMware environments.

If you’re a reader whose using hardware-based clones or snapshots — not necessarily Network Appliance-based, but from any hardware vendor — are you using this functionality? How are you leveraging it in your environment? I’d love to hear about the ways in which you are putting these kinds of technologies to work in the real world.

VirtualCenter 2.5 database index defragmentation

I, like many virtualization administrators, have worked very hard to get the VMware virtual environment set up and running as expected. Now, one of my main tasks is to make sure that we do not do anything to adversely effect server performance. A good place to start in this regard is the VirtualCenter (VC) database. That being said, the VC database is critically important to a successful ESX implementation, so do not do anything that is not advised by VMware documentation or support services. Let’s discuss index defragmentation in particular here when using Microsoft SQL server 2000 for the VC database.

Index defragmentation on statistics

I will save you some work in what to look for in determining which tables will need index defragmentation - statistics. While we all like the statistics and graphing options available in the VMware Infrastructure Client and virtual appliances that may use the table, there can be a great amount of data in that table and it can quickly become fragmented. A fragmented index in a database is similar to a fragmented file system where the ordering of an index is not in the order of the index.

In my VC 2.5 environment, the VPX_HIST_STAT1 table is the heavy hitter. For this database maintenance, I’m going to start with the white paper entitled “VirtualCenter Database Maintenance” available from the VMware website. Here there is a command to check your current fragmentation level:

USE
GO
DBCC SHOWCONTIG (VPX_HIST_STAT1)
GO

I have modified the command to use the table name, as the white paper is VC 2.0 based on the table name, whereas this example is on VC 2.5. The result will look something like the following:

DBCC SHOWCONTIG scanning ‘VPX_HIST_STAT1′ table…
Table: ‘VPX_HIST_STAT1′ (800721905); index ID: 1, database ID: 16
TABLE level scan performed.
- Pages Scanned…………………………..: 505458
- Extents Scanned…………………………: 78097
- Extent Switches…………………………: 457307
- Avg. Pages per Extent……………………: 7.4
- Scan Density [Best Count:Actual Count]…….: 22.34% [113183:905308]
- Logical Scan Fragmentation ………………: 3.81%
- Extent Scan Fragmentation ……………….: 0.47%
- Avg. Bytes Free per Page…………………: 187.2
- Avg. Page Density (full)…………………: 97.69%
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

The key takeaway is the Scan Density percentage. The white paper advises that a number close to 100% is good, meaning that the table index in the example above is quite fragmented. The white paper goes on to identify two correction levels for improving the index. Index defragmentation and, more aggressively, rebuild are the standard options to address the index. If the scan density after a index defragmentation does not do enough to improve the index, database admins will have to begin the rebuild process. A caveat: rebuilding requires VC downtime to perform the database maintenance.

By comparison, here the same command on the VPX_EVENT table. This table is busy, but not near as much as the statistics table:

DBCC SHOWCONTIG scanning ‘VPX_EVENT’ table…
Table: ‘VPX_EVENT’ (36195179); index ID: 1, database ID: 16
TABLE level scan performed.
- Pages Scanned…………………………..: 594
- Extents Scanned…………………………: 86
- Extent Switches…………………………: 132
- Avg. Pages per Extent……………………: 6.9
- Scan Density [Best Count:Actual Count]…….: 66.39% [75:133]
- Logical Scan Fragmentation ………………: 6.73%
- Extent Scan Fragmentation ……………….: 98.84%
- Avg. Bytes Free per Page…………………: 150.5
- Avg. Page Density (full)…………………: 98.14%
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

This table is in better shape, but is much smaller than the statistics table.

Configure statistics logging level

As virtualization administrators, one safeguard we can perform is to limit the logging levels within the VMware Infrastructure Client. To access the logging levels select Administration menu, then VirtualCenter Management Server Configuration, then Statistics. Here you want to limit the number of high level logging to keep the VPX_HIST_STATx tables in check:

In selecting which level works best for your environment, be sure to identify any monitoring tools or virtual appliances that may read the selected tables. Also be sure to benchmark your database size and index fragmentation to see if you gain any improvements. Identifying the parts of the entire VMware Infrastructure environment that you can keep in maintenance mode will make your job as a virtualization administrator much easier.

Did VMware DRS destroy my virtual disk?

I had a peculiar situation come up recently in my VirtualCenter 2.5 environment with ESX 3.02 hosts in this particular environment. I posted the situation on the VMware Communities forum, but have yet to get a clear answer on what happened. Here is my account of what happened so you can either help me out or at least protect yourself against this same situation:

-Server maintenance for a separate task was being performed on all of my ESX hosts (seven systems). One system was put into maintenance mode at a time.
-Once I had system #2 out of maintenance mode, I cloned a particular guest virtual machine. This clone was put locally as it was intended only as a test/failback guest operating system.
-After the clone was created, VirtualCenter migrated the virtual machine from one host to another. (Remember this virtual machine has local storage now)
-After that migration, the virtual machine could not power on because the virtual disk did not migrate with the virtual machine. The virtual disk was also removed from the inventory of the guest machine.

Once this occurred, I looked for the virtual disk file, and was intending to add it back to the inventory. Further, if I needed to move the virtual machine back to the original host, that was an option as well that did not correct the situation. I could not find the virtual disk anywhere on any of the shared storage resources or local storage resources. I ran the following SQL query on the Virtual Center database:

select EVENT_TYPE,CREATE_TIME,USERNAME,VM_NAME,HOST_NAME
from VPX_EVENT where
CREATE_TIME > ‘2008-3-3 00:45:00.000′ and
CREATE_TIME CREATE_TIME < ‘2008-3-3 02:00:00.000′

It goes without saying that you should not proceed into the SQL database without the proper cautions. This query takes a look into the events leading up to the error that occurred when I was attempting to power on the virtual machine. Here are the result that let to the demise of my virtual machine that I placed in a spreadsheet:

I have stripped out the entries that were not pertinent to this virtual machine, and you can see that the vim.event.VmBeingRelocatedEvent occurred after my cloning, and remember this guest had its virtual disk on the local storage of esx-2.intra.net after I cloned it from esx-4.intra.net. Note that the first two yellow entries do not have my admin username being used - thus implying VirtualCenter or DRS is the culprit! You may wonder about the timing of the last three events, there are about three seconds between the vim.event.VmRelocatedEvent and vim.event.VmStartingEvent entries. The scrolling log of the VMware infrastructure client would have completed migration by the vim.event.VmRelocated message on the migration before the virtual power was applied.

Has anyone had this scenario in your virtual environments? As a short term measure I am not putting anything important (even backups as in this case) on local storage. When I get to the bottom of this behavior, I will surely follow up here with what happened.

Coming soon: Application High Availability (AHA)

Scott Lowe recently blogged about things that he loves but are not yet available. VMware, it seems, is getting as bad as Apple in announcing products far ahead of their release dates. This generates great market buzz and sends stock prices soaring (or in the case of VMworld Europe leaving them as low as they have been), but it does little to satiate the desires of systems administrators, it only serves to increase want and provide little respite. Well, then I am about to send you into the Sahara with the promise of an oasis, only to find a sign-post that says “Coming Soon!” at the end of your journey.

Please join me in anticipation of VMware Application High Availability (AHA).

AHA has not been announced by VMware, but it is coming. How do I know this? It is next logical step for their HA portfolio. ESX 3 brought us server HA. 3.5 introduced us to VM HA. And the recent announcement of the VMsafe all but secures the eventual release of AHA.

What is AHA? Simply put, you will be able to right-click on a VM from the VI client and indicate that if Microsoft Exchange fails then the Exchange service should be restarted, or the VM should be failed over to another ESX server. Or perhaps you want to monitor the Apache web server — just check a box. How will VMware achieve this level of fine-grained control? Allow me to refer to the VMsafe product page:

Process execution
VMsafe provided in-guest, in-process APIs that enable complete monitoring and control of process execution.

This API will allow VMware to monitor and control processes within the guest OS. That, my friends, is how AHA will work.

AHA will allow VMware to take even more market share away from companies like Sun and Microsoft, both who have their own clustering technologies. Why cluster at the OS-specific level when you can create clusters the same way no matter the OS or application underneath!

VMsafe is set to debut later this year, and I am quite certain that alongside VMsafe, or shortly thereafter, we will see VMware announcing and releasing its application level high availability software. I hope you aren’t too parched : )