However VMworld hits the bottom rung of the ladder.  The VMworld content team simply provides this feedback.

Here is a list of the most common reasons why sessions were declined:

• There were too many submissions with similar topics.
• The submission was too basic, not enough information was provided in the abstract.
• The session was too single vendor product focused and likely to have a commercial nature.

Having the most common reasons why sessions weren’t picked isn’t exactly helpful.  Given that my abstracts have been accepted at conferences like TechEd and the SQL PASS Summit I can assume that the submission wasn’t too basic.  While my sessions are SQL Server specific, virtualizing SQL Server is a major push for VMware so I’m assuming that I don’t fall into the last bucket.  That leaves that there were just too many submissions, which I can’t believe as currently there are only three sessions on the schedule that talks about SQL Server (as of my writing this).  The first is about virtualizing SQL Server, and the second is how General Mills succeed in virtualizing SQL Server using VMware, and the third appears to be a repeat of a SQL Virtualization session from last year.

The first session is called “Virtualizing SQL High Availability” but the abstract doesn’t talk about SQL Server High Availability, it only talks about SQL Server Performance Tuning within a virtual machine which let’s be honest from a SQL Server perspective is identical to tuning SQL Server on a physical machine.  Make sure the disks are fast enough, that you have enough memory and CPU then do the normal SQL performance tuning.  Looking at the external speakers companies website he appears to be knowledgeable on databases in general, but he doesn’t list anything more current than SQL Server 7 in his publications list for SQL Server, and most of his publications are for Oracle.

The second session called “Virtualizing SQL 2012 at General Mills” looks basically like a fluff piece being led by a Technical Account Manager from VMware (Sales) and a VMware Architect. In other words, they’ll have no one around who can actually answer any SQL Server questions, or talk about SQL Server in depth.

The third session is called “Virtualizing SQL Best Practices” which I’m pretty sure I saw at VMworld last year, and if it is the same session as last year I really hope that it is better than the session last year.  When I saw the session by this same presenter last year the session was full of incorrect and very outdated information, much of which was based on SQL Server 2000 but was being presented for SQL Server 2008 and SQL Server 2008 R2.  If this is an updated version of last years session, I REALLY hope for the attendees sake that the session is updated and the incorrect information is updated.

Now am I writing this because I think I should have been handed a speaking slot at VMworld?  No.  I’m writing this because I think that VMworld needs to do a better service to their potential speakers so that the speakers can improve their abstracts based on VMworld’s requirements (as every conference has different criteria that they use when evaluating session abstracts).  I also feel that VMworld needs to do a better job providing SQL Server specific content to their attendees.  VMworld has 9 Oracle sessions listed (granted I think that only 7 of them are actually Oracle specific).

Hopefully this blog post won’t fall on deaf ears at VMworld and they will do something about their lack of SQL Server content as out of the three sessions listed, I personally don’t have much hope for any of them being very useful to the general VMworld attendee.

Before writing this blog post I emailed VMware asking for more information about why my sessions weren’t selected.  I got a pretty generic response back which doesn’t specify anything about why my specific sessions weren’t selected.

• Take the time to specify some of the details in your session description.  The title, abstract, outline, and session takeaways are key factors in determining whether your session is selected and can affect session attendance.

• Be sure your perspective or story is unique.

• Attendees are there to learn, so educate rather than pitch your product.  Your proposal submission should clearly reflect your intention.

• Make sure your topic is relevant to the audience you’re targeting. Review the content topics before submitting a session.

• Session selections are based on the content described in the submission, and the speaker(s). Speakers should have extensive experience presenting at conferences of a similar nature. If a speaker has presented at VMworld in the past, we will look at speaker scores and take them into consideration during selection.

Again not exactly the more useful information for me as an abstract writer to improve.

Denny

The first index is on the dbo.VPX_PROPERTY_BULLETIN database.  On the system that I happen to be looking at this week not having this index caused the SQL Server to scan this table 6977 times since the database was last restarted about 7 days before I began looking at the system.  This table on this SQL Server only contains about 3000 records, but this system is pretty small.  Just 4 hosts, 4 data stores and about 100 guests at the moment.  The larger this table is the more impact not having this query will have.

CREATE INDEX mrdenny_OPERATION_TYPE ON dbo.VPX_PROPERTY_BULLETIN
([OPERATION_TYPE])
INCLUDE ([OBJECT_MOID], [GEN_NUMBER])
WITH (FILLFACTOR=70, ONLINE=OFF, PAD_INDEX=ON)

The nice thing about this index is that is also fills the requirements of another index which is needed by the SQL Server.

The second index to create is built on the same table, but on different columns.  While the query which needs this index is run MUCH less often, SQL estimates (on this system at least) that adding it will improve the query performance by 69.94%.  In my mind that’s very much worth it, even if the query is only being run a few times a week.

CREATE INDEX mrdenny_OBJECT_MOID ON dbo.VPX_PROPERTY_BULLETIN
([OBJECT_MOID])
INCLUDE ([GEN_NUMBER])
WITH (FILLFACTOR=50, ONLINE=OFF, PAD_INDEX=ON)

The third index that I’ve identified which needs to be created is against the VPX_LIC_USAGE table.  This table has something to do with the licensing and the size of the table on your system will vary.  This vCenter system has over 16k rows in the table but this system has only been installed for a couple of months at this point.  As your vSphere installation ages it appears that this table will continue to grow and grow.  Best I can tell there’s a couple of rows entered into this table every hour for each host in the farm.  Needless to say this table will grow quite large when you’ve got a large VMware vSphere farm.

CREATE INDEX mrdenny_SAMPLE_TIMESTAMP ON dbo.VPX_LIC_USAGE
([SAMPLE_TIMESTAMP])
INCLUDE ([SAMPLE_TS])
WITH (FILLFACTOR=100, ONLINE=OFF)

As you look at these three indexes which I recommend that you create on your vSphere database you’ll notice that there is an ONLINE=OFF flag.  If your vCenter database is running the Enterprise Edition of SQL Server then you can change that on ONLINE=ON which will allow the indexes to be created online instead of causing blocking while the indexes are being created.  If you don’t have the Enterprise edition of SQL Server then you can’t create in the indexes online.  There should be no impact to the vCenter system if you create the indexes while the system is up and running.  The worse that will happen is that queries which are already running slowly will run a little slower than normal.

You’ll notice that I’ve listed this indexes to all start with mrdenny.  This is so that these indexes can be easily identified as coming from my blog (in case the next admin wonders where they are from) and so that you never have to worry about the index names colliding with names that VMware would try and use.

Denny

For a proper disk configuration you’ll want at least three different LUNs or local RAID arrays (I’ll assume you are SAN attached here, so if you are using local disk just replace LUN with RAID array).  These three LUNs should be RAID 1 (or RAID 10, 0+1, whatever your storage provider calls this).  One of these LUNs will host the Main Data File (the files that have a file extension of mdf).  Another one of these LUNs will host the transaction log file (the files that have a file extension of ldf).  The last LUN will be for the files for the tempdb database.

Now these three LUNs shouldn’t be sitting on the same physical disks (if you are using a 3PAR, IBM XIV, etc. that is shared everything you can probably ignore this part).  You especially don’t want the data file and transaction log file on the same LUN.  The reason for this is that both of these LUNs will be written to at the same time so having these on the same physical disks will cause contention at the disk level.  You don’t want the tempdb on the same disks as either of these other LUNs either.  Now if I had to pick one of these LUNs that I would put the tempdb LUN on the same physical disks with I could have it share spindles with the vCenter databases transaction log file.  My reason for this is because most of the time that tempdb database will be used by the vCenter database it’ll be reading data from the main vCenter databases and putting it into the tempdb database for processing.  So there should be less contention if the tempdb shares spindles with the transaction log file.  If possible keep all three LUNs on separate sets of spindles.

When sizing the LUNs don’t forget that the vCenter database will be keeping at least a years worth of data, so make sure that you give the system plenty of room.  At least 20 or 30 Gigs would be a good starting point.  VMware should have some sizing estimates that you can use on their website.  The bigger your VMware environment is the more disk space you’ll need.

When it comes to moving the vCenter databases to the new LUNs (assuming that you aren’t starting with a new vCenter database here) you’ll want to shutdown the vCenter services, then right click on the vCenter databases in SQL Server Management Studio and select all tasks, then detach.  Then move the databases to the correct location.  Then in SQL Server management studio right click on databases and select All Tasks then Attach database.  Locate the database in the new location in the popup window and attach the database.  Once the databases have been attached to the SQL Server instance you can restart the vCenter services to get vCenter up and running again.  This whole process should only take an hour or so depending on how large your vCenter databases are.

Don’t forget about the update manager database and cache folder.  You’ll probably want to put the update manager database on the same LUNs as the other vCenter databases, which is fine.  The cache how ever should go on another LUN as it’ll take a lot of space, and we don’t want all that space causing space problems for the vCenter database files.  Also you’ll probably want to use fast storage for the vCenter database as the data within that database is always changing and slower SATA storage for the update manager’s cache files as they are static are almost never written to or read from so you don’t want to waste expensive storage on storing gigs of update files.

Hopefully this information helps you build a more tuned VMware vCenter server.  If you have questions feel free to contact me, we can probably get them resolved pretty quickly.

However for larger farms lots of changes are probably being made to the configuration of the farm on a daily bases.  Because of this you won’t want to run the database in SIMPLE recovery.  Instead you’ll want to run the database in the FULL recovery mode.  This enables the ability to do point in time recovery of the database once you start doing database transaction log backups.  This means that when (note that I didn’t say “if”) the hard drives on the vCenter server fail and cause the database to be lost
of become corrupt you’ll only loose a few minutes of data at the most.

To ensure that you don’t loose a lot of data from the vCenter database you will need to begin doing transaction log backups.  This will do a couple of things.  First it’ll make it so that you can restore the database successfully.  Second it’ll make sure that the hard drive on your vCenter server doesn’t fill up with one giant transaction log file.  If you don’t have a DBA in your shop, and you’ve got a SQL Server with a license that you paid for (something other than the SQL Express that VMware installs if needed) you can setup the SQL Server Maintenance Plans to handle the backups using SQL Server Management Studio.  You can always bring in a SQL Server consultant to setup backups and make sure that everything is setup correctly and working if you don’t have a SQL Server DBA handy.  If you’ve got a DBA handy in your shop you’ll probably want to have them setup SQL Server backups as per your normal company policies so that the databases are backed up, and those backups tested to ensure that they are functional.

Personally I like to set the recovery mode for the vCenter databases to FULL when working at clients, but it’s totally up to you, as long as you understand what the risks are with each of the options available to you.  (Yes I know that I didn’t cover the BULK_LOGGED recovery model, for the purposes of VMware’s vCenter databases the BULK_LOGGED recovery model and the FULL recovery model are the same.)

Hopefully I have helped demystified the SQL Server backup concepts for you a little bit.

The big thing that you loose when doing this is that you can no longer restore the SQL Server database to a specific point in time if you have a system failure.  If you don’t need the ability to restore so a specific point in time, and restoring to your latest full or differential backup that you have (you do know that you need to backup the vCenter databases right?) then changing the recovery mode from FULL to SIMPLE is just fine.  However if you do need that ability (and I would recommend that you have that ability so that if the vCenter server fails or someone deletes data you can get back as much trending data as possible) then you will want to leave the recovery mode as FULL, but instead setup transaction log backups (hopefully you have a DBA or a database consultant that can help you with this) so that they run every few minutes (I typically recommend every 10-15 minutes).  Just keep in mind that when selecting how often they should run in the event of a server failure you may loose up to that amount of data from the database.  So if you aren’t aloud to loose more than 10 minutes worth of data, don’t backup the log every 15 minutes, back it up every 10, or maybe every 8 just to be safe.  Make sure that you get in writing from your management that this is the config that they want so if you get yelled at you are covered.

If you haven’t ever backed up the transaction log your disk is probably pretty full (or totally full).  In this case you will want to shrink the database just once.  But before you do you need to know what size you should shrink it to.  This can be much easier if a DBA does it, but I’ve got faith in you to not screw it up.

First connect to the SQL Server using Management Studio.  Take a backup of the transaction log to disk by right clicking on the database, selecting Tasks from the context menu then selecting Backup from the pop out menu.  Select transaction log backup, set a file to backup the log to (you’ll need a lot of disk space for this) and make sure that there’s only one file listed in the file list then start the backup.  This will probably take a while.

If you don’t care about loosing the log for now change the database from full to simple recovery and back.  Do this by running this script.

ALTER DATABASE vcdb
SET RECOVERY SIMPLE
GO
ALTER DATABASE vcdb
SET RECOVERY FULL
GO

Now do a full backup as shown above (just do a full database backup instead of a transaction log backup).

Now with everything running as normal wait for how ever long it’ll be between transaction log backups.  Then connect to the SQL Server in SQL Server Management Studio.  Right click on the vCenter database (vcdb) and select Reports from the context menu, then “Disk Usage” from the list of reports.  At the top it’ll give you the size of the transaction log in MB, and the graph on the right will show you a percentage breakdown of how full the transaction log file is.  Take the Used percentage and double it.  Now take that percentage and multiple it by the size of the transaction log.  That’s the minimum size you should set your transaction log for.  Personally I would round up to the 8 gig size above that size.  If you need 150 Megs of log space, set the log for 8 Gigs in size.  If you need 10 Gigs of log space, let the log for 16 Gigs of space.  A little extra space used is fine, it is better for performance of the database if we don’t need to grow this transaction log.  You can shrink the transaction log by using the DBCC SHRINKFILE statement as shown.

DBCC SHRINKFILE ('vbdc_log', 8388608)

Keep in mind that the number, in this case 8 Gigs, is passed in in megs.  Do NOT for any reason create any sort of scheduled task to regularly shrink any database file.  This will do nothing good and will cause performance problems of its own.  If anyone has told you to setup a job which shrinks the database file regularly they are WRONG.

Denny

Today was an interesting (and short) day.  On the final day of VMworld, VMware has decided that they won’t give a product related keynote.  Instead they will have a keynote that is pretty much unrelated to VMware and their technologies.  So today’s keynote was all about the human brain.  There were three PHDs who were speaking at the keynote, about 20 minutes per.  The first (and the best) was Dr. David Eagleman (website | @davideagleman) who is a neuroscientist from Baylor College of Medicine.  He gave a very interesting presentation why people think that time slows down during traumatic events such as falling from a tree or bike.  He and his team came up with an experiment where they basically threw a person off a building (it was actually a big scaffolding looking thing) into a net so they could test if their brain actually thought time was slowing down, if it just felt like it after the fact.

The second Dr. V.S. Ramachandran (website) and third speakers Dr. Deb Roy (website | @dkroy) while good speakers, simply weren’t as good as Dr. Eagleman as he was a very hard act to follow.  Unfortunately I don’t actually remember what Dr. Ramachandran spoke about.  Dr. Roy talked to us about the interactions between people and the way that those interactions can be tracked at the micro level (within the home) and at the macro level (worldwide).

At the micro level he installed camera and microphones in his home and recorded everything for 36 months starting shortly after his first child was born.  His research team then developed some software that tracked movement through the house and matched it to his child’s learning to speak and they were able to visually map on a diagram of the house in what parts of the house different words were learned.  For example the word “water” was learned in and near the kitchen and bathroom while the work “bye” was learned near the front door.

At the macro level he founded a company which tracks just about every TV show in the TV and analyses Twitter, Google+, Facebook, etc. traffic to see what people are talking about online so that studios and marketing companies can see how people are reacting to specific items online when they see them on TV.  It was interesting (all be it a little creepy) to see.

As far as sessions went today, there were only three slots, and I only had two sessions scheduled.  The first that I attended was a Resource Management Deep Dive into vSphere 4.1 and 5.0.  During this session they really went into how vSphere allocates resources (CPU, Memory and IO) to various VMs on the host depending on how reservations, guarantees, resource pools, etc. are all configured.  I’m not going to try to talk too much about this at the moment.  It’s going to take me a couple of times listening to the recording online to catch everything.

One thing that I did want to share that I didn’t know was how much data the DPM (Distributed Power Management) uses when it’s deciding to power down hosts at night and bring them back up in the morning.  When vCenter is deciding to power down a host is looks at the last 40 minutes of data to decide if there is little enough load to bring down a host.  As for bringing a host back online it only looks at the last 5 minutes. vCenter will never power a host down if it will lead to a performance problem.  When deciding to power hosts down performance is considered first with the after effect being that power is saved.  Power will always be used to get performance.

The second session was one on performance tuning of the vCenter database itself, which I figured would be pretty interesting.  It was interesting, but also frustrating as the speaker didn’t really know much about SQL Server (the default database to host the vCenter database).  Some of the information presented was pretty interesting about how the tables are laid out and what is stored in which table, and I’ve got a much better understanding about how the performance data gets loaded into the database and how the rollups are done now.  I also now know that I need to put together some scripts to jump start the process if it gets backed up as well as put together a best practices document for DBAs (and VMware folks that don’t know SQL at all) so that they can get better performance on their vCenter databases.

If you need to find the historical performance data within your vCenter database look into the tables which start with vpx_hist_stat.  There are 4 of these tables vpx_hist_stat1, vpx_hist_stat2, vpx_hist_stat3 and vpx_hist_stat4.  The historical data is rolled up by daily, weekly, monthly and annually into those four tables respectively.  You’ll also want to look into the vpx_sample_time tables of which there are also 4 tables vpx_sample_time1, vpx_sample_time2, vpx_sample_time3 and vpx_sample_time4.

Apparently vCenter 4.0 and below has a habit of causing deadlocks when loading data, especially in larger environments.  The fixes that were provided are pretty much all hit or miss when it comes to if they will work, and his description of the cause of the problem was pretty vague.  The jest of what I got was that the data loading is deadlocking with the code which handles the rollups and causing problems.  Something which could be tried to fix this would be to enable snapshot isolation mode for the database.  Personally I think this would have a better chance of fixing the problem then the crappy work around which he provided (which I’m not listing here on purpose).

The work around which VMware came up with for this problem, introduced in vCenter 4.0 can have its own problem is large environments. This problem is that data is missing for servers at random intervals.  This is because VMware came up with the idea of creating three staging tables and using each staging table for 5 minutes, then processing the data from that staging table into the vpx_hist_stat and vpx_sample_time tables while then moving on to the next staging table.  However if it takes too long to process the data from the first table, and the third table has been used it is now time to move back to the first table and data is now lost as it can’t write the data into the first table.  VMware needs to do some major redesign of this entire process for the next release to come up with a better solution that won’t allow for data loss.  There are plenty of was to do it that won’t cause problems.  Don’t you love it when developers that don’t know databases very well try and come up with screwy solutions to problems?

Based on what was talked about in this session there are a lot of SQL Scripts that need to be written to help people improve performance of their vCenter databases.  Guess I’ve got a lot of script writing to do.

Denny

After digging into the logs we saw that there was a problem with the SSL handshake.  It turns out that another restart of the vCenter services was all that was needed to clear up the problem.

So if you are having problems attaching hosts to vCenter after your upgrade just restart the vCenter services on the vCenter server and you should be fine.

Denny