The WITH (NOLOCK) table hint isn’t a go faster button for SQL Server.  It has actual implications to the data which is being returned by the query.  The biggest of these implications is that the data might not be correct.  You see the WITH (NOLOCK) table hint uses dirty reads to return the data, so it basically ignores the locks which other queries have taken.  This is why the query appears to run faster, because the query isn’t being blocked any more.  The proper approach would be to find the query which is causing the extended blocking and figure out why it is taking so long to run, and fix the performance problems of that query.

The only go faster button that is available in SQL Server is the CREATE INDEX statement.  Anything else isn’t truly a go faster button, and has other side effects which must be understood before being implemented.

Denny

System.Data.SqlClient.SqlException: Error 8115, Level 16, State 1, Procedure SomeStoredProcedure, Line 103, Message: Arithmetic overflow error converting IDENTITY to data type int.  Uncommittable transaction is detected at the end of the batch. The transaction is rolled back.

Ourshort term solution was quick and easy, to change identity seed of the column from 0 to the lowest possible number, in this case -2,147,483,648 which got us back up and running pretty quickly.  This however isn’t a very long term solution for this system.  The application is only about 3 or 4 years old, and has been growing very quickly over the last year or so.  We estimated based on the last few weeks worth of data that we would run out of negative numbers within about 12 months at the most.  We sent an estimate of 9 months to the business when we advised them that the system was back up and running.  We also told them that we wanted this fixed within 5-6 months to be save because if we didn’t get this fixed before running out of negative numbers there wouldn’t be any short term fix and that we’d be looking at a multi-day outage to fix the problem.

We couldn’t just rush into a database side fix, as the fix in this case is to change the data type from INT to BIGINT.  As the application does use this column in a couple of places the .NET application needed to be reviewed to ensure that anything that was looking for an INT was corrected to handle the BIGINT correctly.

Based on the amount of data within the table (about 300 Gigs) it was decided that taking an outage to make the change in place wasn’t really an option as doing the size change in place would require somewhere around a 5 day outage to remove and rebuild all the non-clustered indexes.  To make things a little more complex there is a second table which has a 1=1 relationship with this table, and the second table is even larger (about 800 Gigs), though thankfully the second table doesn’t have any non-clustered indexes.

The solution that was decided on was to move the data from the current table to a table with a BIGINT data type while the application was running.  To do this meant that we needed to get all the data copied over to the new table and in sync while the old table was being used.  It was decided that the easiest way to do this would be to use triggers.  In this case instead of one complex trigger to handle all insert, update and delete operations three separate triggers were used for each of the two tables.  First I created two new tables, with the exact same schema as the old tables, with the exception that the new tables used the bigint data type for the primary key instead of the int data type for the primary key and the new table was setup with the ident being the next available positive number.  Once that was done the triggers were setup.  The insert trigger is pretty basic.  Take the data that was just loaded and stick it into the new table.

CREATE TRIGGER t_MyTable1_insert ON dbo.MyTable1
FOR INSERT
AS
SET IDENTITY_INSERT MyTable1_bigint ON
INSERT INTO MyTable1_bigint
(Col1, Col2, Col3, Col4…)
SELECT Col1, Col2, Col3, Col4
FROM inserted
SET IDENTITY_INSERT MyTable1_bigint OFF
GO

The update and delete triggers required a little more logic.  The trick with the triggers was that I needed to avoid doing massive implicit data conversions.  In order to ensure that SQL was doing what I wanted (which it should be doing anyway, but it made me feel better doing explicit conversions) I explicit conversions into place for the JOIN predicates as shown.  The update trigger is shown first, then the delete trigger.

CREATE TRIGGER t_MyTable1_update ON dbo.MyTable1
FOR UPDATE
AS
UPDATE MyTable1_bigint
SET MyTable1_bigint.[Col2] = inserted.[Col2],
MyTable1_bigint.[Col3] = inserted.[Col3],
MyTable1_bigint.[Col4] = inserted.[Col4],
…
FROM inserted
WHERE cast(inserted.Col1 as bigint) = MyTable1_bigint.Col1
GO

CREATE TRIGGER t_MyTable1_delete ON dbo.MyTable1
FOR DELETE
AS
DELETE FROM MyTable1_bigint
WHERE Col1 in (SELECT cast(Col1 as bigint) FROM deleted)
GO

Once these tables were up and running all the new data changes were being loaded into the table.  At this point it was just a matter of coping the existing data into the table.  There are a few ways that this can be done.  In this case I opted for an SSIS package with a single data pump task, and two data transformations within the data pump task with one transformation for each table.

In order to make the load as fast as possible I used the fast load option and loaded 1000 rows at a time.  Within the data task if there was an error I redirected the rows to another data pump task which simply dropped the rows into the same table, but this time going row by row.  Any failures from that import were simply ignored.  While handling failed rows like this is time consuming it is easier than running T-SQL scripts to verify which rows are needed and which rows aren’t needed.  SSIS also gives an easy option to ignore the foreign key relationship between the two tables so if the child table gets rows first that isn’t a problem as we know that the parent table will catch up.  The SSIS package looked like this:

When all is said and done and the data is in sync between the new and old tables, the current tables will be dropped and the new tables will be renamed and put into place so that the application can continue to run without issue, with just a few minutes of downtime.

So why did this happen?  When the applications database was being designed the developers didn’t think about how many rows the database was going to get over time, so they didn’t account for needing to support more than 2.1 billion rows over time.  If I (or another data architect) had been involved in the project at it’s start this hopefully would have been caught at design time.  However when the application was first being designed the company was brand new and didn’t have the funds for a data architect to help with the application design so this problem was missed.

Hopefully you never hit this problem, but if you do this helps you get out of it.

Denny

The first index is on the dbo.VPX_PROPERTY_BULLETIN database.  On the system that I happen to be looking at this week not having this index caused the SQL Server to scan this table 6977 times since the database was last restarted about 7 days before I began looking at the system.  This table on this SQL Server only contains about 3000 records, but this system is pretty small.  Just 4 hosts, 4 data stores and about 100 guests at the moment.  The larger this table is the more impact not having this query will have.

CREATE INDEX mrdenny_OPERATION_TYPE ON dbo.VPX_PROPERTY_BULLETIN
([OPERATION_TYPE])
INCLUDE ([OBJECT_MOID], [GEN_NUMBER])
WITH (FILLFACTOR=70, ONLINE=OFF, PAD_INDEX=ON)

The nice thing about this index is that is also fills the requirements of another index which is needed by the SQL Server.

The second index to create is built on the same table, but on different columns.  While the query which needs this index is run MUCH less often, SQL estimates (on this system at least) that adding it will improve the query performance by 69.94%.  In my mind that’s very much worth it, even if the query is only being run a few times a week.

CREATE INDEX mrdenny_OBJECT_MOID ON dbo.VPX_PROPERTY_BULLETIN
([OBJECT_MOID])
INCLUDE ([GEN_NUMBER])
WITH (FILLFACTOR=50, ONLINE=OFF, PAD_INDEX=ON)

The third index that I’ve identified which needs to be created is against the VPX_LIC_USAGE table.  This table has something to do with the licensing and the size of the table on your system will vary.  This vCenter system has over 16k rows in the table but this system has only been installed for a couple of months at this point.  As your vSphere installation ages it appears that this table will continue to grow and grow.  Best I can tell there’s a couple of rows entered into this table every hour for each host in the farm.  Needless to say this table will grow quite large when you’ve got a large VMware vSphere farm.

CREATE INDEX mrdenny_SAMPLE_TIMESTAMP ON dbo.VPX_LIC_USAGE
([SAMPLE_TIMESTAMP])
INCLUDE ([SAMPLE_TS])
WITH (FILLFACTOR=100, ONLINE=OFF)

As you look at these three indexes which I recommend that you create on your vSphere database you’ll notice that there is an ONLINE=OFF flag.  If your vCenter database is running the Enterprise Edition of SQL Server then you can change that on ONLINE=ON which will allow the indexes to be created online instead of causing blocking while the indexes are being created.  If you don’t have the Enterprise edition of SQL Server then you can’t create in the indexes online.  There should be no impact to the vCenter system if you create the indexes while the system is up and running.  The worse that will happen is that queries which are already running slowly will run a little slower than normal.

You’ll notice that I’ve listed this indexes to all start with mrdenny.  This is so that these indexes can be easily identified as coming from my blog (in case the next admin wonders where they are from) and so that you never have to worry about the index names colliding with names that VMware would try and use.

Denny

Now in reality many of these 3rd party vendors don’t have anyone no staff who knows much about SQL Server, or databases in general.  You sure don’t get the ability to talk to anyone that these 3rd party vendors have on staff that actually do know anything about databases.  So when you start seeing performance problems you basically have a few different options.

1. Call their support line and talk to someone who thinks there is an “E” in SQL (they spell it sequel).

2. Leave the system with performance problems.

3. Fix the performance problem and hope that the vendor doesn’t notice the indexes that you created.

Obviously option #2 probably isn’t the one that the business unit that uses the application is going to accept.  #1 is what they want you to do, but that’s probably going to be less than useful possibly taking days or weeks in order to get the issue resolved.

In reality #3 is the option that you want to go with, but there is that little problem of the support agreement to deal with.  Obviously I would never recommend invalidating your support agreement … but when you need to get your production application back up and running, you need to get the system back up and running.  Now you can’t do crazy things like change the database schema but usually that isn’t needed to get everything back up and running.

Usually it’s something as simple as updating statistics or creating a new index.  Now you need to hide this index from the vendor so that they don’t complain that you’ve “broken” the system by fixing it.  That’s why I recommend creating the indexes so that they are easy to identify later.  When I create indexes in a vendor system I prefix them with something that the vendor will never use, like my name.  This way I can quickly query for all the indexes and drop them the next time I need to let the vendor into the database (don’t forget to script them out first so you can but them back afterwards).

While the vendor will probably complain that you’ve added these indexes they can’t actually say that you’ve done anything damaging if you’ve removed them as well.  Not to mention they probably won’t actually be able to find out if you’ve created and dropped indexes if you’ve removed them already.

While this probably isn’t the best way to have to deal with 3rd party vendor applications it is the reality of dealing with 3rd party vendors.

Denny

“In this session we will be looking at the best and worse practices for indexing tables within your SQL Server 2008 databases.  We will also be looking into the new indexing features that are available in SQL Server 2008 (and SQL Server 2005) and how you the .NET developer can make the best use of them to get your code running its best.”

If you would like to download the slide deck that I presented from you can download it here.

I hope that you enjoyed the presentation.  I look forward to seeing you at my next presentation.

Denny

Table Indexing for the .NET Developer

Indexing Internals

SQL Server Clustering 101

I hope that everyone had as good a time at SQL Saturday as I did.

Denny

P.S. Something to note, don’t leave early after your raffle ticket is drawn.  We had so many great prizes that we ran out of raffle tickets and had to start drawing evals from the box for prize winners, so you might have won something else if you stuck around till the end.

Currently I’m presenting “Indexing Internals” at 12:45pm on Saturday and then at 2:15pm on Saturday I’ll be presenting “Optimizing SQL Server Performance in a Virtual Environment”.  The schedule is subject to change, but hopefully won’t as I really like those time slots.  Tickets for the two day conference are just $100 and are on sale through August 10th.  Just go to the Dallas Tech Fest home page and you’ll see the Event Bright order form right there in the middle of the page.  It should be an awesome event, I can’t wait to speak there for the first time.

Anyway, I can’t wait to see everyone; I’ll be there until Sunday afternoon when I have to head back to DFW so I can head home.

See you there,
Denny

Index Internals

Optimizing SQL Server Performance in a Virtual Environment

I hope everyone had as good a time at the conference that I had.

Denny

In this session we’ll dig into the internal structures of indexes.  We will explore the differences between clustered and non-clustered indexes, what’s laid out within each page of the indexes and how the SQL Server uses the data within the indexes to find rows quickly.

Knowing how indexes actually work is key to understanding why indexes improve query performance.  Knowing that indexes will help performance is good (and hopefully you know this), but knowing how the index actually works is even more important so that you can make proper indexes so that you aren’t making extra indexes that aren’t needed as having to many indexes will actually slow down SQL Server not make it faster.

If you haven’t signed up yet, now is the time.

See you in Napa September 22nd-24th,

Denny

SQL Server Indexing for the .NET Developer

Using SQL Denali’s Always On

Where should I be encrypting my data

If you attended the sessions please rate my presentations on SpeakerRate.com/mrdenny.

Thanks,

Denny