We it has started to take stupid amounts of time to delete the data, so I started digging into the data deletion procedures.

The first thing I looked at was the actual delete statements.  These seams ok, the actual deletes were happening very quickly (we process deletes in batches of 1000 records per batch to minimize locking of data).  So next I looked at the part of the code where we select the records to be deleted.  Looking at the execution plan, everything looked ok.

But this little chunk of code took about 50 minutes to run.  Pretty bad when only returning 1000 numbers back from the database.

SELECT TOP (@BatchSize) a.PolicyIncidentId
FROM PolicyIncident a WITH (NOLOCK)
JOIN #ComputersToProcess ComputersToProcess ON a.ComputerId = ComputersToProcess.ComputerId
WHERE CaptureTimestamp < ComputersToProcess.StartDeleteAt

The first thing that I did was put a primary key on the @ComputersToProcess table variable.  That turned the table scan into a Clustered Index Scan, but didn’t do anything for performance.

The next thing I did was switch the table variable to a temp table (without a primary key).  This really didn’t do anything to speed up the process as there is still no statistics on the data.  However this time the execution plan actually shows you that there’s no statistic on the temp table.

Now, I didn’t want to put at non-clustered index on the table keeping the table as a heap, and a clustered index that wasn’t a primary key wasn’t going be any more effective than a primary key, so I put a primary key on the table.  While the query cost percentage went up from 2% to 7% the actual run time went down from 50 minutes to just 1 second.

Now I didn’t make any other code changes to the procedures, just changing from the table variable to the temp table, and adding a primary key and this one little three line query went from an hour to a second.  Its amazing how much such a small change can make things run smoother.

Now obviously this isn’t going to fix every problem.  But in my case I’m putting a little over 190k rows into the table variable (now temp table) and this is just to much for the table variable to take.  Keep in mind that with table variables the SQL Server has statistics, but it assumes only a single row per temp table, no matter how much data is actually in the table variable.

Denny

There’s some full text searching going on so it’s never going to really scream as there are 70 million records in one table, 57 million records in another table (with a one to many between them), and another 90 million records in the third table (this is a one to many to the table with 70 million records, and this table has the full text index on it).

Needless to say there was some tuning that I had to do.  The basic jist of the query was…

...
WHERE EXISTS (SELECT *
FROM OPENXML(@hDoc_Computer, '//computer')
WITH (ComputerId INT '@id') a
WHERE a.ComputerId = Application.ComputerId)
AND EXISTS (SELECT *
FROM OPENXML(@hDoc_Logon, '//login')
WITH (LogonId INT '@id') a
WHERE a.LogonId = Application.LogonId)

After getting no where working on indexes and tweaking things here and there (and actually making the query take 16 minutes to run for this customer’s data) I put a couple of table variables in the procedure and loaded those table variables up with the values from the XML Documents.

DECLARE @Computer TABLE
(ComputerId INT)

DECLARE @Logon TABLE
(LogonId INT)

...

INSERT INTO @Computer
SELECT ComputerId
FROM OPENXML(@hDoc_Computer, '//Computer', 2)
WITH (ComputerId INT '@ComputerId')

INSERT INTO @Logon
SELECT LogonId
FROM OPENXML(@hDoc_Logon, '//Logon', 2)
WITH (LogonId INT '@LogonId')

...

And I changed the WHERE clause to use the table variables instead.

WHERE EXISTS (SELECT * FROM @Computer a WHERE a.ComputerId = Application.computerId)
AND EXISTS (SELECT * FROM @Logon a WHERE a.LogonId = Application.LogonId)

This got my query run time down to about 1 minute with the execution plan showing that ~90% of the time spent is being spent on the full text search. So while I wouldn’t normally consider a query run time on 1 minute to be good, in this case it is. (This particular part of the application also goes out to the file server and uses Microsoft Search service to search millions of files for text string matches so this is now the fastest part of the search process.

Now don’t take this post the wrong way. I love OPENXML, it’s a great tool and I use it all over the place so that we can pass in multiple values in a single variable (all our code has to be able to run on SQL 2005 so table input parameters aren’t an option for me). OPENXML just wasn’t the write tool here, sort of.

I wish we could have found this performance problem in QA, but we just have no way to generate enough data to find these kinds of performance problems. But the problem is fixed and the customer is hopefully happy (for now).

Denny

While we have been able to do this in the past, by using XML to pass more than one value in, then break it apart. But this is just such a simpler, easier, more elegant solution.

It is a bit of a process to get it done, but once it is all setup it is a piece of cake to use.

You can’t just create a table as part of the input parameter to the stored procedure like this.

CREATE PROCEDURE YourProcedure
     @YourTable TABLE (Col1 INT)
AS
...
GO

That would be to easy. First you have to create a User Defined Table Type by using the CREATE TYPE command. Then you create an input (or output) parameter using this table type then in your calling code create a parameter using this same user defined table type and load it with data, then call the procedure just as you normally would.

CREATE TYPE MyTableType AS TABLE
   (Id INT)
GO
CREATE PROCEDURE MyProcedure
   @Ids MyTableType OUTPUT
AS
INSERT INTO @Ids
SELECT object_Id
FROM sys.objects
GO
DECLARE @values MyTableType
exec MyProcedure @Ids=@values OUTPUT
SELECT *
FROM @values
GO
DROP PROCEDURE MyProcedure
GO
DROP Type MyTableType
GO

Personally I can’t wait to to begin using this new feature, but it’ll probably be a while before we convert our system to require SQL Server 2008, as we have customers who are still running SQL Server 2000 and aren’t happy about our requirement for SQL Server 2005.

Denny

Temp Tables vs. Table Variables

1. SQL Server does not place locks on table variables when the table variables are used.
2. Temp tables allow for multiple indexes to be created
3. Table variables allow a single index the Primary Key to be created when the table variable is declared only.  There is an exception to this, that if you can create the index inline, for example by creating a unique constraint inline as shown in the comments.  However these indexes (and the table variable in general) will always be assumed to have 1 row in them, no matter how much data is within the table variable.
4. Temp tables can be created locally (#TableName) or globally (##TableName)
5. Table variables are destroyed as the batch is completed.
6. Temp tables can be used throughout multiple batches.
7. Temp tables can be used to hold the output of a stored procedure (temp tables will get this functionality in SQL Server 2008).

Table variables and Temp Tables vs. CTEs

1. CTEs are used after the command which creates them.
2. CTEs can be recursive within a single command (be careful because they can cause an infinite loop).
3. Table variables and Temp Tables can be used throughout the batch.
4. The command before the CTE must end with a semi-colon (;).
5. As Temp tables and table variables are tables you can insert, update and delete the data within the table.
6. CTEs can not have any indexes created on them, source tables much have indexes created on them.

If you can think of anything that I’ve missed, feel free to post them in the comments.

Denny