System.Data.SqlClient.SqlException: Error 8115, Level 16, State 1, Procedure SomeStoredProcedure, Line 103, Message: Arithmetic overflow error converting IDENTITY to data type int.  Uncommittable transaction is detected at the end of the batch. The transaction is rolled back.

Ourshort term solution was quick and easy, to change identity seed of the column from 0 to the lowest possible number, in this case -2,147,483,648 which got us back up and running pretty quickly.  This however isn’t a very long term solution for this system.  The application is only about 3 or 4 years old, and has been growing very quickly over the last year or so.  We estimated based on the last few weeks worth of data that we would run out of negative numbers within about 12 months at the most.  We sent an estimate of 9 months to the business when we advised them that the system was back up and running.  We also told them that we wanted this fixed within 5-6 months to be save because if we didn’t get this fixed before running out of negative numbers there wouldn’t be any short term fix and that we’d be looking at a multi-day outage to fix the problem.

We couldn’t just rush into a database side fix, as the fix in this case is to change the data type from INT to BIGINT.  As the application does use this column in a couple of places the .NET application needed to be reviewed to ensure that anything that was looking for an INT was corrected to handle the BIGINT correctly.

Based on the amount of data within the table (about 300 Gigs) it was decided that taking an outage to make the change in place wasn’t really an option as doing the size change in place would require somewhere around a 5 day outage to remove and rebuild all the non-clustered indexes.  To make things a little more complex there is a second table which has a 1=1 relationship with this table, and the second table is even larger (about 800 Gigs), though thankfully the second table doesn’t have any non-clustered indexes.

The solution that was decided on was to move the data from the current table to a table with a BIGINT data type while the application was running.  To do this meant that we needed to get all the data copied over to the new table and in sync while the old table was being used.  It was decided that the easiest way to do this would be to use triggers.  In this case instead of one complex trigger to handle all insert, update and delete operations three separate triggers were used for each of the two tables.  First I created two new tables, with the exact same schema as the old tables, with the exception that the new tables used the bigint data type for the primary key instead of the int data type for the primary key and the new table was setup with the ident being the next available positive number.  Once that was done the triggers were setup.  The insert trigger is pretty basic.  Take the data that was just loaded and stick it into the new table.

CREATE TRIGGER t_MyTable1_insert ON dbo.MyTable1
FOR INSERT
AS
SET IDENTITY_INSERT MyTable1_bigint ON
INSERT INTO MyTable1_bigint
(Col1, Col2, Col3, Col4…)
SELECT Col1, Col2, Col3, Col4
FROM inserted
SET IDENTITY_INSERT MyTable1_bigint OFF
GO

The update and delete triggers required a little more logic.  The trick with the triggers was that I needed to avoid doing massive implicit data conversions.  In order to ensure that SQL was doing what I wanted (which it should be doing anyway, but it made me feel better doing explicit conversions) I explicit conversions into place for the JOIN predicates as shown.  The update trigger is shown first, then the delete trigger.

CREATE TRIGGER t_MyTable1_update ON dbo.MyTable1
FOR UPDATE
AS
UPDATE MyTable1_bigint
SET MyTable1_bigint.[Col2] = inserted.[Col2],
MyTable1_bigint.[Col3] = inserted.[Col3],
MyTable1_bigint.[Col4] = inserted.[Col4],
…
FROM inserted
WHERE cast(inserted.Col1 as bigint) = MyTable1_bigint.Col1
GO

CREATE TRIGGER t_MyTable1_delete ON dbo.MyTable1
FOR DELETE
AS
DELETE FROM MyTable1_bigint
WHERE Col1 in (SELECT cast(Col1 as bigint) FROM deleted)
GO

Once these tables were up and running all the new data changes were being loaded into the table.  At this point it was just a matter of coping the existing data into the table.  There are a few ways that this can be done.  In this case I opted for an SSIS package with a single data pump task, and two data transformations within the data pump task with one transformation for each table.

In order to make the load as fast as possible I used the fast load option and loaded 1000 rows at a time.  Within the data task if there was an error I redirected the rows to another data pump task which simply dropped the rows into the same table, but this time going row by row.  Any failures from that import were simply ignored.  While handling failed rows like this is time consuming it is easier than running T-SQL scripts to verify which rows are needed and which rows aren’t needed.  SSIS also gives an easy option to ignore the foreign key relationship between the two tables so if the child table gets rows first that isn’t a problem as we know that the parent table will catch up.  The SSIS package looked like this:

When all is said and done and the data is in sync between the new and old tables, the current tables will be dropped and the new tables will be renamed and put into place so that the application can continue to run without issue, with just a few minutes of downtime.

So why did this happen?  When the applications database was being designed the developers didn’t think about how many rows the database was going to get over time, so they didn’t account for needing to support more than 2.1 billion rows over time.  If I (or another data architect) had been involved in the project at it’s start this hopefully would have been caught at design time.  However when the application was first being designed the company was brand new and didn’t have the funds for a data architect to help with the application design so this problem was missed.

Hopefully you never hit this problem, but if you do this helps you get out of it.

Denny

The basic definition of a clustered index is that it has one or more indexed columns, and all other columns are added to the index as non-indexed columns.

Until SQL Server 2005 we didn’t have the ability to add additional columns to the index without sorting them. SQL Server 2005 introduced us to the concept of included columns.

So technically you can still only have 1 clustered index per table (an index created with the keyword CLUSTERED in the CREATE INDEX statement). But you can create as many nonclustered indexes as you need (up to the 999 nonclustered indexes per table that you are allowed) and as long as you include every other column in the database the index will effectively function like a clustered index.

Thanks to Erland and Hugo for the interesting discussion at PASS last November which prompted this post, not to mention letting me hang out with them for a few hours and pick their brains.

Denny

First the things that they have in common.

Both clustered and nonclustered indexes can be made up of more than one column. The columns are put in the index in the order you specify them in the CREATE INDEX statement (or the order they are shown in the UI). They are also sorted in this order as well. Indexes are first sorted on the first column in the index, then any duplicates of the first column and sorted by the second column, etc. You can have up to 16 columns specified as indexed columns.

Neither clustered or nonclustered indexes will guarantee the sort order of the data when it is being returned to you. If the order of the data matters to you, you should always sort the data with the ORDER BY clause in your select statement.

Both clustered indexes, and nonclustered indexes take up additional disk space. The amount of space that they require will depend on the columns in the index, and the number of rows in the table. The clustered index will also grow as you add columns to the table (keep reading, it’ll make sense later on).

Adding indexes (both clustered and nonclusterd) will increase the amount of time that your INSERT, UPDATE and DELETE statement take, as the data has to be updated in the table as well as in each index. If you have filtered indexes in SQL Server 2008 and the records you are updating are not included in all your indexes, SQL Server should only have to update the values in the indexes which the records are stored within.

Columns of the TEXT, NTEXT and IMAGE data types can not be indexed using normal indexes. Columns of these data types can only be indexed with Full Text indexes.

If you wish to rebuild your indexes online (without locking the table) and have Enterprise edition do not index TEXT, NTEXT, IMAGE, VARCHAR(MAX), NVARCHAR(MAX), VARBINARY(MAX) data types as including columns with these data types will require that you rebuild the index offline.

The total size of the key columns can not exceed 900 bytes.  Don’t forget that uni-code characters take up two bytes per character which will reduce the number of characters which your index can hold.

Clustered Indexes

SQL Server only supports a single clustered index per each database table.

Clustered indexes do not support included columns. This is because clustered indexes contain all the columns which aren’t in the index as included columns already. There is no way to override this. Columns with the data type of TEXT, NTEXT and IMAGE are not included within the clustered index. As with the normal table these values are stored out of bounds and only the pointer to the physical location on disk is stored within the index.

Clustered indexes must be placed within the same file group as the physical table which they are created against.

***Update***

Let me point out that SQL Server won’t throw an error if you try and put the clustered index on another file group.  SQL Server will let you do this, however the table will be moved to this file group as well.  The Clustered index and the physical table must be in the same file group.

*************

Nonclustered Indexes

Nonclustered indexes (also called “indexes”) are the normal indexes that are created on your tables. SQL Server supports 999 nonclustered indexes per table.

Each nonclustered index can contain upto 1023 included columns. Columns of the TEXT, NTEXT and IMAGE data types can not be included either.

There are more things that could be put in here, but this will get you a good start. If you think of anything that I’ve missed please feel free to post them below.

Denny