To look at what’s going on lets create a sample new table in a table and take a peak at the data.  To setup this test we create a new database and setup that database for use with AlwaysOn Availability Groups.  The availability group is setup with no readable secondary replicas.

create table MyTest (c1 int identity(1,1),
c2 int,
c3 varchar(100))
GO
insert into MyTest
(c2, c3)
values
(1, ‘test’)
GO 400

This creates a new table in the database with 400 rows in it.  Looking at the output from DBCC IND we can see that this table takes up 2 data pages with a root page (this will become important later on).

Looking at the data for the first data page we can see the following information.

Slot 0 Offset 0×60 Length 23

Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS
Record Size = 23
Memory Dump @0x000000003CACA060

0000000000000000:   30000c00 01000000 01000000 03000001 00170074  0………………t
0000000000000014:   657374                                        est

Slot 0 Column 1 Offset 0×4 Length 4 Length (physical) 4

c1 = 1

Slot 0 Column 2 Offset 0×8 Length 4 Length (physical) 4

c2 = 1

Slot 0 Column 3 Offset 0×13 Length 4 Length (physical) 4

c3 = test

Slot 0 Offset 0×0 Length 0 Length (physical) 0

KeyHashValue = (8194443284a0)

When we update this row changing the value of c2 to equal 2 nothing really changes which we can see from DBCC PAGE again.

Slot 0 Offset 0×60 Length 23

Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS
Record Size = 23
Memory Dump @0x000000003EECA060

0000000000000000:   30000c00 01000000 02000000 03000001 00170074  0………………t
0000000000000014:   657374                                        est

Slot 0 Column 1 Offset 0×4 Length 4 Length (physical) 4

c1 = 1

Slot 0 Column 2 Offset 0×8 Length 4 Length (physical) 4

c2 = 2

Slot 0 Column 3 Offset 0×13 Length 4 Length (physical) 4

c3 = test

Slot 0 Offset 0×0 Length 0 Length (physical) 0

KeyHashValue = (8194443284a0)

Next I’ve changed the settings for the availability group to support readable secondary replicas.  Once that change has been made we change the value of c2 for the same row to equal the value of 3.  Again we can look at this with DBCC PAGE.

Slot 0 Offset 0x1d4e Length 37

Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS VERSIONING_INFO
Record Size = 37
Memory Dump @0x000000003AEEBD4E

0000000000000000:   70000c00 01000000 03000000 03000001 00170074  p………………t
0000000000000014:   65737400 00000000 0000000f 06000000 00        est…………..

Version Information =
Transaction Timestamp: 1551
Version Pointer: Null

Slot 0 Column 1 Offset 0×4 Length 4 Length (physical) 4

c1 = 1

Slot 0 Column 2 Offset 0×8 Length 4 Length (physical) 4

c2 = 3

Slot 0 Column 3 Offset 0×13 Length 4 Length (physical) 4

c3 = test

Slot 0 Offset 0×0 Length 0 Length (physical) 0

KeyHashValue = (8194443284a0)

Looking at these two outputs from DBCC PAGE we can see a couple of differences.  First we see an additional value in the “Record Attributes” field which adds in VERSIONING_INFO to the value.  We also see that the record size has changed from 32 to 37.  Additionally we see that the Version Information has been added.

Looking at the DBCC PAGE output on one of the replicas for the same page as before we see some different information.

Slot 0 Offset 0x1d4e Length 37

Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS VERSIONING_INFO
Record Size = 37
Memory Dump @0x000000003893BD4E

0000000000000000:   70000c00 01000000 03000000 03000001 00170074  p………………t
0000000000000014:   65737440 01000001 0000000f 06000000 00        est@………….

Version Information =
Transaction Timestamp: 1551
Version Pointer: (file 1 page 320 currentSlotId 0)

Slot 0 Column 1 Offset 0×4 Length 4 Length (physical) 4

c1 = 1

Slot 0 Column 2 Offset 0×8 Length 4 Length (physical) 4

c2 = 3

Slot 0 Column 3 Offset 0×13 Length 4 Length (physical) 4

c3 = test

Slot 0 Offset 0×0 Length 0 Length (physical) 0

KeyHashValue = (8194443284a0)

Specifically at this point we see that the Version Pointer now has a value in it. This tells us that the SQL Server has put a copy of the original page into the tempdb database so that we can read it.

A question comes up as to what happens to the page when this row information is added.  Specifically does the page split because of this additional 14 bytes of new data per row.  The answer to this question is “it depends”.  In my testing that I did when I updated a single row the page didn’t split, mostly this would be because there was some free space in the database page.  When I updated rows 2-40 and looked at DBCC IND I saw that in fact the page had split.

Looking at the values within the page there are now 159 rows in the page which was the original database page when there were 322 rows within the database page.  The remainder of the rows were copied into a new database page.

Now that we’ve identified that SQL Server is going to be page splitting older database pages, potentially like crazy what can we do about it?  The answer to that question is to just deal with it and to decrease the fill factor as needed so that page splits happen as little as possible.

To make matters worse when we rebuild the index on the table and look at the output from DBCC PAGE again we can see that the additional flag has been removed from row 1 (seen below).  This tells us that no only will this problem come up the first time that data is modified, it’ll come up every time that index rebuilds are done when the data is changed for the first time after the rebuild.

Slot 0 Offset 0×60 Length 23

Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS
Record Size = 23
Memory Dump @0x000000003E0CA060

0000000000000000:   30000c00 01000000 03000000 03000001 00170074  0………………t
0000000000000014:   657374                                        est

Slot 0 Column 1 Offset 0×4 Length 4 Length (physical) 4

c1 = 1

Slot 0 Column 2 Offset 0×8 Length 4 Length (physical) 4

c2 = 3

Slot 0 Column 3 Offset 0×13 Length 4 Length (physical) 4

c3 = test

Slot 0 Offset 0×0 Length 0 Length (physical) 0

KeyHashValue = (8194443284a0)

Changing the data again, this time changing the first 40 rows (id values 1-40) the new flag comes into place as expected.  If we reorganize the index instead of doing a rebuild this time the flags are left in place.

This tells us that the better option for doing index maintenance on databases which are being protected by AlwaysOn Availability Groups is going to be to use reorganize commands instead of rebuild commands.  This way the 14 byte pointer isn’t removed from the rows so that when they are modified the additional 14 bytes of data doesn’t need to be added.

If you’ve got rows which are changed all the time then this will be a way to handle it.  If the rows never change after the data is reorged then it may or may not be something worth worrying about.

Hopefully this helps answer the questions of what these extra bytes are for and how we can deal with them.

Denny

As a part of this configuration we wanted to do the DBCC CHECKDB as a part of the backup job so that after the DBCC CHECKDB is complete the databases will then be backed up.

The catch here is that we didn’t want to have to dig through the errorlog file to figure out what the database was that’s having the problem, and we didn’t want the job to fail if there was a problem so I started looking at catching errors when running DBCC CHECKDB.  Sadly there aren’t really any good ways to do this.  TRY/CATCH doesn’t work because DBCC doesn’t actually throw error messages like a normal SQL statement does.  It returns the errors, but it doesn’t actually throw the errors so the CATCH block isn’t actually captured.  Running DBCC CHECKDB within an EXEC sp_executesql doesn’t catch the error either for the same reason, the error isn’t thrown it is simply displayed.  (The reason that the errors from DBCC CHECKDB show up in Management Studio in red I would assume is because SSMS is catching the format and displaying it correctly.)

To catch the errors I had to resort to some old school “error handeling” using the @@ERROR system function.  While this isn’t perfect, I’m not looking for perfect here.  I’m just looking for something that says that there’s an error so that I can send an email when there is an error then continue to loop through the databases looking for others with problems.

As this is step 1 of a multistep job this step is configured to move on to the next step on success or failure (as an email will have been sent and the data logged to the ERRORLOG) then we’ll procede to the backups.

The code that I’m using looks something like this…

/*Populate the table #dbs with the databases that need to be checked.*/
DECLARE cur CURSOR FOR SELECT name from #dbs
OPEN cur
FETCH NEXT FROM cur INTO @name
WHILE @@FETCH_STATUS 0
BEGIN
SET @sql = 'DBCC CHECKDB ([' + @name + '])'
EXEC sp_executesql @sql
IF @@ERROR 0
BEGIN
set @subject = 'CHECKDB failure for ' + @name
set @body = 'DBCC CHECKDB failed for database ' + @name + '
Command run was: ' + @sql
exec msdb.dbo.sp_send_dbmail...
END
FETCH NEXT FROM cur INTO @name
END
CLOSE cur
DEALLOCATE cur

There’s some custom code at the top which figures out which databases to process so that the job step runs DBCC CHECKDB on the same databases which the job will backup (this is figured out based on the database size, if the database is online, and if the database is currently being bulk loaded).  This code isn’t shown as it’s not relivant to this specific problem.

Denny

Many people think that the simple recovery model doesn’t use the transaction log.  More specifically they think that it’s there because it has to be there, but that SQL Server doesn’t actually use it.  The reality is that SQL Server still uses the transaction log, much like it does in full or bulk logged recovery modes.  There are some transactions which are going to be minimally logged, but for the most part the INSERT, UPDATE and DELETE commands are going to be fully logged just like normal.

What SQL Server does with the transaction log in simple recovery model is that when the transactions are committed they are written to the transaction log and the pages are dirtied in the buffer pool.  When checkpoint runs the dirty pages in the buffer pool are written to the disk.  Everything up to this point is basically the same as with the other recovery models.  Once the checkpoint has been completed things get different between simple recovery and the other two recovery models.  With the simple recovery model the virtual log files which were just checkpointed and had their dirty pages written to disk will be marked as no longer in use (status=0).  With bulk logged and full recovery this doesn’t happen until the transaction log backup has been completed.

Hopefully this helps dispel the myth.

Denny

Deleting the data from the database is the easy part, a simple delete loop will handle that nicely.  The problem is when you delete rows from a table which contains LOB data the LOB pages aren’t cleared when they are deallocated.  We can see this by running the following code.

CREATE DATABASE Lobtest
GO
use Lobtest
GO
CREATE TABLE t1 (c1 int IDENTITY(1,1) PRIMARY KEY, c2 ntext)
GO
INSERT INTO T1 (c2) VALUES (replicate('a', 20000))
GO
DBCC IND ('LobTest', 't1', 1)
GO
DBCC TRACEON(5201, -1)
GO
DELETE FROM t1
GO
DBCC IND ('LobTest', 't1', 1)
GO
DECLARE @dbid as int = db_id('Lobtest')
DBCC PAGE (@dbid, 1, 231, 3)
GO

You can see that page 231 is a LOB page which is allocated to the table t1. When you look at the actual page using DBCC PAGE after the row has been deleted we can see that there is data in the page, and that the page header shows that the page is still allocated to the table t1. This can be seen by looking in the header of the page for the header value labeled “Metadata: ObjectId = 245575913″.

When you go to shrink the database the SQL Server engine will get to the LOB pages and it will need to figure out if the LOB row is a part of a row which still exists or not. In order to do this SQL Server will need to scan through the pages which make up the table looking for any rows which reference the page it is trying to delete.

When doing shrinks after deleing large amounts of LOB data SQL Server will generate large amounts of IO while figuring this out and the shrink operation will take an extremely long time. (Paul Randle talks more about it here.)

So the question that this person at SQL Saturday had was, how can I reclaim the space from my database within a reasonable time.

The solution that we came up with was actually pretty simple. Do the database deletion as normal. Then backup and restore the database. Then do the shrink, followed by rebuilding the clustered indexes in order to fix the fragmentation issue which the shrink will introduce.

This works for a pretty simple reason, because the PFS page shows that the LOB page isn’t allocated even though the page is full of data (you can verify this by looking at page 1 in file 1 in the sample database created by the script above). When the database engine backups up the database the database engine looks at the PFS pages to figure out which pages to back up. Because the PFS pages show that the pages are empty the database engine doesn’t bother to backup the pages, so when the pages are restored they are restored as blank pages. This means that after the restore the shrink operation can run without an issue.

In the case of this application there was a maintenance window which could be taken advantage of which would allow the backup and the restore to happen.

Another option which we came up with which would require less downtime involved using database mirroring. By configuring database mirroring (which is initialized via a backup and restore process giving us the same basic approach) and then failing over to the mirror we would end up in the same position. We could then shrink the database without issue (probably pausing database mirroring so that we didn’t have to wait for the second server to process the shrink in real time) and then fail back the database to the original server.

As geeky as it was, Mladen, Andre and I had a great time figuring this out, and the attendee had a great time watching us go through all the possible options as we excluded them one by one. And most importantly she got her problem solved.

So if you end up in this situation here’s a solution that will help you shrink the database so that you can reclaim the space that the LOB data pages are taking up without having to wait forever.

Denny

Officially the answer is no, however with a little bit of creative configuration you sure can.

The Overall Environment

To setup SQL Server Express in a Windows Cluster I’m building this on a two node Windows Server 2012 cluster using a file share hosted on my domain controller to host the actual databases.  To ensure that the domain controller is rebooted as little as possible the domain controller is installed in core mode. The cluster nodes are Windows Server 2012 standard edition (which now supports clustering) as is the domain controller.

Installation

As SQL Server 2012 express edition doesn’t support Windows Clustering out of the box the installation will be a little different from doing a normal clustered install under standard or enterprise edition.  To install I did a normal SQL Express install on node1.  The only change from a normal install that I made was that I configured the SQL Server instance to start under a domain account.  When I got to the data directories part I configured the data folder to a network share on the domain controller.

Once the installation on node1 was completed I stopped the SQL Server services.  Then I renamed the folder that I installed the SQL Server database files into.  The reason for this is that I need to configure the second instance to put the database files into the same location.  I can then install SQL Server 2012 express edition onto the second node.

The installation on node2 is done exactly like it was done on node1.

Once the installation is done on both nodes configure the SQL Server service to have a startup type as “Manual” instead of disabled or automatic.  Leave the SQL Agent service as disabled as even though SQL Express installs the SQL Agent the SQL Agent isn’t supported on SQL Express.

Configuring Clustering

Once the installation on Node2 is done the cluster can be configured.  To do this bring up the Failover Cluster Administrator on one of the nodes and connect to the cluster.  If the cluster hasn’t been configured yet run through the normal Clustering Configuration wizard.

We’ll now configure a new cluster role on the cluster.  To do this right click on “Role” then select “Configure Role” from the context menu as shown below.

When the wizard opens click next to get to the list of services.  Then select the Generic Service item from the list as shown below.

On the next screen you’ll be asked what service you wish to cluster.  From this list select the SQL Server service as shown below.

On the next screen you’ll be asked to name the resource group.  Give the group a name which is unique on the domain and click next.  The next screen will ask you to select the needed storage.  Simply click next on this screen as we aren’t using any local shared storage.  The next screen asks you if any registry settings need to be replicated between the machines.  We don’t need to replicate anything as SQL Server doesn’t make much use of the registry for the actual SQL Server service so we can simply click next on this screen as well.  The next screen is simply a screen to review the changes which will be made.  You can simply click next on this screen after reviewing the information on the screen.  When the summary screen displays click finish.

Post Clustering SQL Config Changes

The first change that you’ll need to make is to enable the TCP network protocol on both nodes.  By default SQL Express has the TCP network protocol disabled which need to be corrected before uses will be able to connect to the SQL Server service.

The next change that you’ll need to make is to change the local server name in the master database from the name of the last node which was installed to the cluster name using a script similar to the one shown below.  In the case of this script the nodes are named node1 and node2 and the cluster name is clustersql. Once this script has been run the SQL Server instance should be restarted or failed over to the other node.

exec sp_dropserver ‘nodeb’
GO
exec sp_addserver ‘clustersql’, local
GO

At this point the cluster is up and running and applications can have their databases configured on the SQL Server Instance.

Denny

When turning on this trace flag because you’ve got a large database the goal obviously is to use autostats so you’ll need to have auto stats turned on.  In addition you’ll want to turn on the “Auto Update Statistics Asynchronously” setting for the database or databases which hold the super large table.  The reason that you’ll want to turn on the auto stats async update feature is that if you don’t you may see queries time out when auto stats starts to kick in.

Auto stats as we all know update the statistics when the correct number of rows in the table have changed.  When you’ve got very large tables that then trigger auto stats to run if it takes more than 30 seconds for the update stats command to run the query that triggered the auto stats to time out, which causes that transaction to roll back, which means that the auto stats command will also roll back.  So the next query will then fire the auto stats update and the process will repeat over and over.

The symptoms that you’ll see on the SQL Server are queries which are timing out at random even though the execution plan looks totally normal.  You’ll also see a massive amount of IO being generated on the disks which are hosting the database as auto stats does a lot of querying of the table and as auto stats is running over and over you’ll be thrashing the hard drive pretty quickly.

When you turn on the async auto stats setting on the database when the auto stats is triggered by the SQL Server the query in question doesn’t have to wait for the update stats command to finish.  Instead the update statistics command runs in the background letting the query continue to run as normal.  Now the query will run using the old statistics which is probably OK in this case as they were ok 2 seconds earlier so if they are used for a few more seconds it probably isn’t all that big of a deal.

Now I don’t recommend turning this setting on for every database on the server.  All of the smaller databases will update statistics just fine within the timeout period.

So the table that I ran across in my case where I had to turn this setting on took over 2 minutes in order to manually run update statistics on the table so using synchronous statistics updates via auto stats was basically useless.

Denny

System.Data.SqlClient.SqlException: Error 8115, Level 16, State 1, Procedure SomeStoredProcedure, Line 103, Message: Arithmetic overflow error converting IDENTITY to data type int.  Uncommittable transaction is detected at the end of the batch. The transaction is rolled back.

Ourshort term solution was quick and easy, to change identity seed of the column from 0 to the lowest possible number, in this case -2,147,483,648 which got us back up and running pretty quickly.  This however isn’t a very long term solution for this system.  The application is only about 3 or 4 years old, and has been growing very quickly over the last year or so.  We estimated based on the last few weeks worth of data that we would run out of negative numbers within about 12 months at the most.  We sent an estimate of 9 months to the business when we advised them that the system was back up and running.  We also told them that we wanted this fixed within 5-6 months to be save because if we didn’t get this fixed before running out of negative numbers there wouldn’t be any short term fix and that we’d be looking at a multi-day outage to fix the problem.

We couldn’t just rush into a database side fix, as the fix in this case is to change the data type from INT to BIGINT.  As the application does use this column in a couple of places the .NET application needed to be reviewed to ensure that anything that was looking for an INT was corrected to handle the BIGINT correctly.

Based on the amount of data within the table (about 300 Gigs) it was decided that taking an outage to make the change in place wasn’t really an option as doing the size change in place would require somewhere around a 5 day outage to remove and rebuild all the non-clustered indexes.  To make things a little more complex there is a second table which has a 1=1 relationship with this table, and the second table is even larger (about 800 Gigs), though thankfully the second table doesn’t have any non-clustered indexes.

The solution that was decided on was to move the data from the current table to a table with a BIGINT data type while the application was running.  To do this meant that we needed to get all the data copied over to the new table and in sync while the old table was being used.  It was decided that the easiest way to do this would be to use triggers.  In this case instead of one complex trigger to handle all insert, update and delete operations three separate triggers were used for each of the two tables.  First I created two new tables, with the exact same schema as the old tables, with the exception that the new tables used the bigint data type for the primary key instead of the int data type for the primary key and the new table was setup with the ident being the next available positive number.  Once that was done the triggers were setup.  The insert trigger is pretty basic.  Take the data that was just loaded and stick it into the new table.

CREATE TRIGGER t_MyTable1_insert ON dbo.MyTable1
FOR INSERT
AS
SET IDENTITY_INSERT MyTable1_bigint ON
INSERT INTO MyTable1_bigint
(Col1, Col2, Col3, Col4…)
SELECT Col1, Col2, Col3, Col4
FROM inserted
SET IDENTITY_INSERT MyTable1_bigint OFF
GO

The update and delete triggers required a little more logic.  The trick with the triggers was that I needed to avoid doing massive implicit data conversions.  In order to ensure that SQL was doing what I wanted (which it should be doing anyway, but it made me feel better doing explicit conversions) I explicit conversions into place for the JOIN predicates as shown.  The update trigger is shown first, then the delete trigger.

CREATE TRIGGER t_MyTable1_update ON dbo.MyTable1
FOR UPDATE
AS
UPDATE MyTable1_bigint
SET MyTable1_bigint.[Col2] = inserted.[Col2],
MyTable1_bigint.[Col3] = inserted.[Col3],
MyTable1_bigint.[Col4] = inserted.[Col4],
…
FROM inserted
WHERE cast(inserted.Col1 as bigint) = MyTable1_bigint.Col1
GO

CREATE TRIGGER t_MyTable1_delete ON dbo.MyTable1
FOR DELETE
AS
DELETE FROM MyTable1_bigint
WHERE Col1 in (SELECT cast(Col1 as bigint) FROM deleted)
GO

Once these tables were up and running all the new data changes were being loaded into the table.  At this point it was just a matter of coping the existing data into the table.  There are a few ways that this can be done.  In this case I opted for an SSIS package with a single data pump task, and two data transformations within the data pump task with one transformation for each table.

In order to make the load as fast as possible I used the fast load option and loaded 1000 rows at a time.  Within the data task if there was an error I redirected the rows to another data pump task which simply dropped the rows into the same table, but this time going row by row.  Any failures from that import were simply ignored.  While handling failed rows like this is time consuming it is easier than running T-SQL scripts to verify which rows are needed and which rows aren’t needed.  SSIS also gives an easy option to ignore the foreign key relationship between the two tables so if the child table gets rows first that isn’t a problem as we know that the parent table will catch up.  The SSIS package looked like this:

When all is said and done and the data is in sync between the new and old tables, the current tables will be dropped and the new tables will be renamed and put into place so that the application can continue to run without issue, with just a few minutes of downtime.

So why did this happen?  When the applications database was being designed the developers didn’t think about how many rows the database was going to get over time, so they didn’t account for needing to support more than 2.1 billion rows over time.  If I (or another data architect) had been involved in the project at it’s start this hopefully would have been caught at design time.  However when the application was first being designed the company was brand new and didn’t have the funds for a data architect to help with the application design so this problem was missed.

Hopefully you never hit this problem, but if you do this helps you get out of it.

Denny

To solve this problem, I added a DDL trigger to the server which will cause the new database to be backed up as soon as the database is created.  The trigger itself is rather simple.  Most of the trigger is setting variables.  Then I make sure that the database isn’t a database snapshot, as database snapshots can’t be backed up.  If it isn’t a snapshot we continue with everything else.

Then I create a subfolder for the backups to be put into (the backups for each database go into their own folder, so as this is a new database the folder needs to be created).  Then I commit the transaction, as database backups can’t be taken within a transaction.  Then we do the actual database backup.  I then throw a message to the end user using the RAISERROR statement telling them that they can ignore the other error that they are going to get about the transaction being closed before the trigger was complete.  This is just an annoyance of my needing to commit the transaction before taking the backup.  Sure I could have setup a job which takes the backup and emails if there was a failure, but that just seems to complex for something so simple.  The code for my trigger is below.

CREATE TRIGGER BackupNewDatabase
ON ALL SERVER
FOR CREATE_DATABASE
AS
declare @database sysname, @event_data XML = EVENTDATA(), @folder nvarchar(4000), @file nvarchar(4000)

SET @database = @event_data.value(‘(/EVENT_INSTANCE/DatabaseName)[1]‘, ‘sysname’)

set @folder = ‘X:\Backups\’ + @database

set @file = @folder + ‘\’ + @database + ‘.bak’

if exists (select * from sys.databases where name = @database and source_database_id is null)
BEGIN
EXEC master.dbo.xp_create_subdir @folder

COMMIT

BACKUP DATABASE @database to disk=@file

raiserror( ‘You can ignore the error message which says that the transaction ended within the trigger.’, 16,1)
END
GO

Hopefully you find this solution helpful if you get into a situation like this,
Denny

In this technique we’ll encrypt the data using just a single column.  This technique requires butting some additional logic within the application to figure out if the value is encrypted or not, but other than that logic, which you can leave in and strip out later the changes to the application are pretty minimal as the column stays the same, so that means that the stored procedures don’t need to be changed.

The first thing to remember is that the encrypted data will be larger, possibly much larger than the plain text version of the data.  Because of this you’ll need to increase the size of the field which you’ll be putting the data into.  Now the good news is that if this column isn’t indexed this change should be pretty quick and easy as it should just be a meta change which tells the SQL Server that the column size can be bigger without having to actually change the pages.  You can see this by making some changes to the [HumanResources].[Employee] table within the AdventureWorks database.  By turning on STATISTICS IO and using the ALTER TABLE statement we see that there is no IO generated when we change the size of the LoginID column from nvarchar(256) to nvarchar(512).

set statistics io on
alter table MyTable
alter column LoginID nvarchar(512)

Once the column is made larger the .NET code needs to be modified to see if the data is compressed for not.  Now there is no sure fire way to check to see if a value has been encrypted or not, but a pretty good test is to look at the last two characters of the value.  If they are both an equal sign (==) then it is probably safe to assume that the value is encrypted.  To don’t want to just attempt to decrypt the data and look for an error message, and if there is an error assume that the encrypted value is in plain text, throwing and catching error messages in .NET is very expensive, especially compared to simply checking to see if the last two characters are an equal sign.  This isn’t to say that you shouldn’t have TRY/CATCH logic around the code that decrypts the values as someone could easily enough put two equal signs at the end of their password.

At this point either a .NET app or a T-SQL script can loop through the values in the table which aren’t encrypted and then encrypt them, updating the rows which aren’t already encrypted.

Look for more blog posts in this series on how to encrypt data which already exists within your applications database.

Denny

This tweet from Daniel (@DaniSQL) is a perfect example of this.

An old application that isn’t being used anymore is still available on the Internet facing web farm.  Because this application isn’t being used any more it wasn’t on any lists of deployed applications, so when security audits were done it wasn’t seen as it wasn’t listed as an active application.  However it was apparently able to provide a hacker with a way into the database because it was still connected to a SQL Server instance and it was susceptible to SQL Injection.

The solution to this problem is sadly easy, remove the web based application from the web farm as the application isn’t being used anywhere.  It’s a lot easier than fixing the application, and a whole lot cheaper (10 minutes of a system administrator’s time versus weeks or months of a developers time).

I urge you to audit the applications and websites which are deployed to your web farms, especially the Internet facing web farms and see what’s on there.  When you audit them, don’t audit them against the list of what’s there.  Actually dig into the IIS config of each and every server (yes I’m well aware that doing this sucks) and actually see what’s configured on each machine.  If you don’t know if an application is actually being used ask around.  If it isn’t, remove it (or at least stop the site in IIS) so that you don’t have to worry about scripts breaking into your database and updating your data.

Now thankfully this current attack which is going around is just updating data, but it could easily enough be changed by the attacker to gather data as well, so do yourself a favor and protect yourself.

Denny