All this sounds perfect (except for if I’m out of the country and I can’t get their text messages), except for one little issue.

Adding a new cell phone to send a text message to is as simple as just logging onto the bank’s webpage. Once I log into the site I can simply add another cell phone, verify that I have the cell phone via a text message and then I can use that cell phone to approve any wire transfers. All very convenient. The problem is that is someone else figures out my username and password for the website they to can add a cell phone to my bank account, approve it for use, then start sending wire transfers off all my money to their account.

So while Bank of America has two factor authentication, the second factor is dependent on knowing the first factor. For this to be actually useful two factor authentication it would need to require that I go into a branch with my ID to prove that I’m me and that I can add the phone as a two factor authentication phone. Additionally they should be using as an option one of the phone application based two factor authentication processes so that if I have several phones I can just use the one application, or if I’m not in the country I can still manage my money (which has been a problem a couple of times).

While I applaud the effort that Bank of America has put into having two factor authentication, doing it correctly would be a lot more useful.  As currently you have one factor authentication with an annoyance.

Denny

This precon is a full day session where we will be talking about all the security best practices.

Signing up for the precon is pretty simple, just do to the URL for the precon and fill out the form.

I hope to see you there,
Denny

While attacking this problem, the first thing that I did was look into the SSIS package and see what I had gotten myself into.  There were hundreds of objects within SSIS, most of them T-SQL Scripts of Data Pump Tasks.  So simply going through them one by one wasn’t going to be all that helpful.  The client didn’t have any logging to SQL Server or to a text file setup, but they did have the SSIS package setup to log to the Windows event log.

Opening the Application log greated me with one event for the start of each task, and one for the completion of each task.  Not exactly the most useful information ever, but better than nothing, so I’ll take what I can get.  The first thing that I did was export the Application Event log to a CSV file and download it to my desktop so that I could do the processing on my local machine.  The CSV file has a few hundred thousand lines, so I needed to figure out what I had via scripting.  The first problem that I had was that for each SSIS event there were multiple lines within the csv when opened in Excel which you can see below.

Thankfully this made the rows pretty unique looking, so an Excel Macro was able to clean this up pretty easily.  Before I got started I added some column headers into the excel sheet.  The default columns from Event Viewer are Date, Time, Source, Severity, Category, EventID, User, Computer, Description.  This covered columns A through I.  After that I put columns J through P as being Operator, SourceName, SourceID, Execution, StartTime, EndTime and DataCode as these are the specific items that SSIS is logging that you can see above.  Then it was time to bust out a little VBA and flatten out all this data.  The macro shown below looped through the data and flattened everything out so that all the data was on a single line.  Then is went through and removed all the white lines.  This macro took about 3 hours to run.

Sub CleanUpData()
Dim Row As Long
Dim DataRow As Long
Row = 32768
Do While Range("A" & Row).Value = ""
If Range("B" & Row).Value = "" Then
DataRow = Row
Range("A" & Row).Select
Else
If Left(Range("A" & Row).Value, 4) = " Mes" Then
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 4) = " Ope" Then
Range("J" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 9) = " Source N" Then
Range("K" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 10) = " Source ID" Then
Range("L" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 4) = " Exe" Then
Range("M" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 4) = " Sta" Then
Range("N" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 4) = " End" Then
Range("O" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
If Left(Range("A" & Row).Value, 4) = " Dat" Then
Range("P" & DataRow).Value = Range("A" & Row).Value
Range("A" & Row).Value = ""
End If
End If
Row = Row + 1
Loop

Do While Row 2
If Range(“A” & Row).Value = “” Then
Range(Row & “:” & Row).Select
Selection.Delete
End If
Row = Row – 1
Loop
End Sub

Once that was done I needed to set the start times for each event on the same line as the end time for each event. That way I wasn’t scrolling through looking for stuff while trying to do my data analysis. This I was also able to do with a little Macro. In this case I started on row 14 as that was the first entry that had SSIS data. You may need to change the EndRow=14 line to what ever line your data starts on. This puts the start time and the end time into Columns R and S, but only on the line which is the OnPostExecute line.

Sub FindRunTimes()
Dim EndRow As Long
Dim StartRow As Long
Dim SourceName As String
EndRow = 14
Do While Range("A" & EndRow).Value = ""
If Range("I" & EndRow).Value = " Event Name: OnPostExecute" Then
StartRow = EndRow + 1
Do Until Range("K" & EndRow).Value = Range("K" & StartRow).Value And Range("I" & StartRow).Value = " Event Name: OnPreExecute"
If Range("I" & StartRow).Value = "" Then
MsgBox "Ran out of rows to process for ending row " & EndRow
Exit Sub
End If
StartRow = StartRow + 1
Loop
Range("S" & EndRow).Value = Range("B" & EndRow).Value
Range("R" & EndRow).Value = Range("B" & StartRow).Value
'Range("T" & EndRow).Value = DateDiff("mi", Range("A" & StartRow).Value & " " & Range("B" & StartRow).Value, Range("A" & EndRow).Value & " " & Range("B" & EndRow).Value)
Range("R" & EndRow).Select
End If
EndRow = EndRow + 1
Loop
End Sub

Once that Macro was finished I needed to figure out which of the statements was taking to long to run. Putting a formula into the column Q marked the rows that I needed to look into a little more. In this case I was using anything with a run time of 20 minutes or longer as needing to be looked at.

=IF(HOUR(R595)=HOUR(S595)*60+MINUTE(S595),"", "X"), IF(HOUR(R595)*60+MINUTE(R595)+19>=HOUR(S595)*60+MINUTE(S595)+1440,"", "X"))

Once that formula was in there all the rows that were SSIS objects that took a long time to run had an X next to them. Now I still needed to ignore the rows which were containers, but those were pretty easy to figure out from the names of the objects as most of them had container in them (in the SourceName column which was column K of my Worksheet). At this point I was able to move back to the SSIS package and start digging through the package finding the objects which had long runtimes and getting those looked at.

The whole reason I went through this process instead of just adding logging to the SSIS package was for a few different reasons.

• I didn’t want to make ANY changes to production if I didn’t have to. This SSIS package loads a production data warehouse which the business uses to run their company, so making changes to it wasn’t something I wanted to try.
• If I had simply put logging into place I would have had to have waited another day to being looking at the problems as the ETL process only runs once a day.
• The SSIS package takes 8+ hours to run, so sitting around waiting for it to run wasn’t exactly the best use of my time, or the clients money.

If you get stuck in this same sort of problem, hopefully this approach will help you out.
Denny

Loyalty Programs

Pick an airline, rental car company and hotel chain and do your best to stick with them.  It doesn’t really matter which one you pick, and everyone has their favorites for one reason or another.  In my case I’m a Hilton guy for hotels, a United guy for flights and an Avis guy for car rentals.  I’m signed up for all three of their loyalty programs so I earn points each time I use their services.  In fact Avis and Hilton even offer bonus programs so that I can earn extra United miles each time I stay at a Hilton or rent a car from Avis.

In addition to earning points and miles you get something more important from being members of these programs, upgrades.  Now don’t get your hopes up that you’ll be flying first class every time you book a coach ticket on an airline just because you signed up for their loyalty program.  Free upgrades require having status with the airline, and who ever has the highest status gets the upgrades first.  In the case of United there are four levels, Silver which requires 25k miles of travel, gold which requires 50k miles of travel, platinum which requires 75k miles of travel and “1k” which requires 100k miles of travel.  Now these free upgrades for the most part are only good on flights within the US on the airline of your choice and their partner airlines.  So before you pick an airline do a little research on what the flight options are from your home airport and what partners they have and how the flights to where you will usually be traveling will work.

With car rental companies you get some nice bonuses just for joining their programs, usually free upgrades with a minimum rental level, skipping the counter, etc.  For example with Avis just for joining the program if you rent an intermediate class car or above if there is something nicer available they will automatically assign you that car as a rental for free.  Some upgrades that I’ve been given before include: Prius, Chrysler 300 M, Mustang, and Camaros just to name a few.

Another great way to earn status is to get an airline or hotel credit card.  If you can get qualified for some of the higher end credit cards you can get some really nice perks.  Now these cards aren’t free, but if you use the perks to your advantage they can be worth every penny.  An airline’s higher level card such as the United Mileage Plus Explorer card gets you on the plane earlier (even if you don’t have status to get on earlier yet), a free checked bag and two passes into the United club lounge.  Plus you earn frequent flier mile for every dollar that you spend using the card.  There’s actually a card which is above the Mileage Plus Explorer card called the United Club card, which is the one that I’ve got.  This card gives you all the access to the club lounge you can use, bonus miles when paying for air travel, platinum status at Hyatt, presidents club service at Avis (so double class upgrades on rentals), plus all sorts of other really useful perks like free travel insurance, buyers protection for anything you buy, etc.

With the hotel programs once you stay enough nights (the number of nights per level will vary depending on which hotel chain that you use) you’ll start getting room upgrades for your hotel rooms as well.  Free upgrades to the club or executive level, free WiFi, free breakfast, etc.

The other nice thing about all these loyalty programs is that you are racking up all these points on business trips you can use these points to take yourself and the family on some pretty nice vacations for free or next to free.  I can use my United points to not only pay for flights on United and their partner airlines, but I can also use my United points to pay for hotels, rental cars, etc. Now don’t just use the points, look to see what the rentals will actually cost you because it might be worth paying cash for some stuff.

Keep In Touch

Call home, daily.  Now I know that work schedules and time zones may not permit long phone calls home on a regular basis, but a few minutes when you can helps out a lot.  I always make a point of calling when I’ve landed in my destination city.  I usually wait until after I’ve gotten to my rental car and figured out where I’m going (usually using my phone as my GPS).  Skype is great for this because you can not only talk to each other, but you can see each other as well.  If you have a long trip where you are away for a couple of weeks this can be great so that the people at home can remember what you actually look like.  Of course things like Apple’s Face Time work great for this as well.

Bring Stuff Home

When you get home from somewhere, especially somewhere that you’ve never been before make sure that you bring back something besides dirty laundry.  You’ll want to make sure that you bring back something to show the family that you’ve been thinking about them.  Try to avoid the crap in the airport if you can (granted some airports have some pretty nice stuff).  Don’t forget when packing to leave room in your bags to bring stuff home for the family, especially if you are going somewhere that your spouse would want to go but can’t.

Take The Family With You

If you are going somewhere really cool, try and take the family (or at least the spouse) with you on the trip.  Add a few extra days onto one end or the other of the trip to do some sight seeing and explore where you are.  Especially if you are going somewhere really cool like another country that either one of you have ever been to.  Thanks to my travels I’ve been lucky enough to take Kris (my wife) to Amsterdam (twice), Belgium, and London so far.  Later this year when we go back to Belgium and Amsterdam we are planning on going to Paris as well so that we can go somewhere new as well to places that we already love.  The biggest expense (at least for Americans flying to places like Europe) is the flight cost, a few extra nights in the hotel really isn’t all that expensive.  Book the flights carefully, maybe booking the spouses flight using some miles to keep that cost down.

When flying with the family, especially within the states when you have status with the airlines and there’s a chance that you’ll get an upgrade, make sure to book every ones tickets together on the same purchase.  The reason for this is that if you do get an upgrade, the airline will also upgrade one of the other people in your party so your spouse can also get an upgrade to first class with you.

Use The Down Time

When you are traveling there is lots of down time which can really suck if you’ve got a lot of stuff to do.  Many airports have WiFi available for free or for a small charge.  If you often travel via airports that charge for WiFi look into a Boingo Account.  Boingo either provides WiFi for most airports that do charge, or they are partnered with the airports that charge for WiFi.  If you are often at these airports a monthly Boingo account will probably end up being cheaper than paying for daily access at the airports.  If you sign up for the airlines club lounge they will usually provide you with free WiFi so that you can get online from within the club lounge.

When you get to the airport find yourself a nice place to settle down and work.  This way you can use the down time during layovers or while you are waiting at the airports to catch up on email, projects, meetings, etc.  Personally I like to work on writing for my blog, or the next article that I have due.  On some flights you can get WiFi for just a few dollars a flight so that you can get online during your flight as well.  Again if you fly often paying for a monthly account will probably be cheaper than paying for access on each flight.  Currently Delta has the best WiFi service available.  United doesn’t currently offer WiFi on many if any flights, but they are working on rolling out WiFi to the majority of their fleet (or at least that’s what the video before the safety video keeps telling me).  Even some international flights between the US and Europe offer WiFi on the flight so you can stay connected on those long haul flights as well.

Now the speed on the in flight WiFi isn’t the greatest, but it’s fast enough to VPN into a client’s network and do some work, send and receive email, or even burn a little time on Twitter, though I wouldn’t try playing any games on Facebook over the WiFi.

Pack Light

One of the most important things that you’ll learn is to pack light.  Take only what you need for the trip that you are on otherwise you are just lugging stuff around the airport for no reason.  As you get used to packing you’ll be able to get more and more stuff into smaller and smaller bags.  Personally I’m able to do a 9 day trip in a single roller bag and my laptop bag.  Speaking of roller bags the right sized bag is very important. You want a narrow bag that tapers to a smaller end at the top so that it’ll fit in those funny shaped overhead bins that the A319s and smaller 737s have.

When you select the clothing that will be taking, try to take clothes make from materials that are light and that don’t wrinkle easily.  When packing don’t be to worried about things getting a little wrinkled. That is going to happen no matter how careful you are.  I’ve taken garment bags or roller bags, and everything gets wrinkled about the same.  99% of hotels have an iron in them if you are so inclined.  If you are like me and hate ironing hang the clothes in the bathroom, probably off the hook on the back of the door, and turn on the shower to hot.  This will steam up the room and get most of the wrinkles out of the clothes.  It’ll only take 5-10 minutes in the steam.  Then take them and hang them in the room or the closet with plenty of room to breath so that they can dry completely as they may be a bit damp.  If they are wet you left them in there way to long.

If you need to wash something, there’s a cleaning service in most hotels but it is going to be pretty expensive.  You can usually find a self service room with a washer and a drier in it that you can use.  A full load in the washer and drier will usually cost you a couple of dollars.  If you only need to wash a shirt or something like that using the hotel room sink will usually work pretty well.  After you wash it, lay a towel on the floor and put the shirt you just washed on top of the towel.  Roll it up loosely into a roll, then stand on it to squeeze the water out of the shirt.  The towel will absorb most of the water.  Then you can hang dry the item in the room (not in the closet as it won’t get enough air flow in there).  If you need to wash a few things using this technique just ask house keeping for a few extra towels.  They’ll be happy to deliver these to you.  Most hotel shampoo will make a decent enough clothing wash as it’s usually pretty mild.  Just put it in the water so that it mixes well before putting the clothing into the sink.

Receipts

Managing receipts for work or for a client is probably the worst part of traveling.  Some companies want every single receipt, even all the $1 charges on the toll roads while others want just the big ones as they’ll trust you on all the small stuff.  What I do, which works pretty well for me is I grab a standard size envelope before I leave home, or from the hotel when I check in.  All my receipts go into that envelope at the end of each day of the trip.  As I’m on the road all the time I also make sure to write the name of the trip, or the destination, as well as the dates on it as often I’ll have two or three of these envelopes in my bag by the end of a long trip.  My laptop bag has side pockets that are the perfect size for these envelopes to stick out of.  Just make sure that the envelopes fit snugly in there so that they don’t fall out.  Personally I put my airline boarding passes in there after my flights as well.  Just in case the airline doesn’t credit me correctly for my miles flown.  Sometimes they’ll want the actual boarding pass if you had a paper boarding pass.

Having everything in these envelopes does a couple of things for me.  It makes expense reports and invoices much easier to deal with as everything is already sorted (I make sure that flight receipts from when I purchase the airline tickets go in these envelopes as well).  Secondly it makes tax season much easier as everything is already broken down by trip already, so anything that I need to deduct I can quickly figure out.  If you are so inclined you could even write on the envelope a summary of the receipts in it to make tax season that much faster.  Then instead of giving the accountant a huge box of receipts to figure out, they (or you if you do your own taxes) just have to flip through the envelopes to figure everything out.

Helpful?

Hopefully you were able to get some useful information from this, weather you are a first time traveler or a frequent flier like myself.  The one thing that I’ve learned when traveling is that there isn’t any one way to travel correctly and easily.  The trick is to find things that work for you best and then keep doing them.  Consistency is the key to making travel easy.

Denny

Officially the answer is no, however with a little bit of creative configuration you sure can.

The Overall Environment

To setup SQL Server Express in a Windows Cluster I’m building this on a two node Windows Server 2012 cluster using a file share hosted on my domain controller to host the actual databases.  To ensure that the domain controller is rebooted as little as possible the domain controller is installed in core mode. The cluster nodes are Windows Server 2012 standard edition (which now supports clustering) as is the domain controller.

Installation

As SQL Server 2012 express edition doesn’t support Windows Clustering out of the box the installation will be a little different from doing a normal clustered install under standard or enterprise edition.  To install I did a normal SQL Express install on node1.  The only change from a normal install that I made was that I configured the SQL Server instance to start under a domain account.  When I got to the data directories part I configured the data folder to a network share on the domain controller.

Once the installation on node1 was completed I stopped the SQL Server services.  Then I renamed the folder that I installed the SQL Server database files into.  The reason for this is that I need to configure the second instance to put the database files into the same location.  I can then install SQL Server 2012 express edition onto the second node.

The installation on node2 is done exactly like it was done on node1.

Once the installation is done on both nodes configure the SQL Server service to have a startup type as “Manual” instead of disabled or automatic.  Leave the SQL Agent service as disabled as even though SQL Express installs the SQL Agent the SQL Agent isn’t supported on SQL Express.

Configuring Clustering

Once the installation on Node2 is done the cluster can be configured.  To do this bring up the Failover Cluster Administrator on one of the nodes and connect to the cluster.  If the cluster hasn’t been configured yet run through the normal Clustering Configuration wizard.

We’ll now configure a new cluster role on the cluster.  To do this right click on “Role” then select “Configure Role” from the context menu as shown below.

When the wizard opens click next to get to the list of services.  Then select the Generic Service item from the list as shown below.

On the next screen you’ll be asked what service you wish to cluster.  From this list select the SQL Server service as shown below.

On the next screen you’ll be asked to name the resource group.  Give the group a name which is unique on the domain and click next.  The next screen will ask you to select the needed storage.  Simply click next on this screen as we aren’t using any local shared storage.  The next screen asks you if any registry settings need to be replicated between the machines.  We don’t need to replicate anything as SQL Server doesn’t make much use of the registry for the actual SQL Server service so we can simply click next on this screen as well.  The next screen is simply a screen to review the changes which will be made.  You can simply click next on this screen after reviewing the information on the screen.  When the summary screen displays click finish.

Post Clustering SQL Config Changes

The first change that you’ll need to make is to enable the TCP network protocol on both nodes.  By default SQL Express has the TCP network protocol disabled which need to be corrected before uses will be able to connect to the SQL Server service.

The next change that you’ll need to make is to change the local server name in the master database from the name of the last node which was installed to the cluster name using a script similar to the one shown below.  In the case of this script the nodes are named node1 and node2 and the cluster name is clustersql. Once this script has been run the SQL Server instance should be restarted or failed over to the other node.

exec sp_dropserver ‘nodeb’
GO
exec sp_addserver ‘clustersql’, local
GO

At this point the cluster is up and running and applications can have their databases configured on the SQL Server Instance.

Denny

When turning on this trace flag because you’ve got a large database the goal obviously is to use autostats so you’ll need to have auto stats turned on.  In addition you’ll want to turn on the “Auto Update Statistics Asynchronously” setting for the database or databases which hold the super large table.  The reason that you’ll want to turn on the auto stats async update feature is that if you don’t you may see queries time out when auto stats starts to kick in.

Auto stats as we all know update the statistics when the correct number of rows in the table have changed.  When you’ve got very large tables that then trigger auto stats to run if it takes more than 30 seconds for the update stats command to run the query that triggered the auto stats to time out, which causes that transaction to roll back, which means that the auto stats command will also roll back.  So the next query will then fire the auto stats update and the process will repeat over and over.

The symptoms that you’ll see on the SQL Server are queries which are timing out at random even though the execution plan looks totally normal.  You’ll also see a massive amount of IO being generated on the disks which are hosting the database as auto stats does a lot of querying of the table and as auto stats is running over and over you’ll be thrashing the hard drive pretty quickly.

When you turn on the async auto stats setting on the database when the auto stats is triggered by the SQL Server the query in question doesn’t have to wait for the update stats command to finish.  Instead the update statistics command runs in the background letting the query continue to run as normal.  Now the query will run using the old statistics which is probably OK in this case as they were ok 2 seconds earlier so if they are used for a few more seconds it probably isn’t all that big of a deal.

Now I don’t recommend turning this setting on for every database on the server.  All of the smaller databases will update statistics just fine within the timeout period.

So the table that I ran across in my case where I had to turn this setting on took over 2 minutes in order to manually run update statistics on the table so using synchronous statistics updates via auto stats was basically useless.

Denny

System.Data.SqlClient.SqlException: Error 8115, Level 16, State 1, Procedure SomeStoredProcedure, Line 103, Message: Arithmetic overflow error converting IDENTITY to data type int.  Uncommittable transaction is detected at the end of the batch. The transaction is rolled back.

Ourshort term solution was quick and easy, to change identity seed of the column from 0 to the lowest possible number, in this case -2,147,483,648 which got us back up and running pretty quickly.  This however isn’t a very long term solution for this system.  The application is only about 3 or 4 years old, and has been growing very quickly over the last year or so.  We estimated based on the last few weeks worth of data that we would run out of negative numbers within about 12 months at the most.  We sent an estimate of 9 months to the business when we advised them that the system was back up and running.  We also told them that we wanted this fixed within 5-6 months to be save because if we didn’t get this fixed before running out of negative numbers there wouldn’t be any short term fix and that we’d be looking at a multi-day outage to fix the problem.

We couldn’t just rush into a database side fix, as the fix in this case is to change the data type from INT to BIGINT.  As the application does use this column in a couple of places the .NET application needed to be reviewed to ensure that anything that was looking for an INT was corrected to handle the BIGINT correctly.

Based on the amount of data within the table (about 300 Gigs) it was decided that taking an outage to make the change in place wasn’t really an option as doing the size change in place would require somewhere around a 5 day outage to remove and rebuild all the non-clustered indexes.  To make things a little more complex there is a second table which has a 1=1 relationship with this table, and the second table is even larger (about 800 Gigs), though thankfully the second table doesn’t have any non-clustered indexes.

The solution that was decided on was to move the data from the current table to a table with a BIGINT data type while the application was running.  To do this meant that we needed to get all the data copied over to the new table and in sync while the old table was being used.  It was decided that the easiest way to do this would be to use triggers.  In this case instead of one complex trigger to handle all insert, update and delete operations three separate triggers were used for each of the two tables.  First I created two new tables, with the exact same schema as the old tables, with the exception that the new tables used the bigint data type for the primary key instead of the int data type for the primary key and the new table was setup with the ident being the next available positive number.  Once that was done the triggers were setup.  The insert trigger is pretty basic.  Take the data that was just loaded and stick it into the new table.

CREATE TRIGGER t_MyTable1_insert ON dbo.MyTable1
FOR INSERT
AS
SET IDENTITY_INSERT MyTable1_bigint ON
INSERT INTO MyTable1_bigint
(Col1, Col2, Col3, Col4…)
SELECT Col1, Col2, Col3, Col4
FROM inserted
SET IDENTITY_INSERT MyTable1_bigint OFF
GO

The update and delete triggers required a little more logic.  The trick with the triggers was that I needed to avoid doing massive implicit data conversions.  In order to ensure that SQL was doing what I wanted (which it should be doing anyway, but it made me feel better doing explicit conversions) I explicit conversions into place for the JOIN predicates as shown.  The update trigger is shown first, then the delete trigger.

CREATE TRIGGER t_MyTable1_update ON dbo.MyTable1
FOR UPDATE
AS
UPDATE MyTable1_bigint
SET MyTable1_bigint.[Col2] = inserted.[Col2],
MyTable1_bigint.[Col3] = inserted.[Col3],
MyTable1_bigint.[Col4] = inserted.[Col4],
…
FROM inserted
WHERE cast(inserted.Col1 as bigint) = MyTable1_bigint.Col1
GO

CREATE TRIGGER t_MyTable1_delete ON dbo.MyTable1
FOR DELETE
AS
DELETE FROM MyTable1_bigint
WHERE Col1 in (SELECT cast(Col1 as bigint) FROM deleted)
GO

Once these tables were up and running all the new data changes were being loaded into the table.  At this point it was just a matter of coping the existing data into the table.  There are a few ways that this can be done.  In this case I opted for an SSIS package with a single data pump task, and two data transformations within the data pump task with one transformation for each table.

In order to make the load as fast as possible I used the fast load option and loaded 1000 rows at a time.  Within the data task if there was an error I redirected the rows to another data pump task which simply dropped the rows into the same table, but this time going row by row.  Any failures from that import were simply ignored.  While handling failed rows like this is time consuming it is easier than running T-SQL scripts to verify which rows are needed and which rows aren’t needed.  SSIS also gives an easy option to ignore the foreign key relationship between the two tables so if the child table gets rows first that isn’t a problem as we know that the parent table will catch up.  The SSIS package looked like this:

When all is said and done and the data is in sync between the new and old tables, the current tables will be dropped and the new tables will be renamed and put into place so that the application can continue to run without issue, with just a few minutes of downtime.

So why did this happen?  When the applications database was being designed the developers didn’t think about how many rows the database was going to get over time, so they didn’t account for needing to support more than 2.1 billion rows over time.  If I (or another data architect) had been involved in the project at it’s start this hopefully would have been caught at design time.  However when the application was first being designed the company was brand new and didn’t have the funds for a data architect to help with the application design so this problem was missed.

Hopefully you never hit this problem, but if you do this helps you get out of it.

Denny

On the SQL Server side of things…

• Verify your most recent full backup by restoring it to another server (both user and system databases)
• Make sure that your backups are copied to another servers or USB Drive
• Get a copy of the backups out of the facility (a USB drive is a great way to do this, the network will work if needed)
• Generate some scripts to ensure that you can restore the databases to another machine when ready
• A way to get the transaction log backups off site, maybe FTP’ed to a personal website (after encrypting the files)

On the Hyper-V / VMware side of things…

• Do you have backups of critical VMs?  If not take them now.
• Get a copy of the backups out of the facility (a USB drive is a great way to do this, the network will work if needed)
• Don’t forget to do a system state backup of Active Directory

If there is a disaster, the things that you’ll need at the new data center will include…

• Servers (get your order in as soon as you know that your data center doesn’t exist any more, you won’t be the only one ordering servers)
• Switches, Routers, etc.
• Lots of USB Hard Drives (The SAN won’t be available, and it’ll take weeks or months to get a new one)
• Zip ties/cable ties
• Network cables
• All the backups that you took from above
• Installation media for all your OSes and databases servers
• As lots of things

As you are getting the installations all setup, some things to keep in mind…

• You’ll want to enable Instant File Initialization
• You’ll want to restore system databases, then user databases
• Restore AD from the system state backup, otherwise all the Windows authentication SIDs will be useless within SQL Server

If your data center looses power and it’ll be out for a while, and you don’t have a big expensive generator to run the data center know where you can get a generator.  A few small ones from Lowes, Home Depot, etc. will work as well for keeping the most critical systems online.  Get some large box fans to keep the air in the data center moving.  If you are using small generators, use a different one for the fans as they will have very dirty power, which could damage the power supplies of the servers.  Plastic sheeting can be used to direct air in or out of the data center so that you get cool air coming in from outside and so you can direct the hot air back out (ask me some time about turning a 6 story building into a giant chimney for a data center).

Once we are in a disaster situation we are no longer worried about peak performance, we are worried about keeping the business running for our customers.  Don’t worry about redundant power supplies, or passive nodes, reporting servers, etc.  Just worry about getting the core systems online so that websites work, and so that customers can login to your applications.  If your application supports it be sure to put up a message telling your customers that performance won’t be very good because of the weather.  Just because you know that the weather is a mess doesn’t mean that your customers are aware that there’s a problem.  Most people are pretty understanding and if you tell them that the weather is causing performance problems they will be fine with that.  This will save you a lot of phone calls and a lot of headache.

No matter what, when getting things setup, rebuilt, brought back online, etc. do it safely.  Don’t get hurt and risk hurting yourself or others to get things back up and running.  It just isn’t worth it.

Denny

This tweets which Karen Lopez (blog | @datachick) shared shows the exact problem which she has run into when working with one (or more) of her clients.

Setting up DR (or backups in this case) has nothing to do with insulting the IT staff, or that the programmers don’t know what they are doing.  As IT workers our job is to hope for the best, but plan for the worst.  In the case of this week the east coast of the US is being hit with about the worst case that they can get, a full blown hurricane going all the way up the coast.

I don’t care how good you are at racking servers, installing Windows, writing software, etc. if the power at your data center goes out for a week, and they can’t get fuel to the data center for a week (depending on the number of trees which are down between the highway and the data center, this is a real possibility) the systems will be down and you won’t have planned correctly for the worst.

If you think about this from home perspective instead of the work perspective, when a disaster strikes you don’t want to have to rush to the store to try to find bread and other food items to keep yourself and your family fed during the emergency.  If you live somewhere that has regular natural disasters (which is pretty much everywhere at this point) you hopefully have canned food, bottled water, flashlights, etc. at home so that you can ride out this sort of disaster for a few days at the least without running out of food and water.  Why shouldn’t you plan accordingly at the office as well.

This sort of planning isn’t something that can be done last minute, because you can’t always see the disaster coming so you don’t always have time to plan (or shop) right before (or as) the disaster happens.

With some forethought and proper planning any business can ride out any disaster.  But it requires planning ahead of time and the dedication of the company and the employees to properly setup and test the DR solution.  DR projects can be big scary projects if you don’t have someone on staff who has experience with these sorts of things.  But that’s OK, that’s what we have consultants for who specialize in these sorts of projects.  Not every company needs to keep staff on hand that can plan out DR plans, but you should bring someone in who knows how to plan and execute these sorts of projects successfully.  While the consultant may cost a few dollars an hour, it’ll be much less than a failed DR project, and a whole lot less than a failed DR failover.

Denny

(Thanks to Karen for letting me use her Tweets in this post, and for Thomas LaRock (blog | @sqlrockstar) for letting me steal the picture of his kids.)

In this technique we’ll encrypt the data using just a single column.  This technique requires butting some additional logic within the application to figure out if the value is encrypted or not, but other than that logic, which you can leave in and strip out later the changes to the application are pretty minimal as the column stays the same, so that means that the stored procedures don’t need to be changed.

The first thing to remember is that the encrypted data will be larger, possibly much larger than the plain text version of the data.  Because of this you’ll need to increase the size of the field which you’ll be putting the data into.  Now the good news is that if this column isn’t indexed this change should be pretty quick and easy as it should just be a meta change which tells the SQL Server that the column size can be bigger without having to actually change the pages.  You can see this by making some changes to the [HumanResources].[Employee] table within the AdventureWorks database.  By turning on STATISTICS IO and using the ALTER TABLE statement we see that there is no IO generated when we change the size of the LoginID column from nvarchar(256) to nvarchar(512).

set statistics io on
alter table MyTable
alter column LoginID nvarchar(512)

Once the column is made larger the .NET code needs to be modified to see if the data is compressed for not.  Now there is no sure fire way to check to see if a value has been encrypted or not, but a pretty good test is to look at the last two characters of the value.  If they are both an equal sign (==) then it is probably safe to assume that the value is encrypted.  To don’t want to just attempt to decrypt the data and look for an error message, and if there is an error assume that the encrypted value is in plain text, throwing and catching error messages in .NET is very expensive, especially compared to simply checking to see if the last two characters are an equal sign.  This isn’t to say that you shouldn’t have TRY/CATCH logic around the code that decrypts the values as someone could easily enough put two equal signs at the end of their password.

At this point either a .NET app or a T-SQL script can loop through the values in the table which aren’t encrypted and then encrypt them, updating the rows which aren’t already encrypted.

Look for more blog posts in this series on how to encrypt data which already exists within your applications database.

Denny