Now that it’s too late, what DR prep can we do today?

While preparing for a major disaster is what should be done ahead of time, that’s a little late for some people on the east coast of the US today.  So if you find yourself (or your company or your servers) on the east coast, what are some things you can do today to help keep things running if things go very, very bad.

On the SQL Server side of things…

• Verify your most recent full backup by restoring it to another server (both user and system databases)
• Make sure that your backups are copied to another servers or USB Drive
• Get a copy of the backups out of the facility (a USB drive is a great way to do this, the network will work if needed)
• Generate some scripts to ensure that you can restore the databases to another machine when ready
• A way to get the transaction log backups off site, maybe FTP’ed to a personal website (after encrypting the files)

On the Hyper-V / VMware side of things…

• Do you have backups of critical VMs?  If not take them now.
• Get a copy of the backups out of the facility (a USB drive is a great way to do this, the network will work if needed)
• Don’t forget to do a system state backup of Active Directory

If there is a disaster, the things that you’ll need at the new data center will include…

• Servers (get your order in as soon as you know that your data center doesn’t exist any more, you won’t be the only one ordering servers)
• Switches, Routers, etc.
• Lots of USB Hard Drives (The SAN won’t be available, and it’ll take weeks or months to get a new one)
• Zip ties/cable ties
• Network cables
• All the backups that you took from above
• Installation media for all your OSes and databases servers
• As lots of things

As you are getting the installations all setup, some things to keep in mind…

• You’ll want to enable Instant File Initialization
• You’ll want to restore system databases, then user databases
• Restore AD from the system state backup, otherwise all the Windows authentication SIDs will be useless within SQL Server

If your data center looses power and it’ll be out for a while, and you don’t have a big expensive generator to run the data center know where you can get a generator.  A few small ones from Lowes, Home Depot, etc. will work as well for keeping the most critical systems online.  Get some large box fans to keep the air in the data center moving.  If you are using small generators, use a different one for the fans as they will have very dirty power, which could damage the power supplies of the servers.  Plastic sheeting can be used to direct air in or out of the data center so that you get cool air coming in from outside and so you can direct the hot air back out (ask me some time about turning a 6 story building into a giant chimney for a data center).

Once we are in a disaster situation we are no longer worried about peak performance, we are worried about keeping the business running for our customers.  Don’t worry about redundant power supplies, or passive nodes, reporting servers, etc.  Just worry about getting the core systems online so that websites work, and so that customers can login to your applications.  If your application supports it be sure to put up a message telling your customers that performance won’t be very good because of the weather.  Just because you know that the weather is a mess doesn’t mean that your customers are aware that there’s a problem.  Most people are pretty understanding and if you tell them that the weather is causing performance problems they will be fine with that.  This will save you a lot of phone calls and a lot of headache.

No matter what, when getting things setup, rebuilt, brought back online, etc. do it safely.  Don’t get hurt and risk hurting yourself or others to get things back up and running.  It just isn’t worth it.

Denny

I Hope Everyone on the East Coast has a DR Plan

With hurricane Sandy dropping into the East Coast of the US this week this is a perfect time to think about DR plans.  DR planning isn’t something that people should take personally when it is brought up.  There are some IT professionals who consider DR planning to be a personal insult to their ability to setup and configure systems, and there are some developers who consider DR planning to be an insult to their programming abilities.

This tweets which Karen Lopez (blog | @datachick) shared shows the exact problem which she has run into when working with one (or more) of her clients.

Setting up DR (or backups in this case) has nothing to do with insulting the IT staff, or that the programmers don’t know what they are doing.  As IT workers our job is to hope for the best, but plan for the worst.  In the case of this week the east coast of the US is being hit with about the worst case that they can get, a full blown hurricane going all the way up the coast.

I don’t care how good you are at racking servers, installing Windows, writing software, etc. if the power at your data center goes out for a week, and they can’t get fuel to the data center for a week (depending on the number of trees which are down between the highway and the data center, this is a real possibility) the systems will be down and you won’t have planned correctly for the worst.

If you think about this from home perspective instead of the work perspective, when a disaster strikes you don’t want to have to rush to the store to try to find bread and other food items to keep yourself and your family fed during the emergency.  If you live somewhere that has regular natural disasters (which is pretty much everywhere at this point) you hopefully have canned food, bottled water, flashlights, etc. at home so that you can ride out this sort of disaster for a few days at the least without running out of food and water.  Why shouldn’t you plan accordingly at the office as well.

This sort of planning isn’t something that can be done last minute, because you can’t always see the disaster coming so you don’t always have time to plan (or shop) right before (or as) the disaster happens.

With some forethought and proper planning any business can ride out any disaster.  But it requires planning ahead of time and the dedication of the company and the employees to properly setup and test the DR solution.  DR projects can be big scary projects if you don’t have someone on staff who has experience with these sorts of things.  But that’s OK, that’s what we have consultants for who specialize in these sorts of projects.  Not every company needs to keep staff on hand that can plan out DR plans, but you should bring someone in who knows how to plan and execute these sorts of projects successfully.  While the consultant may cost a few dollars an hour, it’ll be much less than a failed DR project, and a whole lot less than a failed DR failover.

Denny

(Thanks to Karen for letting me use her Tweets in this post, and for Thomas LaRock (blog | @sqlrockstar) for letting me steal the picture of his kids.)

Recommended reading from mrdenny for October 26, 2012

This week I’ve found some great things for you to read. These are a few of my favorites that I’ve found this week.

• Using filetable with full-text? Lose the redundant index
• Cloud Computing: Start with the problem
• How To Support Multiple SQL Server Package Configurations in SSIS
• Tracking Extended Events for a SQL Agent Job
• Most common latch classes and what they mean

This weeks SQL Server person to follow on Twitter is: SQLSoldier also known as Robert L Davis

Hopefully you find these articles as useful as I did.

Denny

But We Are Hosted In The Cloud, We Don’t Need To Think About Architecture!

So you have moved your infrastructure into the cloud and everything is growing and going wonderfully.  Now you’ve got some consultant or job application telling you that you need to think about HA, DR and overall platform architecture.  How should you tell them that they are wrong?

You shouldn’t.

Just because you are hosted in the cloud (and it doesn’t matter which cloud) you still need to plan for system failures and major outages.  In case you missed it today there was a large portion of the Internet which wasn’t available today because of am Amazon AWS outage which happened.  And this isn’t the first time that something like this has happened.

When moving applications and services up to a managed cloud like EC2, AWS, Rackspace, Azure, etc. you still need to take the time to plan out how the application or service is going to run in the event that the primary cloud is offline, or that there’s some sort of network issue, etc.

Denny

Recommended reading from mrdenny for October 19, 2012

This week I’ve found some great things for you to read. These are a few of my favorites that I’ve found this week.

• SQL Server Data Tools in Visual Studio 2012–Safe Refactoring
• Got A Pal in Kalamazoo (#SQLSat160)
• We did it! Office 2013 RTMs today!
• Set the Command line properties (i.e. Quick Edit) with PowerShell
• Migration Guide: Migrating to AlwaysOn Availability Groups from Prior Deployments Combining Database Mirror…

This weeks SQL Server person to follow on Twitter is: CapTechListens also known as CapTech Ventures

Hopefully you find these articles as useful as I did.

Denny

SQL PASS 2012 1st Timers Webcast Recording

So yesterday was my SQL PASS 2012 1st timers webcast and it went great.  Lots of great questions from the audience which I think we even got all of answered.  If you missed the session and wanted to catch the recording it’s posted up on the live meeting site.  At the moment I can’t get the recording posted anywhere else, but I’m working on it.  For now feel free to use the live meeting version.  When you go to the live meeting site it will ask you for a Recording Key, just leave this blank.

Do note that you’ll want to use the low-res version.  The high-res version doesn’t work.

The recording will be available there until October 2013, but you’ll probably want to watch the video before the summit.

See you at the summit,

Denny

SQL Karaoke at the 2012 SQL PASS Summit has a sponsor

I’m pleased to be able to announce that SQL Karaoke night (Wednesday night at Bush Garden in the International District) has a sponsor again this year.  The fine people over at Idera Software have agreed to sponsor the evenings activities this year.  The sponsorship for this year will be very similar to last years.  Anyone and everyone is welcome to attend the insane SQL Karaoke night at Bush Garden (614 Maynard Ave. South Seattle, WA).

However in order to get the free drinks you’ll need to go and see the fine folks at the Idera booth during the day while the exhibit hall is open.  At the Idera booth you’ll be able to pick up a wristband which is good for one night, and one night only.  After that the only thing you need to do is have some dinner and at about 9:30pm Wednesday night head over to Bush Garden.  Wristbands are given out first come, first served using what ever criteria the fine folks at the Idera booth come up with.

Bush Garden is about a 30 minute walk (I’d recommend walking in a group if you want to walk) from the convention center.  From the convention center walk down Pike (towards the water) and turn right on 6th.  Walk for about a mile then turn right on Jackson, then left on Maynard.  It’s about 2 1/2 blocks down on the left.

Unfortunately Bush Garden isn’t in the best of neighborhoods so do use caution.

Personally I recommend a taxi as I’m lazy and it’s a long walk.  A cab should cost you about $5 or so to get there from the convention center and/or hotel.

If you are looking for a great way to unwind and watch some people do some great singing (and some others make total fools of themselves) this is THE party to go to.  It’s also a great place to meet some of the MVPs as well as some of the folks from the product group.  If you aren’t sure what you are getting in for by attending, don’t forget to check out the website.

The wristbands to come with a budget attached, so drinks will be first come first served.  As always please don’t go overboard with drinking.

See you at Bush Garden,

Denny

SQL PASS 2012 First Timers Webcast Is TODAY!!! #sqlpass

Don’t forget that my SQL PASS Summit First Timers Webcast is going to be today at 1pm Pacific / 4pm Eastern (find your local time).  This is the second annual SQL PASS Summit First Timers Webcast as the one I did last year was so well received.

During this webcast I’ll be telling you a lot of what you’ll need to know about the Seattle Convention Center, how to get around town, how to get to the hotels from the airport, where to find places for dinner, how to find your way around the convention center, and much more.

If you’ve been to the PASS Summit in Seattle before I’d recommend watching anyway as there will be some new information for you.  Most importantly where breakfast and lunch will be served as the lunch hall is moving.

You do need to fill out my annoying registration form to get the meeting info.  The form is painless I promise.

I’ll see you this afternoon.

Denny

There is still time to sign up for SQL PASS PreCons

This year the PASS Summit has 14 fantastic PreCons which are available for attendees to sign up for at an additional cost of just $395 per precon.  These precons are full day sessions with some fantastic presenters including myself, Louis Davisson, Rod Colledge, Jessica Moss, Brian Knight, Allen White, Itzik Ben-Gan, Bob Ward, Allan Hirt and Grant Fritchey as well as a several others.

These sessions are a fantastic deal as you get a full day session, hand out, lunch plus lots of time to chat with the presenter and attendees before the sessions, during the breaks as well as after the session.

Can’t decide which session you want to attend?  No problem there are two days of PreCons so you’ll be able to turn the 3 day PASS Summit into a 5 day conference for just a few hundred dollars more.  If you are heading out to the PASS Summit I’d really encourage you to sign up for one of the pre-cons.  I’d love to have you in my pre-con called “SQL Server 2012 in a Highly Available World” where we are going to walk through all of the high availability options which are available in SQL Server 2012 (as well as in the older versions as well), learn how to get them setup and most importantly figure out when we should be using each one.

If you are already signed up you can modify your registration or contact the registration contact listed on the registration page and she’ll be happy to add a pre-con onto your registration.

I look forward to seeing you at the summit, and at my precon,

Denny

Recommended reading from mrdenny for October 12, 2012

This week I’ve found some great things for you to read. These are a few of my favorites that I’ve found this week.

• Error: Executing DDL statment thru SSMS by a Windows Group user
• Rant: Applying Common Sense in IT
• Support anti-harassment policy adoption? Help improve our wiki resources.
• What have I done in 2 years on the PASS Board of Directors
• Calling home, receiving calls and smartphone data from the US

This weeks SQL Server person to follow on Twitter is: SQLEinstein also known as Adam

Hopefully you find these articles as useful as I did.

Denny