I’ve moved lots of companies into data centers before, and each time I’ve done so it has included a site visit during to RFP process to check on things like cooling, generator, site configuration, etc.  During these site visits I stick my head through every door that the sales guy and the engineer who are doing the tour will open.  If they won’t open a door for me they better have a damn good reason to not show me what’s in there.

If I went to do a site visit at a facility that was located just at sea level just a few blocks from the ocean, they’d better have some pretty impressive plans for how they are going to handle flooding.  If the answer was “we’ve never had a problem with flooding” or something similar they’d be off my list as they haven’t done their due diligence to insure that the site will be online during the worst emergencies possible.

Now before you start telling me that I’ve got no idea what I’m talking about, and that data centers in the North East have different problems from data centers in the South West.  I actually do as I’ve moved companies into data centers on both coasts.  Most recently I moved the data center for Phreesia from a managed service facility in Texas to a colo in New Jersey.  As a part of this move we looked at a number of facilities in the New York / New Jersey area.  Many of the New York City data centers were eliminated due to cost, or being just to close to the water as we didn’t want to deal with situations like this exact one.

The data center we settled on is in New Jersey about a 30-40 minute drive from the Phreesia office (once you get out of New York City).  While the data center is near a river, the river is a little over a mile away.  The data center itself is on a slope with a  cinder block wall on the outer edge which will divert water away in the event of a river overflow (it also protects the facility from someone driving a car or truck into the facility).  The generators and fuel pumps are all mounted on raised feet (not very tall, but tall enough) so that they keep running in the event of a flood.  The cables from the generators to the building have all been buried under the ground so that tree branches which are torn loose during the hurricane can’t cut those cables.

Our diligence in selecting a data center paid off.  While the folks mentioned in that article were dragging buckets of fuel up 17 floors worth of stairs the team at Phreesia just sat back and rode out the storm with their families.  The sites didn’t go down and the team didn’t have to rush into a very hazardous situation. The team was able to focus on their families and neighbors without having to worry about the systems.  Those of us that aren’t in the New York area monitored the systems remotely and founds that everything was running perfectly normally just on generator power instead of city power.

This entire event just shows that when doing data center evaluations the worst possible case situation needs to be planned for and expected.  If they aren’t they are going to come back and bite you.  Especially in todays world of storms which are ever increasing in destructive power.

If you are in a facility which has risks such as fuel pumps which are below sea level (they are in the basement and the road is at sea level) then a second data center becomes very important very quickly.  This became very clear during this hurricane when some very large important websites went offline because they didn’t have that DR facility that they could fail over to, unlike sites like StackOverflow (and ServerFault and the Stack Exchange network).

If you are at risk now is an excellent time to sit down with management and go over all the “what ifs” of your current hosting solution and think about the cost of a DR site along with the cost of not having one.

Denny

What does all this have to do with companies DR plans?  Well in the DFW area there is a little hosting company called RackSpace.  RackSpace hosts a large percentage of their customers in the data center in the area which they actually call DFW (granted many companies in the Dallas area refer to that office DFW).  In the case of RackSpace however the facility really is close to DFW, very close.  In fact it’s at the end of Runway 13R/31L at the North/West end of the runway.  The red mark at the top left is the RackSpace CoLo facility, the road at the bottom is runway 13R/31L.

So why I am picking on RackSpace right now? Because if you look at this map you’ll see several tornadoes which touched down not all that far away from RackSpace, just a few miles away in fact.  This was a very close call.

If those tornadoes had touched down just a few miles to the west there would be a lot of companies would be in a really bad state at the moment as they try and figure out just how much data had been lost between the last tapes to be shipped from RackSpace to offsite storage and when the place was torn apart.  Then there’s the problem of how long it’ll take RackSpace to get new servers delivered and racked in another data center (as it’ll probably take a while to get this one dried out and rebuilt).

Best case is that these companies would be looking at several days of downtime, worst case is weeks.  The reality of the situation is that most of the smaller companies would be totally hosed as odds are that RackSpace would be focused on getting their largest clients online first, as the 30 largest clients probably bring in more revenue than the rest combined (I’ve done work for several of RackSpace’s larger clients so I know how much they are paying).  Given that the major computer companies can only produce so many servers at a time, and RackSpace would pretty much need all of them for a couple of months as RackSpace would need probably thousands of servers and storage arrays to be delivered in order to get everything back up and running.

What would make this even worse is that companies that tried to move to another hosting provider to try to get online faster probably wouldn’t be able to.  First they’d need to get their data from tape at RackSpace which would be a problem unto it self as there wouldn’t be anywhere for RackSpace to restore the data.  Secondly the new hosting providers may not have been able to get new hardware delivered as RackSpace would be taking up all the production capabilities.

Now this hell could all be avoided by properly planning for this sort of disaster hitting the RackSpace hosting facility.  RackSpace has several other data centers in the states that you could easily enough setup some DR machines at another facility and setup data replication between the facilities so that if one facility was taken offline you would be able to keep running at the second site.

But again this all requires planning this in advance.  If you are a RackSpace customer I’d recommend talking to your sales team about getting a DR solution up and running within another of the RackSpace facilities.

If you need assistance with these conversations feel free to engage me and we can make sure that your systems are prepared for the next disaster that strikes near (or on top of) your data center.

Denny

1. If there are triggers on your tables, replication doesn’t have a way to ensure that the triggers will be there on the remote site.

2. If you need to add tables, procedures, views, etc. you have to reinitialize the subscription to add the new articles to the subscriber.

3. The failback story is pretty much a mess.  Assuming that you do have to fail over to your DR server failing back isn’t exactly the easiest thing to do.  Basically you have to take another outage while you move the database back.  That or you have to resetup replication in the other direction.

Needless to say that these are some pretty good reasons to not use SQL Server Replication to get data to your DR site.  Especially as there are so many better options such as Database Mirroring, Log Shipping, storage replication, third party storage replication and soon enough AlwaysOn Availability Groups.

If you are using SQL Server Replication to replicate data from your production site to your DR site I urge you to look at the other options which are available to you and you should strongly consider moving to one of the other technology options.

Denny

Rice.

Yep, that’s right good old fashioned white dinner rice, uncooked of course.  The reason that rice works is because rice makes an amazing desiccant.  For those who missed the episode of Numbers where they talked about desiccant’s, a desiccant is something that draws the moisture out of something else.  In the cast of the TV show it was drawing the water out of a dead body, in this case out of my dead laptop (they didn’t use rice in the show).  You know those little packets that come in just about everything that you buy, that say “Don’t eat” on them?  That’s a desiccant.

In the case of the laptop, I quickly unplugged it and pulled out the battery.  Then got a box a bit bitter than the laptop, and dumped a bunch of rice into the box, dropped in the laptop, which I then covered with more rice.  Then the battery and an HP TouchPad that got drenched as well.  Cover everything with a healthy layer of rice and let sit for several days.  The nice thing about this technique is that there’s no head, no chemicals, no cloth or paper towels to touch anything in the laptop.  Just the rice absorbing up any water in the laptop.

When doing this with a car, you can’t put the car in a box and fill it, so a slightly different approach is needed.  In my case we had a car door opened just a crack which let in rain water (yes it really does rain in Southern California).  Using a vacuum and towels we could only get so much water out, but we didn’t want to worry about the floor of the car rusting out.  So we pored about 10 pounds of white rice into the floor board of the car a waited about 6-8 hours for the rice to absorb all the water.  When we vacuumed out the rice everything underneath was nice and dry.

Back to the laptop.  I left the laptop in the rice for about a week in this case, and thankfully the laptop came back in perfect working order.

To pull this off I used about 15-20 pounds of rice total.  When I dumped the cup Kris went to Costco and picked up a 50 pound bag of rice and I used a large bowl to get it into the box.

Hopefully you never have to use this trick, but if you do there it is.

Denny

The first issue is that when the cloud provider is a US company (which most of them are at the moment) then your data falls under US law, as well as any laws which exist if the data is being stored in a non-US data center.  So basically if you are a non-US company and you use these services to host your data, the US government can issue a National Security letter to the cloud provider and get a copy of all the data that is hosted by the US company.  (This effects those of us who live in the US, but there’s nothing we can do about it as we are already here.)

Additionally there’s some major gray area in US law at the moment.  This has to do with the abandoned property laws.  These laws were put together in the 90s or so when leaving data on a server for long periods of time was abnormal.  These laws basically say that if data has been hosted on a 3rd party server (the cloud in this case) for over 90 days this data is abandoned and the US Government can request access to the data, without the need for a search warrant.  Seeing as the whole point of using the cloud is to host your company data on a third party server these laws become a pretty big stumbling block.

Some countries have laws about where the data for their citizens can be stored.  Check with your national government to see if it is even legal for you to store data in another country.  Don’t forget that when picking your cloud provider, even if the data center you pick to host your data is within your country, there wouldn’t be anything stopping your cloud provider from replicating your data to another country for backup or high availability / disaster recovery.

There are lots of other laws to be informed about before moving your data into the cloud.  Be sure to do your research before moving your application to the cloud.

Denny