Comments on: Open IT Forum: How do you prepare your data center for severe weather?

By: coralbasie

coralbasie — Wed, 11 Apr 2012 14:10:54 +0000

one word: backup

By: coralbasie

coralbasie — Fri, 14 Oct 2011 14:06:38 +0000

Backup is key

By: ekardris

ekardris — Wed, 21 Sep 2011 17:22:16 +0000

I think there are a lot of good traditional responses. A strong disaster recovery plan documents what to do if a file is lost, server is lost or site is lost. Often the idea of redunancy is meshed with disaster recovery, but there is a difference. Redundancy is about keeping the system despite a failure while disaster recovery is about what happens after a failure. A RAID system is redundant so even if a drive fails, the system still functions. During a power failure a UPS system can keep the systems running until the generator or the power kicks back on.

If a system is lost, a disaster recovery plan kicks in. If a system is down a redundant system takes over with little or no loss of downtime.

In the Seattle area, we worry about Volcanoes as well as the normal storms and other natural disasters. What we’ve done is chosen to store core business systems in locations that are less likely to be unstable. For example the Easter half of the state is protected from winter storms by a mountain range. So typically mirrored datacenters are stored on the eastern side of the state outside the range of potential volcanic erruptions.

Remote administration is also a big part of the solution. Seattle doesn’t have snow often so when there is snow, the roads are a disaster. Instead, we setup secure remote connections so that system administration and other types of work access can be preformed from home.

Because power is such an issue at the component and site level, NOCs are being created that are not affected by power surges, brown outs or even lightening strikes to the building.

Monitoring tracks systems that are near failure and directs technicians or redirects the load to other sites.

The cloud is another new way to protect against failures. Utlilizing Microsoft or Googles hosted systems means that there are plenty of mirrored and redundant sites around the world if a single site is lost. It’s interesting to note that fortune 500 companies and governement agencies are moving thier systems into these cloud based infrastructures to not only reduce costs but reduce the risks associated with storms and natural disasters.

To prepare for complete system failures:
Backups are checked regularly
regular UPS testing
Regular testing of the generators and fuel sources (water in the fuel is the worst during a rain storm)

By: Featured Member: Lord Voldemort - ITKE Community Blog

Featured Member: Lord Voldemort - ITKE Community Blog — Tue, 20 Sep 2011 06:25:03 +0000

[...] Open IT Forum: How do you prepare your data center for severe weather? [...]

By: pjb0222

pjb0222 — Thu, 15 Sep 2011 18:32:39 +0000

Make sure the other two data centers don’t have maintenence planned during the bad weather window! We also have an offsite recovery facility in case bad things happen to all three. The off site is exercised annually.

Data centers all do off site backups, have hardened buildings, battery backups and generators. Some even have power from separate power line distribution loops and data circuits from separate local communications loops.

By: meandyou

meandyou — Wed, 14 Sep 2011 14:19:59 +0000

To prepare for bad weather, we close the windows.

By: technochic

technochic — Mon, 12 Sep 2011 21:35:34 +0000

We are in the path for hurricanes in Florida so we have to be prepared once the season starts. We have two data centers to start with the most important servers being replicated or set up as clusters in case one data center gets taken out. We have a team of support personelle set up before hand and the members sign forms guaranteeing they are prepared to come in should a hurricane hit be imminent. I weathered three hurricanes on site one year! We have back up generators set up and they are routinely tested to make sure they are in working order. We have found multiple issues during testing that we resolved before an emergency occured. Our data center is prepared to provide comfortable conditions to the emergency support team with air beds, refrigerators, ample water, first aid etc. UPS units are in place to provide power if needed. DR testing is required for tier 1 and 2 systems before they can go live and tapes of backups are stored off site of course. Either data center can function as primary if needed.

By: abushahd

abushahd — Mon, 12 Sep 2011 10:37:58 +0000

Simply,,, we have a Sophisticated and Reliable DRC ( Data Recovery Center ) which is identical to the main Data Center . Moreover, A dedicated Section with dedicated IT professional who work only to make sure the DRC is Identical to the main data center . However, tests are being held peridically where the Network is relayed to the DRC and users start working on the DRC to make sure every thing is fine and the data are up to date ….

By: apolsky

apolsky — Fri, 09 Sep 2011 21:43:15 +0000

banner check rd3 test 3

By: melanieyarbrough

melanieyarbrough — Fri, 02 Sep 2011 17:21:12 +0000

Wow, that’s really thorough, HMSSL2K. How often has your DC gone down because of severe weather? Was this implemented as a reaction to an outage or as preventive? Just curious as to the priority this kind of preparation takes.

Thanks for participating! I’ve added your points.

Best,
Melanie