Open IT Forum: How do you prepare your data center for severe weather?

6345 pts.
Tags:
AS/400
Business Intelligence
CIO
CRM
Data Center management
Database
DataCenter
Development
Disaster Recovery
Disaster recovery planning
Linux
Lotus Domino
Microsoft Exchange
Microsoft Windows
Mobile
Networking
Open IT Forum
Oracle
SAP
Security
SQL Server
Storage
Virtualization
VoIP
Windows Server
Every season has its share of weather threats, and we want to know how you prepare for them. The east coast is gearing up for Hurricane Irene's arrival, and we were wondering what steps you were taking (if any) to make sure your data center doesn't see much (or any) downtime. What unique conditions do you have to prepare for on an annual basis? Best answer gets 100 Knowledge Points! Check out our Open IT Forum on preparing for snow emergencies.

Answer Wiki

Thanks. We'll let you know when a new response is added.

The main problem in our area are ice storms, freezing rain, severe thunderstorms and tornados. Most of these result in power outages which is dealt with by UPS and backup generators. Now our company has experienced growth over the past decade to the point where we are now beginning to implement 2 co-locations for our datacenters with failover in case of disaster. We have always had a disaster recovery plan which we test twice a year by recovering our environment at a remote site, but having the co-locations with replication and failover will reduce our recovery time considerably. The best advice for having your datacenter prepared for severe weather is following the Boy Scout motto and always be prepared!

Discuss This Question: 12  Replies

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
  • Apolsky
    either way
    0 pointsBadges:
    report
  • HMSSL2K
    Our data center is located in central Texas, and we have bank branches throughout the state. At the data center we test our generator every Saturday morning and have a drill once a month for Hurricane, Tornado, bomb scare, and fire. Our branch staff, managers, district managers, ..... are trained in Hurricane, Tornado, bomb scare and fire preparedness, all locations have battery weather radios, flashlights and top of the line first aid kits. All branches’ and the data canter have staff that is trained in First Aid and CPR. Our data center also has a contract for a diesel tanker to be on site if needed. For our banks we have a contract with a vendor that provides a mobile bank, within 24 hours, for each site that is damage or destroyed. We have DR tests every quarter.
    3,175 pointsBadges:
    report
  • MelanieYarbrough
    Wow, that's really thorough, HMSSL2K. How often has your DC gone down because of severe weather? Was this implemented as a reaction to an outage or as preventive? Just curious as to the priority this kind of preparation takes. Thanks for participating! I've added your points. Best, Melanie
    6,345 pointsBadges:
    report
  • Apolsky
    banner check rd3 test 3
    0 pointsBadges:
    report
  • Abushahd
    Simply,,, we have a Sophisticated and Reliable DRC ( Data Recovery Center ) which is identical to the main Data Center . Moreover, A dedicated Section with dedicated IT professional who work only to make sure the DRC is Identical to the main data center . However, tests are being held peridically where the Network is relayed to the DRC and users start working on the DRC to make sure every thing is fine and the data are up to date ....
    4,050 pointsBadges:
    report
  • Technochic
    We are in the path for hurricanes in Florida so we have to be prepared once the season starts. We have two data centers to start with the most important servers being replicated or set up as clusters in case one data center gets taken out. We have a team of support personelle set up before hand and the members sign forms guaranteeing they are prepared to come in should a hurricane hit be imminent. I weathered three hurricanes on site one year! We have back up generators set up and they are routinely tested to make sure they are in working order. We have found multiple issues during testing that we resolved before an emergency occured. Our data center is prepared to provide comfortable conditions to the emergency support team with air beds, refrigerators, ample water, first aid etc. UPS units are in place to provide power if needed. DR testing is required for tier 1 and 2 systems before they can go live and tapes of backups are stored off site of course. Either data center can function as primary if needed.
    57,010 pointsBadges:
    report
  • Meandyou
    To prepare for bad weather, we close the windows. :-)
    5,220 pointsBadges:
    report
  • Pjb0222
    Make sure the other two data centers don't have maintenence planned during the bad weather window! We also have an offsite recovery facility in case bad things happen to all three. The off site is exercised annually. Data centers all do off site backups, have hardened buildings, battery backups and generators. Some even have power from separate power line distribution loops and data circuits from separate local communications loops.
    3,310 pointsBadges:
    report
  • MelanieYarbrough
    [...] Open IT Forum: How do you prepare your data center for severe weather? [...]
    0 pointsBadges:
    report
  • James Murray
    I think there are a lot of good traditional responses. A strong disaster recovery plan documents what to do if a file is lost, server is lost or site is lost. Often the idea of redunancy is meshed with disaster recovery, but there is a difference. Redundancy is about keeping the system despite a failure while disaster recovery is about what happens after a failure. A RAID system is redundant so even if a drive fails, the system still functions. During a power failure a UPS system can keep the systems running until the generator or the power kicks back on. If a system is lost, a disaster recovery plan kicks in. If a system is down a redundant system takes over with little or no loss of downtime. In the Seattle area, we worry about Volcanoes as well as the normal storms and other natural disasters. What we've done is chosen to store core business systems in locations that are less likely to be unstable. For example the Easter half of the state is protected from winter storms by a mountain range. So typically mirrored datacenters are stored on the eastern side of the state outside the range of potential volcanic erruptions. Remote administration is also a big part of the solution. Seattle doesn't have snow often so when there is snow, the roads are a disaster. Instead, we setup secure remote connections so that system administration and other types of work access can be preformed from home. Because power is such an issue at the component and site level, NOCs are being created that are not affected by power surges, brown outs or even lightening strikes to the building. Monitoring tracks systems that are near failure and directs technicians or redirects the load to other sites. The cloud is another new way to protect against failures. Utlilizing Microsoft or Googles hosted systems means that there are plenty of mirrored and redundant sites around the world if a single site is lost. It's interesting to note that fortune 500 companies and governement agencies are moving thier systems into these cloud based infrastructures to not only reduce costs but reduce the risks associated with storms and natural disasters. To prepare for complete system failures: Backups are checked regularly regular UPS testing Regular testing of the generators and fuel sources (water in the fuel is the worst during a rain storm)
    1,795 pointsBadges:
    report
  • coralbasie
    Backup is key
    0 pointsBadges:
    report
  • coralbasie
    one word: backup
    0 pointsBadges:
    report

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Thanks! We'll email you when relevant content is added and updated.

Following