Posted by: Michael Morisy
We received a great response to our call to the community for advice on ensuring that a blizzard doesn’t become a perfect storm for your IT department. New member KFaganJr used the question as an opportunity to document everything his department did to prepare, preparing a fantastic guide for others facing a similar problem:
Most of the tasks were a verification that existing systems were fully functional before the storm hit. These included ensuring all off-site backups we run successfully, a live copy of any important documentation was hosted off-site and up to date, temperature sensors and reporting tools were functional and also tweaked to allow more time for action due to the extra travel time needed.
Being a small department we were able to just discuss things such as who can be where and do what if a major problem hits, but in a larger organization I would have documented it.
It’s important to walk through the process of redirecting traffic if a location goes down. We have four main sites that usually funnel all traffic through the main office, if one goes down then that traffic has to be sent elsewhere to keep everything functional. Things to note… how much time do you have to complete the job if need be, is the alternate connection you rely on functioning properly, is touch services needed to make the switch?
Also, for anyone working with end users, any employees with passwords about to expire were sent additional emails to prevent additional work, reminders of phone system features were sent out that would help users work from home seamlessly. emails to remind users that if they have VPN issues speak to IT before the storm hits if possible. Additional laptops were available to lend out for critical personnel.
The most important task in my mind is making sure everything is running smoothly before the storm though, you don’t want to worry about preparing for disaster and neglect the critical server that has been crashing, have an issue with that server when touch services aren’t available and making all other planning to avoid disaster in vein because of an unrelated issue.
Spadasoe suggested that while virtualization could help offload some effects of traffic spikes if the number of remote users faced a sudden jump, “Our fabric is our fabric so we are limited in what resources we can add.” Planning ahead and mapping out worst-case scenarios were the best tips for staying ahead of outages.
MrDenny shared that view, adding that using multiple sites and duplicated data was a worthwhile investment:
Having user data replicated between sites so that when they VPN into another site because the network link at their office is down the users can still access their data, etc. The setup and recurring costs for a little extra bandwidth is minimal compared to the loss of work from one bad 2+ day snow/ice storm.
One commenting wag suggested simply going with the flow, “cutting Internet access to the office? Then blame it on the provider.” While we can’t officially endorse that approach, we can understand the temptation. Any more tips? Leave them in the comments below or, better yet, add them to the community forum!