The Virtualization Room

A SearchServerVirtualization.com and SearchVMware.com blog

» VIEW ALL POSTS Jul 29 2008   11:09AM GMT

Recovering servers, virtual machines after power failure



Posted by: Eric Siebert
Tags:
Eric Siebert
High availability and virtualization
Virtual machine
Virtualization
Virtualization management
Virtualization strategies

An event like a complete data center power failure is something you never want to experience. Having recently gone through one I thought I would share some lessons learned from it.

This particular data center had a full UPS (uninterruptible power supply) system and backup diesel generator when a routine battery maintenance performed on the UPS shorted some circuits causing power loss to the entire data center. This event made me realize that a little preparedness can go along way in getting servers and virtual machines (VMs) back online after a power failure.

First and foremost, the DNS (Domain Name System) is probably the most important service in your data center. Most servers and workstations use DNS names instead of IP addresses to communicate with each other. Without DNS, servers can’t get to anything by hostname and will effectively be isolated from each other. Most administrators are used to using DNS names, so when DNS is not available they usually do not know the IP addresses of the server and subsequently can’t connect to them. So it is a good idea to have a hard copy of all your servers and their IP addresses somewhere in your data center for you to reference when DNS is not available.

Virtual servers can be even more problematic. If you have all your DNS servers virtualized which cannot be started because of network or shared storage issues, you can run into problems starting other servers and services that rely on DNS. Consider having at least one physical DNS server or having one or two DNS servers running on local storage instead of shared storage.

Another helpful insight: Make sure you know command line procedures for administration on your host servers. You may not be able to connect to your host via a graphic user interface (GUI) until certain systems are up so the command line can be your only way to check the host server health and perform VM operations. Again, it helps to have paper documentation of the host command line utilities and their syntaxes.

Finally you want to make sure you start your servers back up in the proper order due to dependencies that certain servers and applications have. Obviously, with the network unavailable, not much is going to function properly. The storage-area network (SAN) is also critical for your host servers that utilize shared storage for VMs. Windows servers also take a very long time to boot if a DNS server and domain controller are not available when they are starting.

Below is a general order for restarting your servers and applications.

  1. Network
  2. SAN
  3. DNS servers
  4. DHCP servers
  5. Database servers
  6. Application/Web servers

The boy scout motto ‘be prepared’ holds true. A little preparation and planning can go along way to ensuring a smoother recovery.

2  Comments on this Post

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
  • Stc
    First off, I'm sorry about the power failure. Working with a secured redundant SAN can help protect a variety of data so that it can be recovered later even if the whole system crashes.
    0 pointsBadges:
    report
  • Eric Siebert
    STC - I was under the impression that a data center power failure would affect the redundant SAN also. Eric - Been there. More than once. Thinking through the dependency tree, especially the dependencies that cross disciplines is one of those things that is a pain to do, but tends to be revealing. I've found that a DR exercise, even if informal, helps sort through things like that. --Mike
    0 pointsBadges:
    report

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: