Data Center Contest: Successful treatment of Mayo Clinic kill switch

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Gary submitted this entry:

In 1994, I joined the Mayo Clinic Rochester, MN campus as a senior Systems Administrator. My first day was spent getting the guided tour of the data center facilities at the Clinic. We were touring the main data center. This facility housed the Clinic’s mainframe, and a host of other distributed enterprise systems. The day was getting rather late as we were approaching the end of the tour. The lateness of the day also marked the end of the shift for the Data Center operations staff.

Our tour group was being escorted through the command center on our way out to the main data center operations area. Located along the wall next to the data center entrance was a large red kill switch.

Looking back now, this reminds me of the big red “Easy Button” depicted in popular Staples’ television media adds. This kill switch when pressed engaged the emergency power shutdown for the entire data center. The kill switch was within easy reach from the entrance door. There was no protective encasement or other security feature over the kill switch. Somebody could accidentally trip and hit the switch, which would engage the emergency power shutdown.

This fact must have been on the mind of our guide as well. Our guide strategically positioned himself between the kill switch and the door entrance to prevent such an accident from occurring. After everyone safely assembled together on the production floor, our guide resumed the tour. Our group moved over to opposite end of the production floor to view new StorageTek tape silos. From this vantage point, we had a clear unobstructed line of sight to witness the events that followed.

Observing a data center power shutdown is really something to see. One minute, we were gazing at the swinging robotics arm contained within the StorageTek silo fetching tape cartridges. The next minute the only viewable lighting was coming from the emergency lights. In retrospect, the outage lasted only a few minutes. But during this time there were teams of programmers and other IT support specialists that flooded out onto the production floor. During the power outage, our tour guide remained calm and collected. During these events he kept the group together and out of everyone’s way.

We later learned at the end of our tour of the events that led up to the data center power shutdown. A developer had called the command center to request a hard reboot on one of the servers in the command center. Confused, the operator asked the developer where the server was located. The developer instructed the unfortunate operator to look for a red power switch on the server located next to the wall. The operator saw one red switch. The rest was history.

The next day when I was out on the production floor doing server maintenance I noticed there was a plastic security encasement covering that one particular red switch!!!