Did you enjoy yesterday’s spine-tingler? Then you’ll enjoy today’s terrifying tale of data center despair.
“Total shutdown: A cautionary tale”
Sander van Vugt
Don managed a small data center for a school. He was so confident with the data storage on his servers, that he always urged students to store their data on the network drives alone. “Data that is stored there will be backed up by the IT department, so you’ll always have access to the data!” he told wary students.
A few days before the day that senior theses were due, the last Friday of October, when the IT department’s phone started ringing. Mark, the help desk guy, answered it.
“No access to the data volume,” he called over to Don.
Mark hung up the phone, and it immediately rang again.
“One more,” said Mark.
“Tell them we’re working on it,” Don replied.
After half an hour the volume still wasn’t available. With a growing sense of dread, Don decided to reboot the server, backed by Alex, the external specialist. Alex couldn’t find a way to make the volume accessible again either.
When they walked out of the office toward the data center, they met a group of very nervous senior students.
“No comment,” Don said nervously as he pushed past.
Once in the data center, Alex and Don found the server and rebooted it. While booting it gave at least 50 different error messages when it completely crashed and hung.
“What do we do now?” said Don, starting to sweat. The specialist tried to reassure him. “Don’t worry I’ve got it all under control.”
Beyond the sealed, secured doors of the data center, the small crowd had become a shambling, moaning mass of sleep-deprived students.
“You’ve got them under control too, right?” Don asked.
Very late that evening Don and Alex cautiously ventured from the data center to take the train home. The crowd, not to be deterred, shuffled behind, finally caught up to the two technicians at the train station.
Alex’s train arrived before Don’s, so they separated at the railway station. Just as Don’s train arrived, the students shuffled into view. He boarded the train, hoping the doors would close quickly. Don called out to the dead-eyed students. “Wait until tomorrow morning when manual recovery is finished.”
Don and Alex had tried everything, including pulling from the backups that the IT department was supposed to make every day. Unfortunately, they hadn’t verified the tapes so the backups were empty. The manual restoration had not been successful.
Alone in the train car the next morning, Don wondered what he would tell the students. Once the train pulled into the station, he knew it was too late. The shambling horde of students came forward as the train doors opened.
When Don didn’t show up, Alex called him several times but all he got was voicemail. Don, it seems, hadn’t been seen since he got off his train that morning. The only thing left of Don was a broken backup tape at the railway station and his school ID badge. He was never seen again.