Did you enjoy yesterday’s spine-tingler? Then you’ll enjoy today’s terrifying tale of data center despair.
“Total shutdown: A cautionary tale”
Sander van Vugt
Don managed a small data center for a school. He was so confident with the data storage on his servers, that he always urged students to store their data on the network drives alone. “Data that is stored there will be backed up by the IT department, so you’ll always have access to the data!” he told wary students.
A few days before the day that senior theses were due, the last Friday of October, when the IT department’s phone started ringing. Mark, the help desk guy, answered it.
“No access to the data volume,” he called over to Don.
Mark hung up the phone, and it immediately rang again.
“One more,” said Mark.
“Tell them we’re working on it,” Don replied.
After half an hour the volume still wasn’t available. With a growing sense of dread, Don decided to reboot the server, backed by Alex, the external specialist. Alex couldn’t find a way to make the volume accessible again either.
When they walked out of the office toward the data center, they met a group of very nervous senior students.
“No comment,” Don said nervously as he pushed past.
Once in the data center, Alex and Don found the server and rebooted it. While booting it gave at least 50 different error messages when it completely crashed and hung.
“What do we do now?” said Don, starting to sweat. The specialist tried to reassure him. “Don’t worry I’ve got it all under control.”
Beyond the sealed, secured doors of the data center, the small crowd had become a shambling, moaning mass of sleep-deprived students.
“You’ve got them under control too, right?” Don asked.
Very late that evening Don and Alex cautiously ventured from the data center to take the train home. The crowd, not to be deterred, shuffled behind, finally caught up to the two technicians at the train station.
Alex’s train arrived before Don’s, so they separated at the railway station. Just as Don’s train arrived, the students shuffled into view. He boarded the train, hoping the doors would close quickly. Don called out to the dead-eyed students. “Wait until tomorrow morning when manual recovery is finished.”
Don and Alex had tried everything, including pulling from the backups that the IT department was supposed to make every day. Unfortunately, they hadn’t verified the tapes so the backups were empty. The manual restoration had not been successful.
Alone in the train car the next morning, Don wondered what he would tell the students. Once the train pulled into the station, he knew it was too late. The shambling horde of students came forward as the train doors opened.
When Don didn’t show up, Alex called him several times but all he got was voicemail. Don, it seems, hadn’t been seen since he got off his train that morning. The only thing left of Don was a broken backup tape at the railway station and his school ID badge. He was never seen again.
Grab a flashlight, head to your nearest lights-out data center, and hunker down for some creepy data center horror stories courtesy of some data center experts. Are these true tales? Stay tuned for tomorrow’s part two.
Data center fright night, part one
“Night of the Living Data Center”
All was quiet and dark – well, it was a lights-out data center after all. The lone admin had been called in. Something had gone wrong in the facility. He went to push the door and – nothing. He remembered he had to use his two-factor secure key, finger print and retina recognition to get inside.
Creeping past the reception desk, a low moan sent shivers down his spine, but it was just the murmurings of the under-paid security guard catching up on his sleep after a 12-hour shift at the local Walmart. Moving silently along, the admin heard “zip, click, zip, click, zip, click” as he walked along. Glancing behind him, he saw the lights slowly clicking off as he moved along. In his befuddled state, it took some time to realize this was the movement-sensitive lights turning on and switching off along the corridor he was in.
At the final interior door, he waited, listening in case something was beyond the door that shouldn’t be. Mind you, as the data center had been built to LEED quality, the amount of insulation in the walls would have deadened the sounds of the demons of hell escaping from the Solaris box he had in the far corner. Screwing up his courage, he opened the door and tip-toed in. There was enormous “CRACK!”
No, sorry, there was an enormous CRAC – it had been there for 20 years, so it shouldn’t have surprised him. Another low moan, far different to the security guard’s sleeping sounds, went past him in a gust of warm, fetid air. He made a mental note to check the raised floor for holes where cooling might be leaking and get the drains fixed.
He made it to the console and logged in. Strange images appeared in front of his eyes – ones he had never seen before. Immediately switching to CLI mode, the easier, more functional and useful graphical user interface was instantly hidden. His fingers flew over the keyboard. Eventually, he calmed down and started typing. The strange incantations he typed brought up steams of glowing data – log files, audit trails, details of pizza delivery companies.
Eventually, tracing the root cause of the issue caused him to take a sharp intake of breath. “How could this happen?” he wondered. Well, it was a waste of time asking anyone else, since they were all in bed at their homes. With a rising sense of panic, he gathered up the tools of his trade – the old Bell telephone modem he might need where he was going, the book on troubleshooting systems and the 1 million candle flashlight. For good luck, he also took a silver thumb drive. Moving away from the console, he muttered some charms under his breath “agility, flexibility, SaaSability” and “I wish we had outsourced this years ago.”
In the corner was a massive black monolith – the admin had always assumed that this was a throwback to the film “2001: A Space Odyssey” and that it was something facilities looked after – nothing for a tech guru like him to worry about. However, the console had provided directions that said this was the source of the trouble. Reading the book, the admin identified the special way to use the runes on the monolith to access secret bits. Finally, he folded down a screen and pushed a switch. The green and black screen flickered, showing a talismanic figure, which changed over into a single word and sent the admin screaming back into the night.
In dripping, blood-red letters, it said “ABEND.”
IT must be ready for fundamental shifts in business and technological paradigms.
Stephen J. Bigelow, Senior Technology Editor
IT professionals are used to change – the reality of change is as old as IT itself. Everyone swaps servers during technology refresh cycles, and new versions of operating systems or key business applications can have IT staff working overtime to get users updated.
Still, change is always a challenge for IT. And while the logistical and technical demands of change will always stress budgets, tax patience, and fray nerves, the IT department soldiers on to solve problems and fight fires and support the business.
But what happens when a technology fundamentally changes the way a business or industry operates?
Just consider an emerging technology like 3D printing where solid objects are created by depositing materials in successive layers according to a digital model. The resulting 3D model can be used as the basis for molds and other core manufacturing processes – even using materials that are correct for the finished product, allowing a manufacturer to actually fabricate finished goods on demand.
The very notion that a company can produce products on-demand flies in the face of traditional business paradigms.
Consider the manufacturing process itself. Traditional manufacturing is based on economies of scale using mass-production of identical items. This involves further practices including logistics, warehousing, sales – the entire business infrastructure which relies on IT resources and support. When small numbers of products can be inexpensively fabricated on demand using an easily manipulated digital model, the business is profoundly affected; and so are the services and support that IT must provide.
There are implications for IT. Designs would proliferate as the number of models multiplies from designers and customers, requiring data storage and security. The concepts of enterprise resource planning (ERP) would change dramatically because the flow of materials into and out of the business would be radically different. Warehousing for work-in-progress and finished goods would be virtually eliminated. Goods could also be built on-site or at remote locations, greatly reducing transportation demands. Our very definition of manufacturing could change. Just imagine an automotive shop that can fabricate certain parts right on the shop floor, or a military unit that can produce key spare parts in the field.
Of course, such fabrication technology is far from perfect; today. But it’s an example of the way that new technologies and their refinement can re-define concepts and practices that are (in the case of mass-production) centuries-old. It’s a wakeup call that IT professionals must look ahead at the changing needs of business and position staff and systems to handle the types of changes that may appear on the horizon – or risk being discarded as obsolete.
Notoriously secretive — at least when it pertains to the innards of its data centers — Google is now offering a peek at the pipes and people that deliver its products — Google Search, Gmail, YouTube and Google Maps — to your computing device.
“Where the Internet lives” features stunning photographs by Connie Zhou of several Google data centers to show the general public the server racks, cooling infrastructure and personnel at the tech giant’s facilities that had been under wraps until now.
You can also take a virtual tour of the Lenoir, N.C., data center through the Street View function in Google Maps. The company’s sense of humor shines through during the tour; if you examine each frame carefully, you’ll spot a few interesting items.
It’s an intriguing behind-the-scenes look at the facilities — rumored to number about three dozen in total — required to handle a bulk of the day-to-day Internet-based workloads that millions of people use on a daily basis.
Cut costs, improve efficiency. Such is the mantra of many a data center manager. While tech giants like Google and Facebook strive to create better, more energy efficient data centers, a small team of researchers from Cornell University and Microsoft have gone back in time 120 years and come up with a way to eliminate another threat to efficiency: cables.
Mathematician Arthur Cayley published a paper in 1889 called Oh the Theory of Groups – mathematical groups, that is – that was full of graphs and equations.
In 2012, those graphs and equations were used to design a wireless data center network running on a 60 GHz wireless band.
According to the paper’s abstract, the benefits — besides eliminating networking cable and switch costs — would include higher bandwidth and fault tolerance and lower latency. Adopting a spoke-and-wheel rack system for servers would facilitate communication using specially built Y-switches would help direct that traffic between racks.
This would mean a complete change in server form factor. The basic parts would remain the same – hard or solid state drive, CPU and RAM – with networking cards replaced with Y-switches. The paper goes on to mention changes to data center routing protocols, MAC layer arbitration and design schematics for the customized Y-switch. The full document is available on Cornell’s Website.
Once servers are cylindrical, it’ll be exciting to see how the buildings surrounding them change to suit. What do you think? More data centers in old missile silos?
That’s right, science has just upped the ante for cool data center locations. Forget Iceland or Oregon, now there’s a proposal to put a supercomputer on the moon.
Ouilang Chang, a graduate student at the University of Southern California, presented what he calls the Lunar Supercomputer Complexto help ease the burden of scientific “big data” processing and network traffic on terrestrial facilities. Chang says the rate of information is starting to exceed the ability of the networks to keep up. Not only that, but Chang feels the lunar site would give scientists the ability to boost the power and effectiveness of telescopes here on Earth.
In spite of the fact that this is such a vast undertaking, Chang wants to see it done in the next 10-15 years. If you consider that there’s already a plan in place to start mining asteroids, this isn’t too huge a leap.
The Lunar Supercomputer Complex would be built on the dark side far side of the moon, which would allow moderate protection against extreme temperatures and access to potential ice under the surface for cooling. The complex would include communication arrays, a data center with supercomputers, data storage, power, cooling, radiation shielding and ultimate nerd-cred for any data center architects lucky enough to draw the straw to work on the project.
The good news is they can probably save money on locks and security personnel. But would this facility ever be built?
Back in January, when Newt Gingrich proposed his moon base, people jumped on it as an “impossible dream” due to lack of available funding. But maybe if the billionaire asteroid miners take notice, the moon data center could come to fruition.
Here’s the scenario: You’re the only tech-savvy guys in a very small town, and you come across mysterious devices labeled “Pluto Switch” sitting in your distribution center. What do you do?
Post about it on the Internet, of course! And that’s exactly what happened, according to an article on Wired.com.
The gentlemen posted images of the hardware — which turned out to be a custom-built network switch — on a networking forumand tried to get it working. Many forum users offered help, wanted clearer pictures or asked to buy the switch, but in the end a little bit of sleuthing uncovered they were meant for Google.
Google builds much of its own hardware, as shown by hardware job postings on the company’s site, and further deduction revealed the town – Shelby, Iowa – was only 30 miles from one of Google’s data centers. One of the men theorized the switches were left at their facility by mistake during a delivery run.
The device was sent back to Google, the men who found it got T-shirts and Google tried to shut down the post with the images. So far, the forum hasn’t removed the post, but it has scrubbed all reference to the original posters’ identities.
Sherlock Holmes would be proud, Internet sleuths.
The last time I bought a video game, I grudgingly went to a brick and mortar store but checked the ratings on Amazon before I went. Buy local, shop global and whatnot. The last time I bought a computer – or parts for one, rather – I bought from NewEgg after scouring reviews for hours.
If you wanted to shop for enterprise software or hardware, there was no business equivalent of, say, an Amazon where you could see what other users had to say about a particular product. And if you wanted to warn other IT pros about a particular blade server that melted in its rack or you just needed to rant about XenServer, you were relegated to various forums scattered around the Web.
But a new site called IT Central Station attempts to fill that void. According to the company, the site will feature social networking, user validation and a bunch of different categories.
The site presents itself as a peer review site for IT with no vendor bias – though they’ll happily accept ads! – and as a thriving community for IT guys to bemoan terrible products or sing praises for the good ones.
I don’t know about you, but I always take online reviews with a grain of salt. I’ve seen enough padded, biased reviewers to know when I’m being conned. Are you that savvy? Do you think IT Central Station can somehow avoid the bias?
To celebrate the Labor Day weekend, here are a few light-hearted gems from the Twitterverse you might hear on the server room floor.
In response to the news that President Obama would do an “Ask me anything” session on social hub Reddit, users of the service crashed the site.
Twitter denizen Ethan Kaplan posted the following in response:
Next up is a facepalm-worthy tweet from Rob Malda, aka @cmdrtaco. Let’s hope the referenced study by Citrix was exaggerating a bit.
Trolling, for those who don’t know, is a little like playing pranks, only generally mean-spirited and performed on an undeserving target. Naigos, an open source network monitoring tool, sends lots of alerts, which can get annoying very quickly.
This one is straight from the fingertips of the venerable DevOps Borat. This little quip raises fourth generation programming languages to “old man on the mountain” level.
The CoderVersion hashtag floated its way around Twitter and left us giggling. Though there were too many to post, here’s one that might hit close to home.
Early reports on the acquisition picked up this definition of Xsigo’s I/O virtualization appliance and ran with it, lumping the Xsigo purchase in with VMware’s blockbuster acquisition last week of networking virtualization player Nicira.
Then came the social media outrage. The term “SDN-washing” was thrown around.
Quoth Joe Onisick (@jonisick) on Twitter:
“Xsigo is to SDN what McDonald’s is to fine dining.”
A little while later, Nicholas Weaver (@lynxbat) chimed in with this gem:
“Every time a tech reporter compares Xsigo to Nicira, a puppy dies.”