Grab a flashlight, head to your nearest lights-out data center, and hunker down for some creepy data center horror stories courtesy of some data center experts. Are these true tales? Stay tuned for tomorrow’s part two.
Data center fright night, part one
“Night of the Living Data Center”
All was quiet and dark – well, it was a lights-out data center after all. The lone admin had been called in. Something had gone wrong in the facility. He went to push the door and – nothing. He remembered he had to use his two-factor secure key, finger print and retina recognition to get inside.
Creeping past the reception desk, a low moan sent shivers down his spine, but it was just the murmurings of the under-paid security guard catching up on his sleep after a 12-hour shift at the local Walmart. Moving silently along, the admin heard “zip, click, zip, click, zip, click” as he walked along. Glancing behind him, he saw the lights slowly clicking off as he moved along. In his befuddled state, it took some time to realize this was the movement-sensitive lights turning on and switching off along the corridor he was in.
At the final interior door, he waited, listening in case something was beyond the door that shouldn’t be. Mind you, as the data center had been built to LEED quality, the amount of insulation in the walls would have deadened the sounds of the demons of hell escaping from the Solaris box he had in the far corner. Screwing up his courage, he opened the door and tip-toed in. There was enormous “CRACK!”
No, sorry, there was an enormous CRAC – it had been there for 20 years, so it shouldn’t have surprised him. Another low moan, far different to the security guard’s sleeping sounds, went past him in a gust of warm, fetid air. He made a mental note to check the raised floor for holes where cooling might be leaking and get the drains fixed.
He made it to the console and logged in. Strange images appeared in front of his eyes – ones he had never seen before. Immediately switching to CLI mode, the easier, more functional and useful graphical user interface was instantly hidden. His fingers flew over the keyboard. Eventually, he calmed down and started typing. The strange incantations he typed brought up steams of glowing data – log files, audit trails, details of pizza delivery companies.
Eventually, tracing the root cause of the issue caused him to take a sharp intake of breath. “How could this happen?” he wondered. Well, it was a waste of time asking anyone else, since they were all in bed at their homes. With a rising sense of panic, he gathered up the tools of his trade – the old Bell telephone modem he might need where he was going, the book on troubleshooting systems and the 1 million candle flashlight. For good luck, he also took a silver thumb drive. Moving away from the console, he muttered some charms under his breath “agility, flexibility, SaaSability” and “I wish we had outsourced this years ago.”
In the corner was a massive black monolith – the admin had always assumed that this was a throwback to the film “2001: A Space Odyssey” and that it was something facilities looked after – nothing for a tech guru like him to worry about. However, the console had provided directions that said this was the source of the trouble. Reading the book, the admin identified the special way to use the runes on the monolith to access secret bits. Finally, he folded down a screen and pushed a switch. The green and black screen flickered, showing a talismanic figure, which changed over into a single word and sent the admin screaming back into the night.
In dripping, blood-red letters, it said “ABEND.”
IT must be ready for fundamental shifts in business and technological paradigms.
Stephen J. Bigelow, Senior Technology Editor
IT professionals are used to change – the reality of change is as old as IT itself. Everyone swaps servers during technology refresh cycles, and new versions of operating systems or key business applications can have IT staff working overtime to get users updated.
Still, change is always a challenge for IT. And while the logistical and technical demands of change will always stress budgets, tax patience, and fray nerves, the IT department soldiers on to solve problems and fight fires and support the business.
But what happens when a technology fundamentally changes the way a business or industry operates?
Just consider an emerging technology like 3D printing where solid objects are created by depositing materials in successive layers according to a digital model. The resulting 3D model can be used as the basis for molds and other core manufacturing processes – even using materials that are correct for the finished product, allowing a manufacturer to actually fabricate finished goods on demand.
The very notion that a company can produce products on-demand flies in the face of traditional business paradigms.
Consider the manufacturing process itself. Traditional manufacturing is based on economies of scale using mass-production of identical items. This involves further practices including logistics, warehousing, sales – the entire business infrastructure which relies on IT resources and support. When small numbers of products can be inexpensively fabricated on demand using an easily manipulated digital model, the business is profoundly affected; and so are the services and support that IT must provide.
There are implications for IT. Designs would proliferate as the number of models multiplies from designers and customers, requiring data storage and security. The concepts of enterprise resource planning (ERP) would change dramatically because the flow of materials into and out of the business would be radically different. Warehousing for work-in-progress and finished goods would be virtually eliminated. Goods could also be built on-site or at remote locations, greatly reducing transportation demands. Our very definition of manufacturing could change. Just imagine an automotive shop that can fabricate certain parts right on the shop floor, or a military unit that can produce key spare parts in the field.
Of course, such fabrication technology is far from perfect; today. But it’s an example of the way that new technologies and their refinement can re-define concepts and practices that are (in the case of mass-production) centuries-old. It’s a wakeup call that IT professionals must look ahead at the changing needs of business and position staff and systems to handle the types of changes that may appear on the horizon – or risk being discarded as obsolete.
Notoriously secretive — at least when it pertains to the innards of its data centers — Google is now offering a peek at the pipes and people that deliver its products — Google Search, Gmail, YouTube and Google Maps — to your computing device.
“Where the Internet lives” features stunning photographs by Connie Zhou of several Google data centers to show the general public the server racks, cooling infrastructure and personnel at the tech giant’s facilities that had been under wraps until now.
You can also take a virtual tour of the Lenoir, N.C., data center through the Street View function in Google Maps. The company’s sense of humor shines through during the tour; if you examine each frame carefully, you’ll spot a few interesting items.
It’s an intriguing behind-the-scenes look at the facilities — rumored to number about three dozen in total — required to handle a bulk of the day-to-day Internet-based workloads that millions of people use on a daily basis.
Cut costs, improve efficiency. Such is the mantra of many a data center manager. While tech giants like Google and Facebook strive to create better, more energy efficient data centers, a small team of researchers from Cornell University and Microsoft have gone back in time 120 years and come up with a way to eliminate another threat to efficiency: cables.
Mathematician Arthur Cayley published a paper in 1889 called Oh the Theory of Groups – mathematical groups, that is – that was full of graphs and equations.
In 2012, those graphs and equations were used to design a wireless data center network running on a 60 GHz wireless band.
According to the paper’s abstract, the benefits — besides eliminating networking cable and switch costs — would include higher bandwidth and fault tolerance and lower latency. Adopting a spoke-and-wheel rack system for servers would facilitate communication using specially built Y-switches would help direct that traffic between racks.
This would mean a complete change in server form factor. The basic parts would remain the same – hard or solid state drive, CPU and RAM – with networking cards replaced with Y-switches. The paper goes on to mention changes to data center routing protocols, MAC layer arbitration and design schematics for the customized Y-switch. The full document is available on Cornell’s Website.
Once servers are cylindrical, it’ll be exciting to see how the buildings surrounding them change to suit. What do you think? More data centers in old missile silos?
That’s right, science has just upped the ante for cool data center locations. Forget Iceland or Oregon, now there’s a proposal to put a supercomputer on the moon.
Ouilang Chang, a graduate student at the University of Southern California, presented what he calls the Lunar Supercomputer Complexto help ease the burden of scientific “big data” processing and network traffic on terrestrial facilities. Chang says the rate of information is starting to exceed the ability of the networks to keep up. Not only that, but Chang feels the lunar site would give scientists the ability to boost the power and effectiveness of telescopes here on Earth.
In spite of the fact that this is such a vast undertaking, Chang wants to see it done in the next 10-15 years. If you consider that there’s already a plan in place to start mining asteroids, this isn’t too huge a leap.
The Lunar Supercomputer Complex would be built on the dark side far side of the moon, which would allow moderate protection against extreme temperatures and access to potential ice under the surface for cooling. The complex would include communication arrays, a data center with supercomputers, data storage, power, cooling, radiation shielding and ultimate nerd-cred for any data center architects lucky enough to draw the straw to work on the project.
The good news is they can probably save money on locks and security personnel. But would this facility ever be built?
Back in January, when Newt Gingrich proposed his moon base, people jumped on it as an “impossible dream” due to lack of available funding. But maybe if the billionaire asteroid miners take notice, the moon data center could come to fruition.
Here’s the scenario: You’re the only tech-savvy guys in a very small town, and you come across mysterious devices labeled “Pluto Switch” sitting in your distribution center. What do you do?
Post about it on the Internet, of course! And that’s exactly what happened, according to an article on Wired.com.
The gentlemen posted images of the hardware — which turned out to be a custom-built network switch — on a networking forumand tried to get it working. Many forum users offered help, wanted clearer pictures or asked to buy the switch, but in the end a little bit of sleuthing uncovered they were meant for Google.
Google builds much of its own hardware, as shown by hardware job postings on the company’s site, and further deduction revealed the town – Shelby, Iowa – was only 30 miles from one of Google’s data centers. One of the men theorized the switches were left at their facility by mistake during a delivery run.
The device was sent back to Google, the men who found it got T-shirts and Google tried to shut down the post with the images. So far, the forum hasn’t removed the post, but it has scrubbed all reference to the original posters’ identities.
Sherlock Holmes would be proud, Internet sleuths.
The last time I bought a video game, I grudgingly went to a brick and mortar store but checked the ratings on Amazon before I went. Buy local, shop global and whatnot. The last time I bought a computer – or parts for one, rather – I bought from NewEgg after scouring reviews for hours.
If you wanted to shop for enterprise software or hardware, there was no business equivalent of, say, an Amazon where you could see what other users had to say about a particular product. And if you wanted to warn other IT pros about a particular blade server that melted in its rack or you just needed to rant about XenServer, you were relegated to various forums scattered around the Web.
But a new site called IT Central Station attempts to fill that void. According to the company, the site will feature social networking, user validation and a bunch of different categories.
The site presents itself as a peer review site for IT with no vendor bias – though they’ll happily accept ads! – and as a thriving community for IT guys to bemoan terrible products or sing praises for the good ones.
I don’t know about you, but I always take online reviews with a grain of salt. I’ve seen enough padded, biased reviewers to know when I’m being conned. Are you that savvy? Do you think IT Central Station can somehow avoid the bias?
To celebrate the Labor Day weekend, here are a few light-hearted gems from the Twitterverse you might hear on the server room floor.
In response to the news that President Obama would do an “Ask me anything” session on social hub Reddit, users of the service crashed the site.
Twitter denizen Ethan Kaplan posted the following in response:
Next up is a facepalm-worthy tweet from Rob Malda, aka @cmdrtaco. Let’s hope the referenced study by Citrix was exaggerating a bit.
Trolling, for those who don’t know, is a little like playing pranks, only generally mean-spirited and performed on an undeserving target. Naigos, an open source network monitoring tool, sends lots of alerts, which can get annoying very quickly.
This one is straight from the fingertips of the venerable DevOps Borat. This little quip raises fourth generation programming languages to “old man on the mountain” level.
The CoderVersion hashtag floated its way around Twitter and left us giggling. Though there were too many to post, here’s one that might hit close to home.
Early reports on the acquisition picked up this definition of Xsigo’s I/O virtualization appliance and ran with it, lumping the Xsigo purchase in with VMware’s blockbuster acquisition last week of networking virtualization player Nicira.
Then came the social media outrage. The term “SDN-washing” was thrown around.
Quoth Joe Onisick (@jonisick) on Twitter:
“Xsigo is to SDN what McDonald’s is to fine dining.”
A little while later, Nicholas Weaver (@lynxbat) chimed in with this gem:
“Every time a tech reporter compares Xsigo to Nicira, a puppy dies.”
The IT world has had a decades-long love triangle with air- and water-cooling. Air-cooling takes IT to the prom, but now water-cooling is holding up a boom box outside IT’s window to win it back.
IBM has made so many headlines with the “world’s fastest” supercomputer, Sequoia. But it also made waves by introducing a new commercial supercomputer, the SuperMUC. It boasts direct hot-water cooling and superb energy efficiency – using 40% less energy than air-cooling, says IBM.
The PR video from the Leibniz Supercomuting Centre says the SuperMUC’s cooling system is based on the human circulatory system – a fun medicine/technology crossover. Cold water goes in directly to the processors and carries hot water out to a heat exchanger, which then heats the facility.
Apparently, the facility housing SuperMUC has successfully eliminated CRACs from the equation and is saving Leibniz a million euros a year. IBM used to cool mainframes with water, but increased processor density and cheaper air conditioning drove data centers to adopt air-cooling. Now that energy costs are on the rise and there’s an emphasis on going green, companies are once again looking to liquids to cool their machines. Plus, according to Robert McFarlane, Principal at Shen Milsom and Wilke, it’s hard to argue with the fact that “water is approximately 3,500 times more efficient than air.”
The hurdle for many facilities is infrastructure. Liquids require pipes. Even SuperMUC wouldn’t be able to use that capillary-inspired cooling system without the supporting infrastructure.
Internap, a data center hosting facility in various U.S. cities, has built the newest expansions of their facility with underfloor piping infrastructure to get glycol directly to servers. Older parts of the facility use hot/cold aisle air-cooling with the underfloor space used only for air.
Then there’s Google, which built a waste water processing facility to provide water for cooling, thus eliminating some of the strain on the community.
But both of those examples are new builds. It will be interesting to see how invasive and disruptive adding water-cooling infrastructure would be to an existing data center.
Do you think more facilities going to pony up the infrastructure cost and switch (back?) to water-cooling, or is the relative comfort of air-cooling enough to keep data centers happy?