Grand Prize Winner

An anonymous entry takes the Nintendo Wii grand prize.
Click to see why this member wishes to remain anonymous.

Data Center Contest: Now that’s what I call multitasking

The Tales from the Data Center contest is now over and we had a lot of fun with it. The lucky winner has been selected and will be receiving a Nintendo Wii.

Visit the ITKE Community Blog or subscribe to our newsletter to learn more about our other contests. If you’d like to subscribe to our newsletter (currently twice a week) just login to IT Knowledge Exchange and click the Edit My Account link. Then check the box to receive our IT Knowledge Exchange Community Newsletter.

I listed some other contest entries below for your amusement. Thanks for stopping by to see who won.

Community Manager

Enjoy these other contest entries…

Data Center Contest: Disaster recovery using non-Microsoft “Windows”
Data Center Contest: Successful treatment of Mayo Clinic kill switch
Data Center Contest: Ooh, ooh that smell …can’t you smell that smell?
Data Center Contest: Oh what a tangled web we weave
Data Center Contest: Bathroom or Server room?
Data Center Contest: Saran Wrap to the rescue!
Data Center Contest: The 2:00 AM Wind Tunnel
Data Center Contest: Nuns bring network to its knees
Data Center Contest: The great Chicago flood of ‘91
Data Center Contest: Zut alors! How many cables?
Data Center Contest: Hose down those hard drives!
Data Center Contest: The Orville Redenbacher backup technique
Data Center Contest: Last one out, turn off the lights

We also received a couple of photos with this contest entry. These ought to curl your toes…

Community Manager

From a Member who wishes to remain anonymous:

Evolution is supposed to be an systematic increase in order, complexity and efficiency. Well, along the way, Darwin’s concept was forgotten as this “wiring closet” was evolving.

The original panel and wiring was installed in the late 70′s and as extensions and additional outside lines were added, and changed, this arrangement became more …um…interesting.

About half of this company’s PBX telephone gear still runs through this panel, along with all six of their fax lines. The eight incoming telco lines are also present here (somewhere…).

The most recent addition is the network switches and their APC battery backup (barely visible above the switches) that handle about 20 of the company’s 50 workstations.

There is no documentation, no labels, no tags. What the image does not show is the fact that there is about 8 feet of space between the bottom edge of the photo and the floor. So any work done on this panel must be done from a ladder.

Long term plans are to replace the current Executone PBX system (circa 1992) with a new VOIP system, until then, nobody wants to even breathe on this panel for fear of knocking something loose.

Community Manager

Cynthia submitted this entry:

This probably isn’t the Craziest Story you’ve had, but for us, it’s pretty funny. We have a policy that won’t allow us to take photos inside the facilities, so I have to share the story instead.

One of our IT employees smelled an “electrical burning” smell. After sniffing around, we found the iSeries console had been fried. On the table where it sat was quite a bit of water and more was dripping from the ceiling. We had just had a particularly hard downpour and come to find out, the roof had sprung a leak.

Maintenance acted immediately bringing us a 55-gallon trash can and a huge roll of plastic! They temporarily fixed the roof by putting some plastic down and holding it with a few bricks!

We pushed the iSeries and it’s attached devices a few inches away from the dripping, draped plastic over it’s front and sides, and dug our last dumb terminal out of storage to replace the console unit. It took almost 2 weeks to get the roof repaired. Thankfully, we were blessed with mostly dry weather.

It never ceases to amaze me that our maintenance department doesn’t seem concerned about the state of our computer room which houses hundreds of thousands of dollars worth of equipment.

I wish I had a picture of the iSeries & tape drive draped in plastic, with the huge blue trash can, ceiling tiles and floor tiles removed, and a ladder in the middle of it all. It was a mess!

Community Manager

Gary submitted this entry:

In 1994, I joined the Mayo Clinic Rochester, MN campus as a senior Systems Administrator. My first day was spent getting the guided tour of the data center facilities at the Clinic. We were touring the main data center. This facility housed the Clinic’s mainframe, and a host of other distributed enterprise systems. The day was getting rather late as we were approaching the end of the tour. The lateness of the day also marked the end of the shift for the Data Center operations staff.

Our tour group was being escorted through the command center on our way out to the main data center operations area. Located along the wall next to the data center entrance was a large red kill switch.

Looking back now, this reminds me of the big red “Easy Button” depicted in popular Staples’ television media adds. This kill switch when pressed engaged the emergency power shutdown for the entire data center. The kill switch was within easy reach from the entrance door. There was no protective encasement or other security feature over the kill switch. Somebody could accidentally trip and hit the switch, which would engage the emergency power shutdown.

This fact must have been on the mind of our guide as well. Our guide strategically positioned himself between the kill switch and the door entrance to prevent such an accident from occurring. After everyone safely assembled together on the production floor, our guide resumed the tour. Our group moved over to opposite end of the production floor to view new StorageTek tape silos. From this vantage point, we had a clear unobstructed line of sight to witness the events that followed.

Observing a data center power shutdown is really something to see. One minute, we were gazing at the swinging robotics arm contained within the StorageTek silo fetching tape cartridges. The next minute the only viewable lighting was coming from the emergency lights. In retrospect, the outage lasted only a few minutes. But during this time there were teams of programmers and other IT support specialists that flooded out onto the production floor. During the power outage, our tour guide remained calm and collected. During these events he kept the group together and out of everyone’s way.

We later learned at the end of our tour of the events that led up to the data center power shutdown. A developer had called the command center to request a hard reboot on one of the servers in the command center. Confused, the operator asked the developer where the server was located. The developer instructed the unfortunate operator to look for a red power switch on the server located next to the wall. The operator saw one red switch. The rest was history.

The next day when I was out on the production floor doing server maintenance I noticed there was a plastic security encasement covering that one particular red switch!!!

Community Manager

From a Member who wishes to remain anonymous:

My first job in IT (at the time it was just called Data Processing) was in the early 80′s at a community college that I had just graduated from with an Associates in computer programming. I was hired as an operator for an IBM system 38 that was housed in a large room on the first floor of the school with a cubicle in the corner, which served as my office and of the programmer who had been there for 2 years before me.

Anyway, our back up system consisted of a weekly ritual of saving the most important files to diskettes housed in magazines that held 10 diskettes each. The normal weekly back up used about 10 magazines. The monthly backup was of the whole system, which used about twice as many magazines. It was cumbersome, but that was my main job since I worked 2nd shift. After the back up finished, I had to place each magazine on a bookshelf that sat in the corner.

The bad part was our disaster recovery system, for which we did drills for once a month. When the alarm went off, the programmer had to run outside and stand outside the window to our office. I then had to power down the computer and take each magazine of diskettes and toss it out the window to her until all of them were out. Only then could I “save” myself and join everyone else outside. The programmer then had to take the magazines and put them in her car and drive off site. Needless to say, all of the students and faculty got a big kick out of our little ritual.

Well, wouldn’t you know it, an actual fire broke out at the school one day and it just so happened to be the day the programmer wasn’t there. My boss, the data processing manager, had to run down the hall and take her place. Everything went without a hitch since the fire was in a part of the building far away from the computer room. After I threw all of the magazines out the window, I proceeded outside where I noticed that my boss had already put them in his car and was driving away from the building.

After all of the hoopla died down and the minor fire was put out, we were allowed back inside. I asked my boss when he returned where the magazines were and he informed me not to worry about them, he would just keep them in his car and for me to initialize and use new diskettes for that night’s backup, which I did.

Well, a week later while preparing for the monthly back up, I had to go to my boss and ask for the magazines since I didn’t have enough to do the whole back up. It was then that he informed me in a small voice that he had forgotten to take them out of the back of his trunk, and the diskettes had all warped and melted from the heat.

Needless to say, he then authorized me to go and buy new diskettes and magazines and made me promise not to tell anyone what happened. Also, soon afterward, he hired a company to house all of our back ups off site and we adopted a rotation system for our backups as well as purchasing a tape drive.

Community Manager

Adrian submitted this entry:

We’d just moved into a new data centre and had a Lights Out operation in this one area where we had a Tech replacing some part. We left him there for a while and when he left, he reached back to turn the lights off and instead hit the Emergency Power Off (EPO) button, dropping power to the entire area.

We quickly put covers over all the EPOs (like you see on a fire alarm) and labeled them more clearly.

Community Manager

From a Member who wishes to remain anonymous:

Like the proverb says, “A picture is worth a thousand words.” This photo was sent in by one of our members, stating that an IT coworker had taken this picture while visiting a newly acquired company. As you can see — this data center has all the comforts of home.

Community Manager

Shava submitted this entry:

I was the manager of tech support for a Data General OEM back in the very early 80′s. One day, our only customer in Montana opened his call with, “I want you to get [undisclosed] from Data General tech support FIRED!”

He had a one Eclipse computer shop. The Eclipse was, essentially, a
DEC PDP-11 knock off. It was entirely held in one 19″ rack mount, and still had a switch bank to manually boot the system in case of boot sector corruption. We’re talking ancient of days, here.

The CPU unit was mounted above a 5MB fixed/removable disk. This is to say, the whole computer ran off a 5MB fixed platter (pre-Winchester sealed drives) and to back it up, you opened the drawer and put in another 5MB removable cartridge. This wasn’t a floppy – it was a 24″ round, 4-5″ deep, maybe 12# assembly, with a handle in the center to heft it out when you had to remove it.

The gentleman with the complaint had set his accounting data to back up and gone to the movies. Coming back to the office from the movies, he placed the remnant of his extra large tub of buttered, salted popcorn atop the 19″ rack. Removing the backup cartridge, in preparation for inserting the transaction logging removable cartridge, he swung the backup cartridge to set it on top of the rack.

His popcorn dumped into the disk drive, salt and butter and all. Remember, these drives are not sealed.

So, knowing the drive enclosure was supposed to be kept very clean and free of dust, he went to the men’s room and grabbed a wad of coarse (linty) paper towels and tried to wipe out the salt, butter, and popcorn as well as he could – probably shoving a good deal of it into the chamber where the ceramic read heads were recessed during cartridge change.

By this point, I had the guy on speakerphone, and my crew of about a dozen was just about DYING trying to suppress raucous laughter.

Having done “what he could” to clean up the mess, he put in the transaction disk, and spun up the drive. At this point, the drive started to scream and wail – no doubt, a massive head crash. We could all imagine the heads shaving oxide off of those big brown platters like a Dremel tool.

He tried spinning the drive down, but the noise didn’t stop right away, so he tried spinning it back up – and then in panic, he cut the power to the machine. He could smell smoke

He then opened the drawer to look inside.

Now, these disks weren’t *sealed* but they tried to keep air out when they were running, so there was a *pop* as the removable cartridge came out. But that was a minor effect as, heated probably over 500 degrees F, some bits of popcorn and the atomized butter ignited, causing a Flash Paper-like burst of flame, singeing the guy’s eyebrows a bit.

Everyone in my office listening to this guy was completely losing it by this point.

“Thank God this didn’t happen in a couple weeks when we’re putting the sprinklers into the computer room!” he puffed.

“Yes, sir,” I said calmly, “and this is when you called Data General?”

“I hate calling field service,” the guy admitted, “And they never get here fast – we’re a couple hours from the nearest tech. But I called him up and you know what the jerk told me?”

“Uh, no sir.”

“He told me my contract didn’t cover me for errors of gross stupidity! I want that man *fired*!”

Well, weeks later, the man did get his computer and drive replaced (the CPU had fried from the heat or some effect of atomized grease on the insulation, who knows?). Although it’s not clear how much of the damage his contract should have covered, DG covered it all, including (I think) three site visits in all. We always speculated that they thought they got value back in the hilariousness of this man’s story.

Community Manager

From a Member who wishes to remain anonymous:

It was a few years ago (not as many as you’d think, given the equipment in use at the time), and we were still running a VAX system on an old Alpha server. Every Friday evening late, that system was taken down for maintenance and backups, and then was brought back on-line before 6 AM the next morning.

Very early one Saturday morning (around 1:00 AM), a user called the computer center wanting to get into the VAX system, and was told it was unavailable due to the scheduled maintenance and backup cycle. Not thrilled about that, they tried to cajole the operator into stopping the cycle and bringing up the system. When informed that was impossible, a very disgruntled user hung up.

The very next week, at roughly the same time, the same user called with the same issue, and same demands for the system. History repeated itself again and again. On several subsequent Friday evenings, the same user and request provided the same operator some additional entertainment for their shift.

Finally one night, having had enough abuse for one day, the operator told the user he would really like to be able to help out, but the VAX had been disassembled, and the drives were currently being hosed down. Obviously the computer couldn’t be put back together sopping wet, but as soon as it was dried out, he’d gladly reassemble it and bring it up for use. The estimated time to complete that looked suspiciously like the standard up time for the normal cycle.

That was the last time that user called. To this day, those of us who know the story wonder exactly what kind of a picture was left in the caller’s mind.

Community Manager

Tom, Data Center Specialist, submitted this entry:

I took a job at this company that was based out of Paris, France. We were located in Fort Worth, TX. I get there and one of my responsibilities was to “Maintain the Data Center.” My second day there (the first day was strictly orientation) I was shown the “data center”… a closet with one Server Rack, a wiring rack and seven large Compaq Proliant 1600 Servers, sitting on two wooden crates which were filled with cardboard.

My UPS power came from four APC 1400 UPS devices. Now attached to these four UPS devices were three servers, one was a test server, one was a remote RAS box that allowed the Italian team to monitor their software, and the other was on the Checkpoint firewall. If a power surge occurred, everything dropped except those value pieces of equipment but no one would work.

Now the best was the wiring rack. First of all, nothing had ever been removed. So if they removed a data cable from a server, they left the other end connected. Also, we have 400 cables for a 200-employee operation. Each server was able to have two network links, and could be paired, but they only had one link each. So the remaining 189 cables were just there for decoration, I believe. Oh and by the way, no cable was labeled, so we have to hunt and find where everything was connected. Also, there was only a Layer 2 Switch in place so no real routing could be accomplishing (VLANs, etc).

The first thing I did was attach the proper mission critical servers to the UPS devices. I was unable to get all the devices on the UPS devices, but I was able to keep the overall main network devices and main storage on backup power. Then I started to map out the data cables. I would trace each cable to its connection point and label it (both on the cables ends – yes, each end was labeled) as well as creating a master wiring grid in VISIO. I removed over 189 cables that were not being used. But it was still in bad shape and most of the cables were 50 ft in length for a 6 ft connection point.

It took me three months to convince leadership that we have to renovate the so-called data center. I ordered a used Server Rack (for $50.00 dollars) to provide a real home for the seven servers that were resting on the wooden crates. I also ordered a 20KVA UPS Array (APC) to accommodate the true power consumption in the room, as well as removed several older servers that served limited purpose by consolidating them to two servers (removed four). I also had to rewire the wiring rack since, as you would need the “Indiana Jones” outfit to cut through the wiring to see anything. Since budget was limited I had to remake the cables from the existing cables. We had two day (one weekend) to accomplish the full job.

When we finished we had two server racks full loaded with proper labeling on the server devices, as well as the cables being labeled. I bonded the NICs for each server and ran dual pair network cables back to the switch. I upgraded the switch from an Extreme Alpine switch to a Cisco 4500 Switch with proper Layer 3 switching. I even labeled the ports in the switch so we could identify each connection from anywhere. We had all the devices connected to the new centralized UPS device that gave us a 45-minute window to have a proper shutdown if a power outage would occur. From the revamp we had better overall network operation (better packet routing and VLANs help the network congestion), better backup power solution – even though it was not as I desired, but budget was tight, and everything was labeled so we could find everything in a timely manner.