October, 2008

Data Center Contest: Last one out, turn off the lights

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Adrian submitted this entry:

We’d just moved into a new data centre and had a Lights Out operation in this one area where we had a Tech replacing some part. We left him there for a while and when he left, he reached back to turn the lights off and instead hit the Emergency Power Off (EPO) button, dropping power to the entire area.

We quickly put covers over all the EPOs (like you see on a fire alarm) and labeled them more clearly.

Are the Uptime Institute’s Data Center Rating Tiers Out of Date?

“Share your ‘Tales from the Data Center’ and you could win a Nintendo Wii”

Let me start by saying I have the utmost respect for the Uptime Institute’s Pitt Turner, P.E., John Seader, P.E., and Ken Brill and the work they have done furthering the cause of providing some standards to an otherwise standard-less subject like data center design. However, as a data center designer I feel their definitive work, Tier Classifications Define Site Infrastructure Performance, has passed its prime.
The Institute’s systems have been in use since 1995, which is positively ancient in the world of IT.
In its latest revision, the Uptime Institute’s Tier Performance Standards morphed from a tool for IT and corporate decision makers to consider the differences between different data center investments into a case study for consulting services pushing for certification against their standard.
While the data their standard is based upon has been culled from real client experiences, the analysis of the data has been interpreted by only one expert company, ComputerSite Engineering which works in close collaboration with the Uptime Institute. Surely, the standard could be vastly improved with the outside opinion and influence of many of the, just as expert, data center design firms that exist.
Case in point, the Uptime Institute has repeatedly defended the notion that there is no such thing as a partial tier conforming site (Tier 1+, almost Tier III, etc.). They argue that the rating is definitive and to say such things is a misuse of the rating guide. While I understand the argument that a site is only as good as its weakest link, to say that a site incorporating most, but not all of the elements of the tier definition is mathematically and experientially wrong.
PTS’ actual experiences bear this out. Our clients that have all the elements of a Tier II site, except for the second generator, are clearly better than those with no UPS and/or air conditioning redundancy (Tier I). Therefore, if not for Tier I+, how do they suggest to account for the vast realization between the real availability of the two sites?
It is interesting that most data center consulting, design, and engineering companies nationwide utilize elements of the white paper as a communications bridge to the non-facility engineering community, but not as part of their design process. In fact, most have developed and utilize their own internal rating guides.
While I will continue to utilize their indisputable expertise as a part of my own interpretation in directing PTS’ clients with their data center investment decisions, I suggest that clients would be wise not put all of their eggs in the Institute’s basket at this point in time.

What is your outlook on the Uptime Institute’s Tier Performance Standards? Is the four-tier perspective outdated or is it still a meaningful industry standard?

Pete

Data Center Contest: Now that’s what I call multitasking

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

From a Member who wishes to remain anonymous:

Like the proverb says, “A picture is worth a thousand words.” This photo was sent in by one of our members, stating that an IT coworker had taken this picture while visiting a newly acquired company. As you can see — this data center has all the comforts of home.

Let’s Talk About: The Role of Sprinklers in Computer Room Fire Protection

A number of clients have asked us about the viability of replacing their ‘wet’ sprinkler systems with a dry-type fire suppressions system, such as FM-200. Not many IT personnel understand the role of water-based fire suppression systems, but all realize the potential for water in the data processing environment to be a “bad thing.”

The short answer is that sprinkler systems protect the building and dry-type systems protect the equipment. In most cases a dry-type system cannot take the place of a sprinkler system, it can only be installed in addition to it. At the end of the day, the local fire inspection is the authority and has jurisdiction over what is permissible. This is the reason that pre-action sprinkler systems are primarily used for computer room fire protection. That being said, fire prevention provides more protection against damage than any type of detection or suppression equipment available. For Tier I and Tier II computer rooms, PTS often recommends installing only a pre-action sprinkler system activated by a photo-electric smoke detection system and forgo a dry-type system and VESDA system. We find the most effective strategy is to emphasize prevention and early detection. This allows the client to maximize availability by investing in solutions for areas of higher risk, such as fully redundant power and cooling systems.

For more information on fire protection, read our vendor white paper “Mitigating Fire Risks in Mission Critical Facilities,” which provides a clear understanding of the creation, detection, suppression and prevention of fire within mission critical facilities. Fire codes for Information Technology environments are discussed. Best practices for increasing availability are provided.

Pete

Data Center Contest: The Orville Redenbacher backup technique

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Shava submitted this entry:

I was the manager of tech support for a Data General OEM back in the very early 80’s. One day, our only customer in Montana opened his call with, “I want you to get [undisclosed] from Data General tech support FIRED!”

He had a one Eclipse computer shop. The Eclipse was, essentially, a
DEC PDP-11 knock off. It was entirely held in one 19″ rack mount, and still had a switch bank to manually boot the system in case of boot sector corruption. We’re talking ancient of days, here.

The CPU unit was mounted above a 5MB fixed/removable disk. This is to say, the whole computer ran off a 5MB fixed platter (pre-Winchester sealed drives) and to back it up, you opened the drawer and put in another 5MB removable cartridge. This wasn’t a floppy – it was a 24″ round, 4-5″ deep, maybe 12# assembly, with a handle in the center to heft it out when you had to remove it.

The gentleman with the complaint had set his accounting data to back up and gone to the movies. Coming back to the office from the movies, he placed the remnant of his extra large tub of buttered, salted popcorn atop the 19″ rack. Removing the backup cartridge, in preparation for inserting the transaction logging removable cartridge, he swung the backup cartridge to set it on top of the rack.

His popcorn dumped into the disk drive, salt and butter and all. Remember, these drives are not sealed.

So, knowing the drive enclosure was supposed to be kept very clean and free of dust, he went to the men’s room and grabbed a wad of coarse (linty) paper towels and tried to wipe out the salt, butter, and popcorn as well as he could – probably shoving a good deal of it into the chamber where the ceramic read heads were recessed during cartridge change.

By this point, I had the guy on speakerphone, and my crew of about a dozen was just about DYING trying to suppress raucous laughter.

Having done “what he could” to clean up the mess, he put in the transaction disk, and spun up the drive. At this point, the drive started to scream and wail – no doubt, a massive head crash. We could all imagine the heads shaving oxide off of those big brown platters like a Dremel tool.

He tried spinning the drive down, but the noise didn’t stop right away, so he tried spinning it back up – and then in panic, he cut the power to the machine. He could smell smoke

He then opened the drawer to look inside.

Now, these disks weren’t *sealed* but they tried to keep air out when they were running, so there was a *pop* as the removable cartridge came out. But that was a minor effect as, heated probably over 500 degrees F, some bits of popcorn and the atomized butter ignited, causing a Flash Paper-like burst of flame, singeing the guy’s eyebrows a bit.

Everyone in my office listening to this guy was completely losing it by this point.

“Thank God this didn’t happen in a couple weeks when we’re putting the sprinklers into the computer room!” he puffed.

“Yes, sir,” I said calmly, “and this is when you called Data General?”

“I hate calling field service,” the guy admitted, “And they never get here fast – we’re a couple hours from the nearest tech. But I called him up and you know what the jerk told me?”

“Uh, no sir.”

“He told me my contract didn’t cover me for errors of gross stupidity! I want that man *fired*!”

Well, weeks later, the man did get his computer and drive replaced (the CPU had fried from the heat or some effect of atomized grease on the insulation, who knows?). Although it’s not clear how much of the damage his contract should have covered, DG covered it all, including (I think) three site visits in all. We always speculated that they thought they got value back in the hilariousness of this man’s story.

Data Center Contest: Hose down those hard drives!

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

From a Member who wishes to remain anonymous:

It was a few years ago (not as many as you’d think, given the equipment in use at the time), and we were still running a VAX system on an old Alpha server. Every Friday evening late, that system was taken down for maintenance and backups, and then was brought back on-line before 6 AM the next morning.

Very early one Saturday morning (around 1:00 AM), a user called the computer center wanting to get into the VAX system, and was told it was unavailable due to the scheduled maintenance and backup cycle. Not thrilled about that, they tried to cajole the operator into stopping the cycle and bringing up the system. When informed that was impossible, a very disgruntled user hung up.

The very next week, at roughly the same time, the same user called with the same issue, and same demands for the system. History repeated itself again and again. On several subsequent Friday evenings, the same user and request provided the same operator some additional entertainment for their shift.

Finally one night, having had enough abuse for one day, the operator told the user he would really like to be able to help out, but the VAX had been disassembled, and the drives were currently being hosed down. Obviously the computer couldn’t be put back together sopping wet, but as soon as it was dried out, he’d gladly reassemble it and bring it up for use. The estimated time to complete that looked suspiciously like the standard up time for the normal cycle.

That was the last time that user called. To this day, those of us who know the story wonder exactly what kind of a picture was left in the caller’s mind.

Data Center Contest: Zut alors! How many cables?

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Tom, Data Center Specialist, submitted this entry:

I took a job at this company that was based out of Paris, France. We were located in Fort Worth, TX. I get there and one of my responsibilities was to “Maintain the Data Center.” My second day there (the first day was strictly orientation) I was shown the “data center”… a closet with one Server Rack, a wiring rack and seven large Compaq Proliant 1600 Servers, sitting on two wooden crates which were filled with cardboard.

My UPS power came from four APC 1400 UPS devices. Now attached to these four UPS devices were three servers, one was a test server, one was a remote RAS box that allowed the Italian team to monitor their software, and the other was on the Checkpoint firewall. If a power surge occurred, everything dropped except those value pieces of equipment but no one would work.

Now the best was the wiring rack. First of all, nothing had ever been removed. So if they removed a data cable from a server, they left the other end connected. Also, we have 400 cables for a 200-employee operation. Each server was able to have two network links, and could be paired, but they only had one link each. So the remaining 189 cables were just there for decoration, I believe. Oh and by the way, no cable was labeled, so we have to hunt and find where everything was connected. Also, there was only a Layer 2 Switch in place so no real routing could be accomplishing (VLANs, etc).

The first thing I did was attach the proper mission critical servers to the UPS devices. I was unable to get all the devices on the UPS devices, but I was able to keep the overall main network devices and main storage on backup power. Then I started to map out the data cables. I would trace each cable to its connection point and label it (both on the cables ends – yes, each end was labeled) as well as creating a master wiring grid in VISIO. I removed over 189 cables that were not being used. But it was still in bad shape and most of the cables were 50 ft in length for a 6 ft connection point.

It took me three months to convince leadership that we have to renovate the so-called data center. I ordered a used Server Rack (for $50.00 dollars) to provide a real home for the seven servers that were resting on the wooden crates. I also ordered a 20KVA UPS Array (APC) to accommodate the true power consumption in the room, as well as removed several older servers that served limited purpose by consolidating them to two servers (removed four). I also had to rewire the wiring rack since, as you would need the “Indiana Jones” outfit to cut through the wiring to see anything. Since budget was limited I had to remake the cables from the existing cables. We had two day (one weekend) to accomplish the full job.

When we finished we had two server racks full loaded with proper labeling on the server devices, as well as the cables being labeled. I bonded the NICs for each server and ran dual pair network cables back to the switch. I upgraded the switch from an Extreme Alpine switch to a Cisco 4500 Switch with proper Layer 3 switching. I even labeled the ports in the switch so we could identify each connection from anywhere. We had all the devices connected to the new centralized UPS device that gave us a 45-minute window to have a proper shutdown if a power outage would occur. From the revamp we had better overall network operation (better packet routing and VLANs help the network congestion), better backup power solution – even though it was not as I desired, but budget was tight, and everything was labeled so we could find everything in a timely manner.

DC-power Sparks Interest among Data Center Designers

  “Share your ‘Tales from the Data Center’ and you could win a Nintendo Wii”

 Rising data center power and cooling costs have renewed industry interest in DC-power, but is DC really a cure-all or just more snake oil?

Up until this point, DC-power hasn’t been desirable for a number of reasons. The large amount of copper wiring requisite for DC-power distribution is expensive and takes up valuable space in the data center. DC-engineered facilities are also more complicated to install and maintain, creating headaches that most data center professionals don’t want or need.

However, tech industry leaders, such as Sun Microsystems and Intel, are paying attention to DC-power. They know data centers are desperate to cut power and cooling costs and are banking that many larger data centers can afford the upfront investment in DC-engineered facilities.

This summer, more than 20 tech companies teamed up with the Department of Energy’s Lawrence Berkeley National Laboratory to run a side-by-side demonstration of a conventional AC-powered rack of servers next to a DC-powered rack. The hope was that DC might prove to be the cheaper, cooler-running, and more reliable data center power option.

AC vs. DC

Could DC-power have finally come of age? According to a whitepaper by American Power Conversion Corp., one of our trusted data center vendors, AC-power and DC-power should be compared based on efficiency, cost, compatibility and reliability. Let’s use this as our guide for comparison.

Efficiency:
DC-power is often touted as being more efficient than AC-power because it requires fewer power conversions that can result in power losses. On the surface this makes sense, but it doesn’t take into consideration that DC cabling must be greatly oversized to deliver the same power. You end up trading increased infrastructure costs and a loss of space for a relatively minor gain in data center efficiency.

Cost:
There isn’t a real cost advantage to making the switch to DC, even though DC-power might lower your overall energy costs. As I mentioned earlier, DC-power requires more extensive infrastructure and engineering. There’s a higher upfront cost which offsets most long-term savings.

Compatibility:
This one is a toss-up as well. Compatibility with AC- or DC-power supplies is determined largely by the type of equipment your facility uses. Keep in mind, though, that it is difficult to obtain DC versions of many pieces of equipment. If your data center uses a mix of AC and DC products, it’s probably a wise decision to go with AC-power.

Reliability:
For the most part, reliability is controlled by battery system rather than the type of power used. When compared with an equivalent DC-powered system, AC-power is just as reliable for data center use.

PTS’ Perspective

The use of DC-power in the data center isn’t common – and with good reason. There is no clear benefit to switching from AC- to DC-power, but there are quite a few drawbacks. Rising energy costs are enough to make companies reevaluate the concept, but AC will remain the dominant choice for data center power in the foreseeable future.

Pete

Data Center Contest: The great Chicago flood of ‘91

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Mark submitted this entry:

The year was 1991 and I was working for a Chicago company located downtown as an analyst. The company resided in a 32-story building. This was the year the subway tunnels experienced a crack, allowing water to enter throughout the lower regions of all the office buildings downtown.

The building in which I worked had all of its power supplies in the basement, which stopped working, thus the air ventilation stopped working as well. This meant immediate evacuation of the building since the windows were all sealed closed, so people wouldn’t accidentally fall from the windows.

So after gathering as much paperwork and stuff as possible, not knowing how long we would be out, we proceeded to the doors to leave and the doors were locked because the electric was off and they are electric-powered locks. People are starting to panic thinking the air is going to run out soon, crazy talk, but within minutes a fireman comes with an axe to break the newly installed security doors.

So we proceed to walk down 30 flights of stairs with boxes of paperwork in hand until we reach bottom. Several were stopped by our empty-handed supervisor, who asked if we would go back to retrieve his briefcase and etc from his office. We replied with “lots of luck” and headed for the local bar, only to see on TV, our supervisor being carried out on a stretcher from trying to retrieve his stuff.

Today’s Question: Are “free” computer room site assessment services worth the money you pay for them?

 “Share your ‘Tales from the Data Center’ and you could win a Nintendo Wii”

It has become commonplace for the myriad of IT and support infrastructure OEMs to offer free site assessment services in an effort to woo clients into purchasing their equipment.While it was already difficult enough for small- to mid-size design consulting service providers to build credibility and brand-identity in the ultra-competitive world of computer room design, in the past few years these firms have seen some of their most valuable vendor partners become chief competitors.

This is not just a case of sour apples. The design services provided by most OEMs do their clients a disservice. Clients are usually only provided the part of the picture that suits the manufacturer and they are forced to fill in the blanks. Unfortunately, the blanks are often not identified. This leads to some very unhappy bean counters.

One leading power and cooling system manufacturer’s entire go-to-market strategy is based on allowing inexperienced enthusiasts to represent themselves as capable designers by providing them with access to an online configuration tool. Being an expert in its use myself, I can safely say the information it provides is rudimentary at best. Our team at PTS Data Center Solutions uses this tool only for ordering purposes and never for design. These online tools are being used by the manufacturer’s own systems engineers, reseller partners, or end users themselves to try to simplify the inherently complicated subject of computer room support infrastructure design.

The manufacturer’s configuration tool only provides solution recommendations for the equipment they manufacture. Much of the rest of the complete solution is missing, including the infrastructure they don’t sell, the labor to install any of it, and/or the engineering services to produce the design documentation required to file the necessary permits. Worse, little advice is provided as to the best project delivery methodology. While I would be the first to admit the traditional consulting engineering community has been slow to adapt to the latest design practices, the truth remains that as-a-matter-of-course changes to facilities still require the services of a licensed engineer. This includes the sizing of the power and cooling infrastructure.
That’s not to say the use of tools doesn’t have its place. Any consultant-recommended solutions should always be based on sound engineering using the latest technologies, such as computational fluid dynamic (CFD) modeling.

Individuals seeking computer room solutions are better served by hiring experienced, licensed, capable design engineers that are well versed in all of the major infrastructure solutions. This ensures that for a moderate amount of money spent in the planning stage you come away with a properly designed project with a well-defined scope, schedule, and budget.

Pete