Prank Video: Busted in the Data Center

All work and no play can make the data center a dull place. How about a little Candid Camera meets Data Center? The APC creative team obliges as they get “creative” with a few unsuspecting colleagues. This is the first of several prank videos – so stayed tuned for more. Thanks.

Community Manager

[kml_flashembed movie="http://www.youtube.com/v/HrjkK2FOuN0" width="425" height="350" wmode="transparent" /]

In Focus: High Density Devices Strain Data Center Resources

A few weeks back I commented on the current boom in data center development. Spurring this trend is the growing need for greater processing power and increased data storage capacity, as well as new Federal regulations which call for better handling and storage of data.
In the scramble to keep up with these demands, the deployment of high density devices and blade servers has become an attractive option for many data center managers. However, a new report from the Aperture Research Institute indicates that “many facilities are not able to handle the associated demand for power and cooling.”
The study, based on interviews with more than 100 data center professionals representing a broad spectrum of industries, reveals that the deployment of high density equipment is creating unforeseen challenges within many data centers.
Highlights of the report include:

• While the majority of data center managers are currently running blade servers in their facilities, traditional servers still comprise the bulk of new server purchases. Mixing blade and non-blade servers in such small quantities can unnecessarily complicate the data center environment and make maintenance more difficult.

• The rising power density of racks makes them more expensive to operate and more difficult to cool. More than one-third of the respondents said their average power density per rack was over 7KW, a scenario that setting the facilities up for potential data center cooling issues and unexpected downtime.

• Respondents report that the majority of data center outages were caused by human error and improper failover.

• What’s really jaw-dropping is that while more than 22% of outages were due to overheating, 21% of respondents admit that they don’t know the maximum power density of their racks. The report points out that “[o]ver 8% of respondents are therefore using high-density devices without tracking power density in a rack, dramatically increasing the potential for outages.”

High density equipment can help data centers keep up with business demands, but only if you can keep things running smoothly. Proper management of power and cooling is essential for meeting the end user’s availability expectations.

Pete

Data Center Contest: Ooh, ooh that smell …can’t you smell that smell?

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Cynthia submitted this entry:

This probably isn’t the Craziest Story you’ve had, but for us, it’s pretty funny. We have a policy that won’t allow us to take photos inside the facilities, so I have to share the story instead.

One of our IT employees smelled an “electrical burning” smell. After sniffing around, we found the iSeries console had been fried. On the table where it sat was quite a bit of water and more was dripping from the ceiling. We had just had a particularly hard downpour and come to find out, the roof had sprung a leak.

Maintenance acted immediately bringing us a 55-gallon trash can and a huge roll of plastic! They temporarily fixed the roof by putting some plastic down and holding it with a few bricks!

We pushed the iSeries and it’s attached devices a few inches away from the dripping, draped plastic over it’s front and sides, and dug our last dumb terminal out of storage to replace the console unit. It took almost 2 weeks to get the roof repaired. Thankfully, we were blessed with mostly dry weather.

It never ceases to amaze me that our maintenance department doesn’t seem concerned about the state of our computer room which houses hundreds of thousands of dollars worth of equipment.

I wish I had a picture of the iSeries & tape drive draped in plastic, with the huge blue trash can, ceiling tiles and floor tiles removed, and a ladder in the middle of it all. It was a mess!

In Focus: Receptacle Level Load Monitoring & Control

PTS Weighs in on Data Center Humidity Issues

Mark Fontecchio’s recent article on data center humidity issues at SearchDataCenter.com not only created buzz in the data center blogs, but generated quite a discussion amongst our team at PTS Data Center Solutions.

Data center humidity range too strict?

While some data center professionals find the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE)’s recommended relative humidity of 40% to 55% to be restrictive, I think the tight ASHRAE standards have to be adhered to until further research proves otherwise.

PTS’s engineering manager, Dave Admirand, PE notes that the reason the telecommunications industry is able to operate within a wider humidity range of 35% to 65% is because of their very strict grounding regimen. In a well grounded system an electrical charge has no place build up and is more readily dissipated to ground. Mr. Admirand recalls his days at IBM when the ‘old timers’ would swear by wearing leather soled shoes (conductive enough to make connectivity to the grounded raised floor) and/or washing their hands (presumably to carry dampness with them) prior to entering their data centers to avoid a shock from discharging the build up on themselves onto a surface.

Relative humidity vs. absolute humidity

While I think both relative and absolute humidity should be considered, many in the industry are still designing to and measuring relative humidity. PTS mechanical engineer John Lin, PhD points out that only two values of air are independent and data center professionals have to control the air temperature. While we can only control one of the humidity values, it is possible to calculate the absolute humidity (humidity ratio) based on air temperature and relative humidity. Therefore, data centers are fine as long as the both temperature and relative humidity are within the permissible range.

Coy Stine’s example is right on the mark. The high temperature delta between inlet and outlet air that can be realized in some dense IT equipment may lead to some very low humidity air inside critical electronics which can lead to electrostatic discharge (ESD). My experience, however, is that I am not encountering data loss scenarios at the estimated 50-100 data centers I visit each year simply due to ESD concerns. This leads me to believe that there is a slight tendency to ‘make mountains over mole hills’ regarding ESD.

After further reflection on Stine’s scenario about the low relative humidity air at the back of the servers, I was reminded again by Mr. Admirand that it won’t much make a difference since that air is being discharged back to the CRAC equipment. Furthermore, even if the air is re-circulated back to the critical load’s inlet the absolute moisture content of the air remains constant and the mixed air temperature is not low enough to cause a problem. John Lin contends this is the reason why we only control temperature and relative humidity.

It’s been our stance at PTS that the most important goal of humidity control is to regulate condensation. The only real danger to very warm, high moisture content air is that it will condense easily should its temperature drop below the dew point temperature.

Separate data center humidity from cooling units?

I have no doubt that R. Stephen Spinazzola’s conclusion that it is cheaper to operate humidity control as a stand-alone air handler is on target. However, experience dictates the approach is an uphill sell since the savings are indirect, realized only as part of operational savings. The reality is that the upfront capital cost is greater to deploy these systems, especially in a smaller environment where it is harder to control humidity anyway.

Humidity control is very dependent on the environment for which you are designing a system. In a large data center, it is actually easier to do because most of the entire building is presumably data center controlled environment. However, for SMEs with tenant space computer rooms the idea of humidity control is much more difficult since it is dictated by the overall building humidity environment. At best, a computer room is a giant sponge – the question is whether you are gaining from or giving off water to the rest of the building.

The design and construction of a data center or computer room, including its cooling system, should meet the specific environmental needs of its equipment. For now, our approach at PTS Data Center Solutions has been to utilize humidity control within DX units for both small and large spaces. Conversely, we control humidity separately when deploying in-row, chilled water cooling techniques for higher density cooling applications in smaller sites.

Pete

Data Centers Go to Washington

In recent years, the power and cooling costs of the average data center have gone through the roof. Data centers rack up more than $3-billion in energy costs each year. That number is expected to rise dramatically within the next decade as more data centers are built. Adding to the energy drain are factors such as inefficient cooling systems, more powerful servers, and rising energy prices.

Pete

Data Center Contest: Successful treatment of Mayo Clinic kill switch

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Gary submitted this entry:

In 1994, I joined the Mayo Clinic Rochester, MN campus as a senior Systems Administrator. My first day was spent getting the guided tour of the data center facilities at the Clinic. We were touring the main data center. This facility housed the Clinic’s mainframe, and a host of other distributed enterprise systems. The day was getting rather late as we were approaching the end of the tour. The lateness of the day also marked the end of the shift for the Data Center operations staff.

Our tour group was being escorted through the command center on our way out to the main data center operations area. Located along the wall next to the data center entrance was a large red kill switch.

Looking back now, this reminds me of the big red “Easy Button” depicted in popular Staples’ television media adds. This kill switch when pressed engaged the emergency power shutdown for the entire data center. The kill switch was within easy reach from the entrance door. There was no protective encasement or other security feature over the kill switch. Somebody could accidentally trip and hit the switch, which would engage the emergency power shutdown.

This fact must have been on the mind of our guide as well. Our guide strategically positioned himself between the kill switch and the door entrance to prevent such an accident from occurring. After everyone safely assembled together on the production floor, our guide resumed the tour. Our group moved over to opposite end of the production floor to view new StorageTek tape silos. From this vantage point, we had a clear unobstructed line of sight to witness the events that followed.

Observing a data center power shutdown is really something to see. One minute, we were gazing at the swinging robotics arm contained within the StorageTek silo fetching tape cartridges. The next minute the only viewable lighting was coming from the emergency lights. In retrospect, the outage lasted only a few minutes. But during this time there were teams of programmers and other IT support specialists that flooded out onto the production floor. During the power outage, our tour guide remained calm and collected. During these events he kept the group together and out of everyone’s way.

We later learned at the end of our tour of the events that led up to the data center power shutdown. A developer had called the command center to request a hard reboot on one of the servers in the command center. Confused, the operator asked the developer where the server was located. The developer instructed the unfortunate operator to look for a red power switch on the server located next to the wall. The operator saw one red switch. The rest was history.

The next day when I was out on the production floor doing server maintenance I noticed there was a plastic security encasement covering that one particular red switch!!!

In Focus:Data Center Site Selection

For companies seeking to develop a new data center facility, high-quality site selection is of the utmost importance. By choosing a site location wisely, companies can save both time and money, while achieving scalability, flexibility and high availability. Choosing a site that minimizes the natural and man made threats to continuous operation is the first step in provisioning a new data center. There are many factors to consider, including:

• Natural Hazard Threats
• Physical Location Threats
• Terrorist Activity Threats
• Environmental Contamination Threats
• Site Accessibility
• Amenities Access

It is interesting to note that the priority level of these factors is highly changeable. For instance, a decade ago it would have been more common for companies to seek site locations that with close proximity to major cities and airports. However, in the wake of September 11^th, data centers are more likely to spring up in smaller cities, reducing the likelihood of damage from terrorist attacks, but most especially in those areas of the country that have the lowest operation costs including utility rates, land acquisition costs, labor rates, tax rates, and cost-of-living expenses. To help navigate the complex process of site selection, many companies employ data center consultants for assistance in selecting an appropriate geography on which to locate their data center. Site selection services are the optimal way to ensure your mission critical facility is set up in both a location and a building that can support constant availability.

Pete

Data Center Contest: Disaster recovery using non-Microsoft “Windows”

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

From a Member who wishes to remain anonymous:

My first job in IT (at the time it was just called Data Processing) was in the early 80′s at a community college that I had just graduated from with an Associates in computer programming. I was hired as an operator for an IBM system 38 that was housed in a large room on the first floor of the school with a cubicle in the corner, which served as my office and of the programmer who had been there for 2 years before me.

Anyway, our back up system consisted of a weekly ritual of saving the most important files to diskettes housed in magazines that held 10 diskettes each. The normal weekly back up used about 10 magazines. The monthly backup was of the whole system, which used about twice as many magazines. It was cumbersome, but that was my main job since I worked 2nd shift. After the back up finished, I had to place each magazine on a bookshelf that sat in the corner.

The bad part was our disaster recovery system, for which we did drills for once a month. When the alarm went off, the programmer had to run outside and stand outside the window to our office. I then had to power down the computer and take each magazine of diskettes and toss it out the window to her until all of them were out. Only then could I “save” myself and join everyone else outside. The programmer then had to take the magazines and put them in her car and drive off site. Needless to say, all of the students and faculty got a big kick out of our little ritual.

Well, wouldn’t you know it, an actual fire broke out at the school one day and it just so happened to be the day the programmer wasn’t there. My boss, the data processing manager, had to run down the hall and take her place. Everything went without a hitch since the fire was in a part of the building far away from the computer room. After I threw all of the magazines out the window, I proceeded outside where I noticed that my boss had already put them in his car and was driving away from the building.

After all of the hoopla died down and the minor fire was put out, we were allowed back inside. I asked my boss when he returned where the magazines were and he informed me not to worry about them, he would just keep them in his car and for me to initialize and use new diskettes for that night’s backup, which I did.

Well, a week later while preparing for the monthly back up, I had to go to my boss and ask for the magazines since I didn’t have enough to do the whole back up. It was then that he informed me in a small voice that he had forgotten to take them out of the back of his trunk, and the diskettes had all warped and melted from the heat.

Needless to say, he then authorized me to go and buy new diskettes and magazines and made me promise not to tell anyone what happened. Also, soon afterward, he hired a company to house all of our back ups off site and we adopted a rotation system for our backups as well as purchasing a tape drive.

Data Center Contest: Last one out, turn off the lights

We’re gathering stories and photos for a cool contest sponsored by American Power Conversion (APC). First place in our Data Center Contest wins a Nintendo Wii game system. And we’re giving away twenty (20) copies of System Specifications and Project Manual for Data Centers (a $250 value!) to members just for entering the contest with a valid entry. Enjoy the contest entry below — and feel free to comment.

Community Manager

Adrian submitted this entry:

We’d just moved into a new data centre and had a Lights Out operation in this one area where we had a Tech replacing some part. We left him there for a while and when he left, he reached back to turn the lights off and instead hit the Emergency Power Off (EPO) button, dropping power to the entire area.

We quickly put covers over all the EPOs (like you see on a fire alarm) and labeled them more clearly.