ASHRAE and Green Grid team up to standardize data center energy monitoring protocols

I recently spoke with Roger Schmidt, distinguished IBM technologist and past chairman of The American Society of Heating Refrigerating and Air-conditioning Engineers (ASHRAE) Technical Committee 9.9 about ASHRAE’s newly official partnership with The Green Grid. The new agreement will open up the collaboration between the two groups, allowing them to write white papers and books together.

In fact, they’re collaborating on a book right now: Real-time power consumption measurements for data centers. Tahir Cader, a technical director for ISR Inc. is leading that effort. The plan is to publish this book through ASHRAE as a joint ASHRAE and Green Grid publication. You can expect this book to be available first quarter 2009.

“This is a really important book for data center operators,” Schmidt said. “It will help them understand where energy is being used in their data centers. What are the tools, where are the measurement points?” This book will explain how to calculate PUE and how to measure the various elements in the data center: pumps, chillers, cooling towers, and UPS systems. It will also explain how to measure humidity, lighting, and IT equipment load.

Other ASHRAE publications on the horizon
ASHRAE T.C. 9.9’s eighth book is in the final stages of the review process. It is called Particulate and gaseous contamination in datacom environments. Joe Prisco of IBM is leading that publication. TC 9.9 hopes to have it available for the book store at the ASHRAE winter conference Jan 24-28, 2009.

This book covers the nitty-gritty of gases and particulates that can contaminate data center equipment and cause failures. “What are the filters I can deploy? How do we get a handle on what is the acceptable limit of various gaseous contamination? Bromine, Chlorine, the sulfur contaminants — we don’t know what levels cause equipment failures,” Schmidt said. “We’ve had failure in the field because we’ve had gases in clients’ data centers. It does ruin equipment. You’ve got to be really careful.”

Two others publications are in the works: The extreme green data center and Green tips for datacom equipment centers. These publications won’t be ready until the summer conference 2009.

ASHRAE T.C. 9.9 and a new DOE data center workshop
ASHRAE T.C. 9.9 is also partnering with the Department of Energy to combine the DOE’s data center energy efficiency workshop with ASHRAE’s workshop in New York State, to create a national workshop by the end of the year, according to Schmidt. “It will be a one or two day event, sponsored by the DOE and ASHRAE. We would provide one of our datacom books for the workshop. This would be a great advertisement for TC 9.9.”

ASHRAE to integrate the new thermal envelop
The newly expanded environmental envelop approved in the last month for data center equipment will be part of an update to ASHRAE TC 9.9’s first book: Thermal Guidelines for Data Processing Environments. “That new envelop will allow people to use wider temperature and moisture range for datacom equipment — it’s really an important document,” Schmidt said.

New certification from ASHRAE, but does it matter?

The American Society of Heating, Refrigerating and Air-conditioning Engineers, better known in the industry as ASHRAE, has a new certification program focusing on facility operations and performance, something of which data center operators can take advantage.

But should they take advantage of it, and does it matter?

In a survey this year of SearchDataCenter.com readers, we found that almost half (47%) of the 579 data center IT and facility employees we questioned had “no certifications to date.” Furthermore, more than two-thirds said that certification has neither been a factor in hiring, promotion, nor a salary increase/bonus.

So then the question becomes: Why bother with certification?

Those that offer it - such as ASHRAE, Marist College’s Institute for Data Center Professionals, and APC’s Data Center University - claim that the certifications help data center pros keep up-to-date on what’s going on in the industry. Here’s a blurb from the IDCP site:

…the mission of the IDCP is to support the professionals responsible for and working in data centers by providing a variety of credit-bearing and non-credit classes appropriate for employee development and training.

And from Data Center University:

The changing nature of data centers‚ and the technology that impacts them‚ makes it even more critical that employees remain up to date on the current theories and best practices for issues around topics of power‚ cooling‚ management‚ security‚ and planning.

There is no question that there is a knowledge gap when it comes to finding comprehensive data center pros. Pete Sacco, president of PTS Data Center Solutions, told me that he has a difficult time finding people with the breadth of knowledge in both IT and facilities management that he can hire. It usually requires education - either in academia or in the workforce - of computer technology and engineering, and he said that not too many people out there have it. And even though going to data center conferences and events that groups like AFCOM, The Uptime Institute, The Green Grid and Gartner put on can help, it may not provide that level of detailed education that you need to solve problems in the field.

Many of the data center managers out there started as overall facilities managers, and are now taking on the task of handling energy-sucking data centers, which are completely different animals from handling the HVAC in your typical office environment.

I think that although certification isn’t popping up now as being important, it may in the future. And even if it doesn’t, the knowledge required to handle these data centers is and will continue to be important, especially if that knowledge becomes rarer in the future than it already is now.

Powering down your servers

I didn’t know this, but I guess yesterday was “Power IT Down Day.” Hewlett-Packard, Citrix and Intel joined forces to propose the idea of asking PC users to install power management software that can decrease the power use of their machines when not in use.

I found out about “Power IT Down Day” this morning from Ken Oestreich, the director of product marketing at Cassatt Corp., who wrote a blog post about “Power IT Down Day” in his Fountainhead blog yesterday. Oestreich wrote that, although it’s a good idea, Power IT Down Day is also “missing the boat” because its sponsors stress PC power management, but not server power management. He goes on:

At the time of this writing, the official website at HP showed over 2,700 participants, saving an estimated 35,000 KWh. But here’s a sobering statistic: At a recent Silicon Valley Leadership Group Energy Summit, Cassatt piloted Server power management software. The organization using the software operated a number of its own data centers — and the case study findings showed that if this software were used enterprise-wide, the annual savings could be 9,100KWh for this enterprise alone.

You’ll never guess what Cassatt does. That’s right! It makes server power management software.

But the fact remains that Oestreich has a point. In a report to Congress last year, the federal Environmental Protection Agency recommended server power management as one way to reduce data center energy levels. Other industry groups like The Green Grid and The Uptime Institute recommend the same.

The good news is that data centers are listening. In our own purchasing intentions survey last year, only 18% said they were using power-down features on their servers, with another 13% saying they planned to sometime last year. Those numbers have since jumped. According to our new survey results from this year, 31% have implemented power-down features, and another 22% said they plan to sometime this year.

So I guess data center managers have gotten the hint. Maybe “Power IT Down Day” doesn’t have to be extended after all…

Microsoft expected to announce Des Moines data center location tomorrow

According to the Des Moines Register, Microsoft is expected to officially announce its plan to build its newest data center there, specifically in West Des Moines.

Microsoft had already said they were looking in the area for a suitable data center location. It is expected to compare in size to one Microsoft is building in San Antonio, Texas, and employ about 75 high-tech workers at about $70,000 a pop.

Iowa has become a popular place lately for data center facilities, with Google already there building in Council Bluffs, which is right next to Omaha, Neb., about 120 miles west of Des Moines (which in turn is about 300 miles west of Chicago, for those unfamiliar with the Midwest). Why is it popular?

1. Cheap land
2. Cheap electricity
3. (Relatively) few natural disasters
4. Plenty of colleges from which to draw a workforce.

It also doesn’t hurt that Iowa is thirsty for data centers to the point of offering financial incentives to two of the largest companies in the country (Microsoft and Google) to go there. Do those incentives benefit both sides? Not everyone agrees.

Data center shift-change checklist from The Planet

Last week, Eric Bush, Data Center Operations Supervisor at hosting giant The Planet, posted a checklist for data center perimeter patrols.

The perimeter patrol is an integral part of our data center operations designed so our staff can constantly monitor and control the data center’s status and its operational readiness. Each patrol takes around an hour to perform and is a top-to-bottom inspection of our facility and server environment.

The list provides a great guideline for ensuring your data center’s health.

Link VIA Rich Miller’s Twitter feed.

Green Grid publishes data center utilization tool

The Green Grid has another tool for data centers to measure their companies’ efficiency, saying it could be a valuable way to communicate with upper management on the state of the data center.

The IT efficiency-focused group has published a new paper on the “productivity indicator.” Christian Belady, the principal power and cooling architect at Microsoft who was the driving force behind the PUE/DCIE metric, edited the paper and said it should be used as “a communication tool” between various members of a company – IT workers, data center facility folks and company executives.

“What this does is give you a quick visual of how you’re doing, especially if you’re communicating up to executives,” he said.

The paper suggests building a radial graph with five “spines,” with each spine representing a metric:

• DCIE
• Server utilization: The activity of the server processors relative to its maximum ability in the highest frequency state.
• Data center utilization: The amount of power drawn by the IT equipment relative to the actual capacity of the data center.
• Network utilization: Percentage of bandwidth used compared to bandwidth capacity.
• Storage utilization: The percentage of storage used compared to the overall storage capacity.

The paper doesn’t say how to come up with each of these numbers, but there are tools and software out there to get the data points for each of them (see the definition of storage utilization, for example). And if for some reason you are still trying to figure out how to measure network utilization, for example, you can still plot a graph using the productivity indicator, but with fewer spines. Here is a sample picture of a productivity indicator radial graph:

Belady and John Tuccillo, a Green Grid member from APC, said businesses can add target lines if they want as well. They could have targets for six months out, a year, and 18 months out. Companies can use it for whatever data center metrics they’re actually using. So if it’s not a pentagon, it might be a square or a triangle with four or three data points, respectively. In which case it might look like this:

Companies can also break down one of those categories, such as data center utilization, into a more detailed radial graph all its own, such as this:

They emphasized that this isn’t something that companies should use to compare to other companies. Instead, it’s a way for businesses to realize their existing energy situations and set target goals for themselves down the road.

“Different companies have different risk thresholds. A business may say, ‘You know what? My storage utilization, because of my business plan, should only be at 50%,’” Tucillo said. “One of the strengths of this tool is that it allows for the end user to weigh the spines to what their business practice is.”

Core NAP develops homegrown data center thermal mapping tool

Jeremy Porter, Senior Internet Data Center Architect at data center and managed services provider Core NAP, has developed a very low-cost thermal monitoring system for the company’s newest data centers. Porter has bussed a system of low-voltage thermal sensors together over Cat5 cable. The monitors report back to a database that can map data center temperatures in real time. Porter plans to be able to put multiple monitors in cabinets, under floors and in the cable runs above the racks. The sensors from Maxim IC report to USB readers plugged into Linux hosts. The hosts log data to a local Web server, and Core NAP plans to combine that info with Visio maps of the data center.

“We bid out the price to buy some of these thermal mapping products,” Porter said. “The systems start around $1,000 and cost around $100 per sensor. We’re able to deploy our system for well under $25 per sensor, including bus and reader. The software is fully supported in the Linux kernel so we don’t have to write any drivers. When I told management how much it would cost it didn’t take me long to get them to fund the project.”

Porter says Core NAP customers are interested in high-density server configurations, and modern blade servers can throw off hot-cold aisle set ups, so thermal mapping is critical to staying on top of customers’ density demands.

Key points to consider in the evolution of the PUE data center metric

Yesterday I blogged about the Uptime Institute’s criticisms of PUE, specifically that the Green Grid metrics whitepaper does not explicitly discuss the need to gather PUE data over time. Microsoft’s Christian Belady (one of the key developers of the PUE metric) responded with the following:

“Ken [Brill of The Uptime Institute] has some valid points, clearly there needs to be more clarity and refinement in the definition to make it a rock solid benchmark. PUE is a “living metric” that the industry and in particular the Green Grid is working. But with all of the issues that are in the process of being resolved, here are three basic facts:
1) All metrics can and will be gamed regardless of the crispness of their definition. Show me a metric and I can come up with a way to game it.
2) Companies that are measuring PUE are improving their PUE over time. So they are improving their efficiency
3) Companies that are not measuring, are likely not improving. So these companies will be at a competitive disadvantage.

Microsoft is a company that is measuring (since 2004) and improving our PUE benchmarking against ourselves. There is no reason any other company cannot do this. Comparison with other companies is useful but less important to us as long as we demonstrate continuous improvement in our PUE. We hope that the issues people have with PUE for external benchmarking will be cleaned up in time but we do not plan on waiting until then for continuous improvement in our own operations.”

Here are a few key points to consider in the ongoing evolution of PUE:

Gaming PUE is going to happen
A lot of data center providers have included PUE ratios in press releases lately, many of them incredibly low. Rich Miller at Data Center Knowledge says he’s seen it before. “That’s pretty much what happened with the Uptime Tier System, which set forth a four-tier rating system for data center reliability. Data centers began describing themselves as equivalent to ‘tier three-plus’ or even ‘tier five.’”

PUE will need to evolve into a dynamic quality control metric
Dave Ohara at GreenM3 has a great explanation of how data center pros should use PUE in a dynamic way. “What helped me to think of PUE as a dynamic number is to think of it as quality control metric. The quality of the electrical and mechanical systems and their operations over time are inputs into PUE. As load changes and servers will be turned off the variability of the power and cooling systems influence your PUE. So, PUE can now have a statistical range of operation given the conditions. This sounds familiar. It’s statistical process control.”

Standards and training needed on how and when to measure PUE
Data center managers getting started with a PUE measurement program need some guidance — where, when and how do you take the most meaningful measurements? Microsoft’s Mike Manos and Belady have put together an excellent PUE Strategy post on their blog, The Power of Software. This checklist takes PUE newbies from measuring by walking around with a clipboard to data center chargeback. The Uptime Institute’s Pitt Turner has a great webcast on how to measure PUE on UPS and PDU equipment. The next step will be to get everybody doing this in the same way — which is where ASHRAE TC 9.9 comes in. The organization supports PUE and announced plans to develop a publication that would standardize PUE measurement methodology in November 2007, but no word so far on the progress of that project.

Uptime warns data center pros against being benchmarked on PUE

Uptime Institute executive director Ken Brill warned panelists at an online seminar today to be wary of very low Power Usage Effectiveness (PUE) ratios touted by some data center operators. “If your management begins to benchmark you against someone else’s data center PUE, you need to be sure what you’re benchmarking against,” Brill said.

Brill said he’s seen companies talking about a PUE of 0.8 — which is physically impossible. “There is a lot of competitive manipulation and gaming going on,” Brill said. “Our network members are tired of being called in by management to explain why someone has a better PUE than they do.”

If you’re going to compare your PUE against another company, you need to know what the measurement means. “You need to know what they’re saying and what they’re not saying,” Brill said. “Are you going to include the lights and humidification system? If you’re using free cooling six months of the year, do you report your best PUE?”

Brill conceded that The Green Grid’s PUE whitepaper has gained traction in the industry, spurring more action and debate than any other efficiency effort so far. But Brill takes issue with the measurement’s use of the term “power”. According to Brill, the fundamental problem with PUE is that it’s a snapshot in time. Power by definition is a spot measurement, Brill said. Power over time is “energy”. So power is measured in kilowatts, energy is measured in kilowatt hours.

Proponents of PUE like Microsoft’s Christian Belady have advocated measuring PUE over time, but Brill said that is not expressed explicitly in the standard.

I think it’s a bit of a stretch to assume C-Level execs are even aware of PUE (let alone calling data center staff out on the carpet about it).

I recently wrote an article about a data center manager that made huge efficiency improvements at a massive facility, saving hundreds of thousands of dollars through engineering projects. I asked him what his CIO thought about the data center efficiency he was achieving, and he told me the CIO had no idea. He’d never actually met the CIO…

Nonetheless, Brill makes a very important point. The first goal of PUE is to make a ratio to improve on internally. But the larger goal is to use the metric to compare data centers — as a benchmark against competitors, or as a way to compare various configurations, geographical locations, and technologies. Without standardization, comparative measurements will be meaningless.

Are your executives measuring you against competitors’ PUE? We’d like to hear from you.

IT job slump not affecting data center facility managers?

There has been a significant slump in the IT job market recently. InfoWorld (via Slashdot) is reporting nearly 50,000 IT jobs lost in the last year. The Wall Street Journal, Gartner, and others are all reporting that tech-spending is slowing, and a slumping economy means less hiring. Systems and network admins, helpdesk employees and other IT workers are reporting hard times on the U.S. job market.

“We have not seen extreme measures being taken by IT organizations, such as hiring freezes, but we do expect to see enterprises take a more conservative and ‘wait-and-see’ approach to staffing for the rest of 2008,” said Gartner research vice president Lily Mok in a recent report.

Nonetheless, data center facility manager jobs are still in high demand. The New York Times reported on it recently and I’m still getting emails from Google’s recruiters asking if I know anybody looking for a job on with the Google facility engineering team.

So what gives? Is there a data center facilities job gap on the horizon? Is working knowledge of Ohm’s Law and computational fluid dynamics protection against a down economy?