Data center design

Multi-tiered data centers

Building a data center with one level of redundancy across the facility might seem foolish to some. Do your Web servers need to have the same uptime as your mainframe?

And expensive, too. Peter Gross, the vice president of critical facilities at Hewlett-Packard who was the CEO of EYP Mission Critical Facilities before HP bought them, said that a Tier 4 data center costs about $3,500 per square foot, while a Tier 1 costs $1,000 per square foot.

“With a 50,000-square-foot data center, the cost of a Tier 4 would be about $180 million,” he said. “But if you could do half at Tier 4 and half at Tier 2, it might cost $140 million. So you could save $40 million right there.”

That is the idea behind a new offering from HP and EYP - the ability to build multi-tiered data centers. Gross said the concept consists of building modular data center blocks, each with their own level of redundancy. You can connect them in different backup power configurations — for example N+1 or 2N — and thus can provision your data center in a more detailed way according to what suits your business.

Though HP is calling it a multi-tier specification, the redundancy definitions won’t have anything to do with The Uptime Institute’s tier classification system, which is the de facto standard on data center reliability. Gross said they’re just using the word “tiers” because that’s what data centers are used to hearing. Although using that word might confuse their potential customers, I think.

Gross also said that the redundancy levels will not be restricted to the four-level system from Uptime. Instead, a customer will tell HP EYP, for example, that it wants a 2% failure probability in the next five years for this portion of the data center, and a 7% failure probability in the next five years for that portion of the facility. Then HP EYP will go off and design a data center to that specification, and (presumably) draw up some kind of contract that solidifies it.

Tidbits from The Green Grid Technical Forum

The Green Grid Technical Forum continues today in San Jose. Yesterday was for members only; today it’s open to anyone. I spoke yesterday to two board members, John  Tuccillo from APC and Mark Monroe from Sun Microsystems. They updated me on some of the things going on with The Green Grid, which was formed a couple years ago to address the growing issue of data center power consumption. Here’s a rundown on a few of the details.

Data center design guide

Starting this year, The Green Grid will start putting together a data center design guide, something it’s calling Data Center 2.0. It’s aiming to release the first draft of the design guide in February 2010, a year from now. The guide will purportedly be a top-to-bottom look at how to build an efficient data center, looking at everything from design to construction to operations.

Data center Second Life

Tuccillo and Monroe also told me plans for The Green Grid Academy, which the group sees as a way to more easily disseminate information, metrics and tools that they’ve created. As Tuccillo said, “we need to embed deeper learning so the material can become second nature.” The concept: A user goes to The Green Grid Academy, gets themselves an avatar, and is then walked through a virtual world set up as a vendor-neutral data center. The user can then choose to take classes on various elements of operating a data center, such as Green Grid metrics.  It will be free. Right now it is available only to members, as the group wants them to vet it and find any bugs. They hope to have it ready for everyone in the first half of this year.

“This will give us a platform for end user type education,” Monroe said.

Making data center end user progress

When The Green Grid formed more than two years ago, it was mostly a conglomeration of vendors, and it took some heat because of that. Even now, the board of directors all come from vendor companies — AMD, APC, Dell, EMC, HP, IBM, Intel, Microsoft, and Sun.

But it has made some progress in bringing more end users into the fold. Last fall it formed an end-user advisory council, a 10-member group whose members include data center users from AT&T, British Telecom and eBay. Their job is to guide the board of directors so the focus of The Green Grid doesn’t lean too far on the IT supplier side. They’re involved in the data center design guide mentioned above, for example. And The Green Grid has been strict about who gets on the advisory council. A company like Microsoft could conceivably have a representative on there, since it’s one of the largest builders of data centers in the world. But The Green Grid determined that any IT vendor could not have representation on the end user council.

In addition, Tuccillo and Monroe said they’re growing their end-user ranks. The Green Grid has 200 members now, with about 18% or 36 members being solely end users.

Data center corrosion: Bring it on?

CHICAGO — Joseph Prisco, an IBM engineer, yesterday told ASHRAE TC 9.9 members that it might be a good idea to start monitoring data center pollution — particulate and gaseous contamination that could cause IT equipment corrosion.

In particular, Prisco singled out ionic chemical compounds such as sulfur and chlorine salts, as well as outside gases such as sulfur dioxide and hydrogen sulfide that can make their way into the data center. He said all data centers should start looking to prevent this data center pollution from happening, and that those facilities using airside economizers should definitely be in on the mix.

William Tschudi, a member of the Lawrence Berkeley National Laboratory, had some questions about that. LBNL did a study about two years ago that found no discernable damage to IT equipment from using outside air to run a data center. He asked Prisco yesterday whether equipment corrosion happened quickly enough to shorten the normal server refresh rate. Prisco said it depends on the environment.

Another angle: Who cares?

Christian Belady, the principal power and cooling architect at Microsoft, said he’s not that concerned about equipment corrosion. He said a better way to look at it is to expect the refresh rate to be short. That way you can replace the equipment, which tends to get more energy efficient with each iteration. So in this way, Belady was almost encouraging the data center pollution. He was saying that equipment corrosion could help overall data center efficiency.

It may be good to keep those comments in context. Microsoft buys tens of thousands of servers every year. Most of them are x86 servers. They’re relatively cheap and therefore disposable. In the case of more-expensive Unix and mainframe servers, it may be more prudent to keep an eye on the corrosive dusts and gases that can find their way into the equipment.

CRG West preps data centers for cloud computing customers

John Savageau was appointed CTO of the colocation firm CRG West back in September. We recently caught up with Savageau to discuss how the job is going so far and what trends he’s seeing in the data center market.

Tell me about your new role at CRG West so far.
Savageau: Since we’ve been rapidly growing our footprint, I spend half my time talking to customers, and half talking to our operations staff. As we continue to build out our data centers, we have a lot more flexibility to design to the 200-watt per square foot world we’re dealing with today for utility compute farms. The cloud computing community is rapidly growing. We’re building facilities to translate those requirements.

What role is cloud computing playing in your planning?
Savageau: I’ve been a grid fanatic for years since SETI @ Home, bringing large numbers of distributed CPUs together to solve problems. I think it’s important to encourage the growth of cloud computing into our data centers. Cloud computing is a marriage of grid computing and SaaS, and we had better be thinking about attracting cloud computing companies (or even doing it ourselves in the future).

The ability to have elastic computing as close to the non latency points as possible is important.

What’s the relationship between cloud computing and latency issues?
Savageau: For example, in New York or Chicago, we’d like to have the ability to create a zero latency cross-connection point, a cloud environment where trading companies can do business at an exchange point. If you’re looking at traders, latency and transactions are synonymous with loss of money. You have to talk about latency in fractions of milliseconds. Companies losing transactions are losing money.

Or in the entertainment industry, we’re talking about video on demand. The less hops that occur, the less delay you put in between the origination and the end user eyeballs, the better the experience. Lowering latency is going to be critical in a digital world.

How is cloud computing changing your data center infrastructure strategy?
Savageau: The mechanical side is easier to deal with than the switching side. It’s mostly dealing with the watts per square foot. If we look at what server deployments look like for cloud companies, we’re talking about putting in 25 racks of Verari blade servers, which will require us to have 250-amp, three phase power.

We’ve learned a lot over the past few years in regards to deploying high density rooms. Now we understand that 100 amps of 208v, three phase power is the mechanical design to meet most customer requirements. But building high density is build-to-suit.

Believe it or not, build-to-suit gives companies a really great opportunity to start thinking green. If you’re not thinking green you’re hemorrhaging money and having a negative impact on the environment. Thinking green on deployment, building the best efficiency is a religion with us now.

On the data center deployment side, strategies like cold aisle containment, extraction of heat into sealed plenums can be a huge factor in how much it costs for you and your customer.

How is cold aisle containment working out in your data centers?
Savageau: We’re doing cold aisle containment, in all of our new deployments. We looked at hot aisle containment, but we prefer to spend our energy cooling the intake side of the servers. The primary consideration is providing cool intake to servers.

When you walk into the data centers, the cold air is completely sealed. The server doesn’t care how much heat is coming in the back end. If you concentrate your efforts on the cold air you’re going to have a much happier server. Cold aisle containment can reduce the electrical draw of cooling systems up to 25%. That makes the customer really happy.

ViaWest data centers go green, which makes dollars and cents

We all know how big business operates: Each and every quarter, the bottom line must be met. This pressure on businesses creates short-term thinking and investing, less  risk taking, and enormous scrutiny of capital expenditures. All of this leads many companies to be hesitant to make big changes, even when “going green” is all the rage.

Recently I was heartened to find a company that is not as bothered with the short-term motivations of publicly traded companies, and for which going green is a meaningful proposition. ViaWest operates 16 data centers in five states (Colorado, Nevada, Oregon, Texas, and Utah), and as a private company, it can afford to invest in the long-term benefits of going green in data centers.

On a recent visit to the company’s Hillsboro, Ore. data center (formerly Fortix, acquired in 2006), I learned how passionate its staff is about sustainable operations. Jim Linkous, the vice president and general manager for the company’s Oregon data center, and Casey Vanderbeek, senior sales engineer, were happy to tell me about some of the company’s achievements in this area, as well as some future plans. The Hillsboro facility has earned Portland General Electric’s Gold status for using wind power to start with, but ViaWest has a comprehensive green initiatives plan that goes beyond buying sustainable power.

• Utilizing clean, renewable, wind power energy programs (Colo. - Windsource; Ore. - CleanWind; Utah - Blue Sky)
• Deployment of high efficiency cooling units which have yielded in excess of 50% in total energy savings
• Preference toward clean chilled water systems for cooling over direct expansion systems
• Through geographic planning, ViaWest has maximized the use of free cooling/ambient air opportunities
• Regular Thermal analysis and reviews to ensure effective cooling throughout the data center facilities
• Specialized Hot/Cold aisle management & ducted air return to optimize, and reduce the need for additional cooling units
• Strict recycling programs at all ViaWest facilities - Approximately one ton of cardboard is recycled in every 90 day period
• Utilization of high efficiency lighting throughout all ViaWest facilities
• New construction and expansion projects that promote the use of recycled building materials
• Server Virtulization stragey to enable long-term efficiency and decrease the average deployment size by 20%

For lack of a better term, they have a holistic approach to sustainability that others should take note of. The company’s green strategy isn’t a gimmick; it is part of their overall business strategy, and it’s paying off.

Founded in 1999, the company has state-of-the-art managed hosting and colocation facilities and continues to grow. In 2006 the company received a $31 million infusion of debt financing. This allowed the company to acquire Fortix among others. This investment has paid off, with increases in outsourcing to data centers over the past few years. ViaWest exceeded industry market growth predictions in 2007 and Linkous expressed confidence in a continuation of that growth in the coming year, despite an economic downturn. In fact, the company has already leased a nearby building at the Hillsboro campus that is being prepared with power supplies for future expansion to meet current and future customer needs. This has been spurred by the increased demand for managed hosting and colocation throughout the industry.

Along with successful growth the company continues to focus on implementing sustainable initiatives that may cost more at the outset but that are better for the environment and the bottom line over the long term. As the facilities and equipment age out, the company is upgrading the cooling systems with new, more efficient technologies. Senior Vice President of Sales Operations Steve Prather shared an example in which cooling units in the company’s Cornell, Colo,. facility were replaced with cooling towers that are 53% more efficient. The company has also taken advantage of free cooling where it is geographically appropriate.

Prather emphasized that ViaWest does not take a one-size-fits-all approach in its green initiatives throughout 16 data centers. Rather it evaluates facilities individually and optimizes efficiencies based on a location’s unique characteristics.

This stands in contrast to the company’s otherwise streamlined approach to facility management. Linkous emphasized that in large part the data center operations side of the house has been successful because of the leadership of its COO and cofounder, Nancy Phillips, who has firmly established company “best practices.” Linkous and Vanderbeek shared that these guidelines enable the facilities to run similarly smoothly, regardless of locale. Their evidence that this model is successful is the relatively low annual churn rate of ViaWest customers (less than 1%).

Whether it’s recycling cardboard (1 ton per 90 days), using windpower, taking advantage of virtualization technologies, or conducting regular thermal analysis in existing facilities, if it saves energy and resources, ViaWest is willing to spend the money to make it happen, knowing it will pay off over time.

I think that data center managers should take a look and consider if going green still really seems so hard. And if you’re at one of the “big guys” that is focused on quarterly profits and you just want to scream, make some noise, use ViaWest as an example of what you could be doing differently. At SearchDataCenter.com we like to point out these examples of green initiatives done right, because we know you might need some ammunition when you go before your board or CIO and request capital expenditures to make improvements. As always, email us with your green data center success stories.

While explaining a few of the facility’s features, Vanderbeek said,“A lot of going green is so simple: placing chillers properly, doing hot-aisle and cold-aisle containment right.” It really can be that simple. And it really can pay off.

Microsoft rolls out container data center strategy for cloud computing

Say goodbye to chillers and CRAC-units, say goodbye to raised floors and traditional disaster recovery. And say hello to the new paradigm, courtesy of Microsoft’s data center team.

Microsoft’s goal in 2008 was to shake up the data center community in a big way, starting with Mike Manos’ announcement at AFCOM that Microsoft would be deploying containerized data centers, to Christian Belady’s “Data center in a tent” experiment with a PUE of 1.0. Mission accomplished.

These guys are pushing the envelope like no one else in the industry — rabble rousing at ASHRAE TC 9.9 meetings, calling out vendors, and blogging about it every chance they get. They’re literally scaring people who have built their reputation and businesses on traditional data center design — and I don’t just mean the people selling chillers and raised flooring. These engineers are mad scientists, flipping their noses at decades of conventional wisdom.

You can read Microsoft’s proposal for yourself at Mike Manos’ blog, but the basic concept is this: data center trailers with minimal building envelop, using unconditioned outside air to cool servers. The servers will run on outside air with temperatures ranging 10-35 C and 20-80% relative humidity. “For this class of infrastructure, we eliminate generators, chillers, UPSs,” Belady wrote in the blog.

Here is a video:

Video: Microsoft Generation 4.0 Data Center Vision

The applications these servers are supporting have a built-in failover, Microsoft calls it “geo-redundancy”. If the server (or servers) die, the application automatically shifts over to another batch of servers, and Microsoft technicians replace the servers on a maintenance schedule.

For applications that demand higher redundancies, Microsoft will build more robust infrastructure. But thanks to its chargeback program, Microsoft’s business units will be less likely to adopt the more expensive higher-redundancy configurations if they can prove the bare bones approach works.

Microsoft doesn’t want you to put your data center in a tent. If you want to run big iron, have redundant components, pay big bucks for people to babysit your servers and keep them cool, that’s your business.

But they do want you to know that they plan to run all of their data centers at 1.125 PUE by 2012.

27 tips for good data center design

Last month, Techtarget held its Data Center Decisions conference in Chicago, and the second-day keynote was given by Ken Brill, the executive director and founder of The Uptime Institute. One of the things Brill said was that there are 27 points to a good hot/cold aisle design, and that most data centers only implement a handful of them.

So that got me to thinking: What are those 27 points? I got in touch with Robert “Dr. Bob” Sullivan, a staff scientist at Uptime that came up with the hot/cold aisle design back in the early 1990s. Earlier this year he wrote a paper on good data center design, and included 27 points. Not every one directly involves hot/cold aisle, but they’re all worth checking out. Aside from one general point, I’ve separated them into five groups: raised floor and overhead space, hot/cold aisle, power and cooling equipment, perforated tiles, and cabling.

Hopefully this can serve as a practical checklist for those users out there designing a new data center or retrofitting an old one.

It’s important to note that a lot of these points refer to a subfloor cooling environment, rather than overhead cooling. Here is the first general point, followed by the five groups:

1) Monitor and manage the critical parameters associated with equipment installation, by area of the computer room (no more than two building bays):

• Space: number of cabinets and rack unit space available vs. utilized
• Power: PDU output available vs. utilized
• Breaker positions: available vs. utilized
• Sensible/redundant cooling capacity available vs. utilized
• Floor loading: acceptable weight vs. installed cabinet and equipment weight plus dead load of floor and cables, plus live load of people working in area. Compare the actual floor load with the subfloor structural strength.

Raised floor and overhead space

2) Create a raised floor master plan

3) Establish minimum raised-floor height

• 24″ if the cabling is overhead, with no chilled water or condenser water pipes under the floor blocking the airflow
• Recommend 30-36″ if there are airflow blockages

4) Establish a minimum clearance of 3′ from the top of the cabinets to the ceiling

5) Seal all penetrations in the subfloor and perimeter walls under the raised floor and above the dropped ceiling

Hot/cold aisle

6) Install computer and infrastructure equipment cabinets in the cold aisle/hot aisle arrangement

• 14′ cold aisle to cold aisle separation with cabinets 42″ deep or less
• 16′ cold aisle to cold aisle separation with cabinets > 42″ to 48″ deep

7) Utilize proper spacing of the cold aisle

• 48″ wide with two full rows of tiles which can be removed
• All perforated tiles are only located in the cold aisle

8 ) Utilize proper spacing of the hot aisles

• Minimum 36″ with at least one row of tiles able to be removed
• Do not place perforated tiles on the hot aisles

9) Ensure cabinets are installed with the front face of the frame set on a tile seam in the cold aisle

10) Require cabinet door faces to have a minimum of 50% open perforation – 65% is better

11) Prevent internal hot air recirculation by sealing the front of cabinets with blanking plates, including empty areas in the equipment-mounting surface, between the mounting rails, and the edges of the cabinets (if necessary)

Power and cooling equipment

12) Put PDUs and remote power panels in line with computer equipment cabinet rows occupying cabinet positions

13) Place cooling units at the end of the equipment rows, aligned with hot aisles where possible

14) Face cooling units in the same direction — no “circle the wagons” aka, uniformly distributed cooling

15) Limit maximum cooling unit throw distance to 50′

16) Create appropriate cooling capacity, with redundancy, in each zone of the room (zone maximum is one to two building bays)

• Install minimum of two cooling units even if only one is needed
• Install one-in-six to one-in-eight redundant cooling units in larger areas

17) Use only sensible cooling at 72F/45%rh when calculating the capacity of cooling units

18) Place chilled or condenser water pipes in suppressed utility trenches if the computer room is built on grade

19) Ensure all cooling units are functioning properly

• Set points and sensitivities are consistent
• Return air sensors are in calibration – calibrate the calibrator
• Airflow volume is at a specific level
• Unit is functioning properly at return air conditions
• Unit produces at least 15 degree delta T at 100% capacity

20) Be sure the cooling unit’s blower motor is turned off if the throttling valve sticks (chilled water type units) or if a compressor fails (air conditioning type unit)

21) Adjust chilled water temperature to eliminate latent cooling

Perforated tiles

22) The maximum number of perforated tiles is the total cooling unit airflow divided by 750cfm = maximum number of perforated tiles to be installed

• Install only the number of perforated tiles necessary to cool the load being dissipated in the cabinet/rack in the area immediately adjacent to the perf tile
• Turn off cooling units that are not required by the heat load (except for redundant units)

23) Do not use perforated tile air flow dampers and remove all existing dampers from the bottom of perforated tiles (reduces maximum air flow by 1/3, the often close unexplainably and they potentially can produce zinc whiskers)

Cabling

24) Seal all cable cutouts and other openings in the raised floor with closures

25) Spread power cables out on the subfloor, preferably under the cold aisle to minimize airflow restrictions

26) If overhead cable racks are used, the racks should run parallel to the rows of racks. Crossover points between rows of racks should be located as far from the cooling units serving the area as practical

27) Place data cables in trays at the stringer level in the hot aisle