Data Center Co-Location Space Seekers Beware - Tales from the Data Center

Tales from the Data Center

Aug 22 2008   3:47PM GMT

Data Center Co-Location Space Seekers Beware



Posted by: Pete Sacco
DataCenter

This story was just shared with me today,

 ”Absolutely true story.

In a colocation in central Ohio, we rent a couple of racks. The primary rack there has two power sources. They’re not A/B, it’s all primary power, because the rack is /full/ of machines.

The colo staff was doing maintenance on something unrelated down in the floor and kicked one of our plugs out of the socket.

Now, if that isn’t bad enough, with half of our rack down, they went to plug it back in, and plugged it into the wrong circuit. 15amps + 15amps on a 20amp circuit is no one’s idea of a good time.

We’re moving out of that datacenter as soon as our contract is up. In the meantime, we’ve gone with a much higher class facility in upper NJ.”

My response to him is the one I share with all the victims. The sad truth is that the data center co-location and managed services industry is fraught with ‘buyer beware’ risks such as this story.

Since there are few standards and no regulation, it is up to the individual renting space in these facilities to perform their own investigations in comparing a data center operator’s claims to their actual capabilities.

PTS is hired by many clients to evaluate and compare data center spaces prior to relocating their equipment.

Today an IT manager doesn’t just need to be an IT expert, they need to be a facility expert too!

 - Pete Sacco

Comment on this Post


You must be logged-in to post a comment. Log-in/Register

Dave mc  |   Aug 26 2008   11:09PM GMT

I agree you need to be very careful in the selection of a hosting site, as a facility manager who often reviews data centres this is a selection of quick list items to watch out for.

It all comes down to money at the end of the day. A high quality fully redundant centre cost a lot more money in capital cost can even higher ratios for ongoing maintenance and upgrades. I see a lot of centres where the intentions were good in the original concepts but as return on investment issues come on line the cost cutter come in, then its “O” why did this fail ???.

Insufficient generator capacity (very common)

Only testing standby power off line not in actually transferring the computer operations to the generator. Generator testing is not just the engine but also all the switchgear, fuel systems & cooling etc.

Only testing standby power 2 to times per year. Generator testing only tests that it worked this time but regular testing gives you a more reliable trend and indicator of comfort the entire system is reliable.
Regular testing ensures the generator can handle the load and the transition and that the UPS systems function. Battery charging of the UPS often causes the required power to exceed the capacity.

2N or N+1 what does this really mean in relation to UPS systems and power feeds. If you cant turn the UPS off completely without going to maintenance bypass it is not a high level redundancy. It is not just failures but the ability to effectively perform maintenance and how long it takes to return to full operations should the failure occur.
We are seeing UPS providers saying we can just change modules no need to go to maintenance bypass, this is ok if every part is hot swappable with no single control card or other point of failure, but the larger units often are not.
Same applies to rack mounted UPS, can I turn it off and not effect the operations or reduce my level of operational security?, if you have duel core both on separate UPS system it is ok, if not you have a problem if that is the level of supply you require.
Same redundancy requirements apply to the cooling and water systems. If you cant turn entire units off for both repairs and maintenance for a long period to allow for parts and delays in repairs you have insufficient cooling.

STS units are interesting.
A lot of Static Transfer Switches are often not capable of performing the task in all situations especially in the higher level data centres where you have multiple UPS systems and multiple power reticulation systems that can supply the duel core racks.
The question to ask is CAN MY STS HANDLE OUT OF PHASE TRANSFERS. Most can not, if the phases are out of sync for more than a couple of degrees.
If you have a power failure to part of your site and the generator is supplying one UPS and either a different Generator or mains power on the other, (or situation where two separate supplies to the UPS system are not from the same source) the duel feeds will most likely not be in sync.
In this situation if the two feeds are out side the sync tolerance the STS will not transfer or the supply will fail then the STS transfers and all you know is you have had a very quick interruption.
And before the major manufactures jump up and down I have yet to only find two manufactures that can perform this task (one in Europe and one in Australia) Don’t argue with me with fancy statements prove me wrong I welcome it as an operator I need the diversity.

No I wont discuss names, I am not a supplier I am an operator with over 30 years experience and managed over 40 major data centres. I have seen this STS issue many times.
The trick in assessing a hosting data centre is to ensure your total data centre design is sufficient to your needs and then maintain it to the correct level. Not every one needs the higher level redundant centres, that is why different site have different levels of offerings and prices. Don’t just believe the mother hood statements if you need a critical site.


 

PTSacco  |   Aug 27 2008   7:30PM GMT

Dave,

Thanks for the comment.

However, I would like to offer a few comments in response to your STS concerns.

True, most STS’ have a limited range for out-of sync transfers. The type of load being served by the STS is key to how great a difference between power source phase angle and angular rotation at the load is acceptable.

Most STS’ will not transfer a load between two out-of-phase power sources because it may damage electronic equipment like variable frequency drives or chopper style power supplies. It will almost certainly cause physical harm to most motors and motor control circuits because the back EMF from the motor windings is not synchronized to the power source.

The key, as you pointed out was that the two or three sources that the STS can chose between need to be synchronized to the same standard. One way to accomplish this feat is to supply all the UPS bypass inputs from the same power source so that they are all synchronized within a small window of difference, (no two UPS’ are exactly the same). A second possibility is to use a controller that provides the same synchronizing signal to the sync control circuit of multiple UPS systems/modules. This will cause them all to operate within the synchronizing window of the STS’. However, the source for the sync signal must also be the power source for the UPS’ bypass input, or the UPS systems/modules will probably be in alarm with a “Bypass unavailable or Bypass out of sync” warning.

Finally, the engineer’s and designer’s points of view for reviewing a power distribution system must change from micro to macro if we intend to completely analyze a design. All the details from the smallest branch circuit to the electrical service feeders must be seen with respect to the intent of the system design. Redundancy is the key to providing reliable electrical power and high levels of availability. That redundancy needs to extend from the electrical service down to the branch circuits, (2 to 8 power cords – depending upon the type of IT equipment). In fact, one of the easiest means of providing redundant power supplies is to power critical equipment and the associated redundant equipment from different power supplies. This requires no synchronization, (dual corded servers, Chillers or A/C’s with multiple compressors, any system configured to operate in an (n+1) redundant fashion, etc.).

In any case, make sure you understand you load profile before transferring non-sync sources.

- Pete Sacco