Question

Damaged UPS protected equipment

Data center infrastructure, Data center power supplies, UPS

We had a downstream UPS battery drain outage related to improper installation of a new upstream UPS. This resulted in UPS power failure when the batteries drained, after which multiple cameras and other devices did not work. All but one of these power supplies to these devices were fine. I cannot find any literature on what could have damaged protected electronics when the UPS shut down. Can you point me to any sources? We would like to keep this from happening again so I need to understand the cause.

All I can really do is speculate, because I don’t know what in the power supply actually failed, and because there are several possible power anomalies that could have occurred.
My first guess would be the same thing that causes most electronics failures – the inrush of power that occurs when the device is powered “On”. In other words, I have my doubts that it was anything about the actual UPS shutdown that damaged the power supply, particularly because only one was damaged out of how ever many you have. Power supplies are the least reliable parts of most electronics equipment. So my first suspicion would be that the power supply had a weakness to begin with, but that it didn’t show up until the device was re-energized, at which time the weak component failed with the initial inrush of current. There’s not much you can do to avoid this kind of problem. It’s just one of the things that happens with electronics, and is one of the big reasons we use UPS in the first place. Of course we want to maintain nice, smooth, clean power to our computer hardware, but an even bigger reason to use a UPS is to protect the equipment from the unstable inrush of current that occurs after a power failure.
When the utility restores power, there are usually several very short interruptions or fluctuations in the first thirty seconds or so. (You’ve probably seen lights blink a few times when utility power was restored.) These can be very damaging to electrical or electronics equipment. The UPS keeps these ups and downs and sudden voltage variations from getting to our hardware. However, whenever power is restored to a piece of equipment, no matter how stable the power source may be, there is a very short term “peak” draw or “inrush current”, and that is more often than not when hardware fails. This can occur whether the device is left on and re-powered as the UPS is restored, or if it was turned off and then turned back on after everything was stable. (The latter is still the best practice when power fails.)
There are other possibilities, but without knowing a lot more, I wouldn’t know if they were even logical. When your UPS failed, it should have gone into Automatic Static Bypass, which means that your equipment was operating on straight Utility Power until the UPS was restored. If you have good quality Transient Voltage Surge Supressors (TVSS) on your electrical system, correctly installed at the right locations, they should minimize your exposure to power line spikes, but some variations can still get through, and they can damage sensitive electronics. There is also a very small possibility that a surge or spike occurred during the Static Transfer itself, but that is improbably with a good UPS. If your UPS is a Line Interactive or Standby type, rather than a Full Double-Conversion design there is a possibility that power restoration anomalies crept through to the equipment. You don’t define the “improper installation” details of the new UPS, but if poor grounding or bonding was involved, then all bets are off as to what could have happened.
In short, I would have to say that your UPS should protect you in the future if is it now properly installed and grounded. (Few Data Centers are actually properly grounded, but that’s a bigger subject than I can go into here.) If you have not put high quality surge protection on your power panels, particularly so there is power protection when the UPS is in Bypass, you most certainly should, but make certain that whomever specifies and installs the protectors knows what they are doing. They don’t provide much protection if chosen or installed incorrectly.