What is our primary use case?
UPSs are supporting critical load for data center colocation, and that is true across our national portfolio. We run real estate, but our real estate is "critical-load real estate." We've got data center floors that we lease space out to for IT companies. Those IT companies can be using a rack - a single equipment cabinet - or they can be 25,000 square feet of a data center hall for their enterprise. It's a mixed environment, although we try to specialize at each of our locations in a certain sweet spot. We also have shelf space that we lease to companies, large companies which come in and bring their own UPSs and run that space themselves. It's a good world to be in. I learned by watching some of our customers doing things that were mistakes. I get to learn from their mistakes.
In Denver, we've got four strings. Each string has four 750 kVA UPS modules. I'm using them in Cleveland as well. I've got three systems there, and each system has three 750 kVA modules.
Our organization has always had UPSs; that's part of the business that we do and we do rely on UPSs. Each of our locations, at some point during the year, switches over to UPS before we transition to generators. They are used actively every year. Our organization functions by supplying UPS.
How has it helped my organization?
I've got a critical-load bill from the utility here in Denver which runs about $130,000 monthly. That critical-load ESS mode moves us from 92 percent efficiency in the UPS to 99 percent efficiency in the UPS. It saves me a seven percent delta between my UPSs which are running pretty much all of that $130,000. It equates to about $11,000 monthly that it saves me if I'm in ESS.
Also, the inherent functionality of a UPS is that you're running your load on batteries. Utilities across the country these days are very dirty, more so on the brownout side than on the spike side, but computer equipment doesn't like unsteady power. So for our customers, it saves them a good amount of money in the failure rate from the equipment that's plugged into the UPSs because they're getting a steady load, steady power.
The way to look at it would be is if we did not have a UPS. If we had to switch over to generators from the utility power, there would be outages. Each one of our contracts with our customers has service level agreements that guarantee 100 percent power availability. Just considering our Denver facility, it has had 100 percent power availability since 1997. It's the integrity that the UPS provides the company that allows us to lease the space that we lease for colocation customers.
We use PredictPulse and the more information I can have from a predictive standpoint, the better I am at preparing my customers. When I talk about 100 percent availability, that is including any maintenance windows. Both our "A" and our "B" power sides are always 100 percent energized. When we do work, we transition from one system and put that load on the surviving system. It is a process. It is not something that can happen quickly, and it takes a lot of planning. In any kind of situation where I can get more information from the machine, telling me that something's going to need to be replaced before it actually fails, the greater the advantage I have in preparing the process for transition.
What is most valuable?
The most valuable feature is the control. We started with PredictPulse at the very beginning. Eaton could not get it to function properly. We got to a stage where we said, "We're just going to ignore PredictPulse because we can't get it to operate and you can't get it to operate." Eaton did step up and they got it to operate in our Denver facility, and now we couldn't live life without it. It does add a level of comfort knowing that PredictPulse is telling us about potential errors before they happen.
My preference is to keep it in ESS mode. It saves me a good amount of money. I've watched it work, flip back and forth with utility outages, and I've got complete comfort that the switching supports our critical load in a timely manner. We do drop out of ESS if there are severe storms in the area, or the utility is proving to be a little unreliable. That's simply because it concerns us when the UPS is switching back and forth so often. But it has been good.
The footprint is also good. I always want it smaller, but I always want to be able to have more room to do things inside it, so I definitely understand that give and take. The good thing is that it fits in the same space as my old UPSs. It is a little bit smaller, but it's a little bit larger than the Toshiba that we have. But size is not an issue in size, at least in my implementation.
What needs improvement?
I do like the ESS mode which I operate with the 9395s. However, I've got an issue with one of my UPS strings here in Denver: That ESS mode is too sensitive to utility noise. The utility shows no outages but I've got the UPS switching it in and out of ESS mode. That's a little bit aggravating and it's a concern that Eaton hasn't been able to remedy that yet. I don't know if that's Eaton or my manager who is dragging their feet.
I like the touch screen. I don't like the fact that they've decreased the size of the touch screen. I'm actively installing a new 9395 system right now. It's operating, but the commission is in two weeks. These new modules came in, and I found that they had moved from a larger ten-inch screen to a seven-inch screen, which is a big reduction in size. I didn't like the fact that they said, "Well, we can give you the larger screen, but it's going to cost you more money because these are being phased out." This was at the time that a company was saying, "Let's partner," and we had been partnered together for four or five years. There were two issues: Number one is that I don't like that we found it out once the piece of equipment was delivered to us, that we weren't told beforehand that that change had taken place. And number two is that they then wanted to charge me $27,000 to go to the screen that's on all my other UPSs. I voiced my frustration about that and their response was, "Yeah, they're being phased out."
Regarding the integration between the UPS and the batteries, I would say the same thing about any UPS manufacturer right now. I'm not sure if Eaton has started using lithium-ion, but it's almost like you have a UPS sitting there which feeds a string of UPS batteries, and then you get a battery monitoring system, which is another item. Although right now it doesn't hamper our operation, I would like for Eaton to figure out a way to integrate the batteries more seamlessly into a UPS system.
I definitely selected the Eaton UPS based on the features and their capabilities. They're continuing down the road with that. They produced a Unity system and there's no power factor within the system, so that's a plus. I know they've gone to a one-megawatt system. I would like for them to get a larger system in the future and I know they're working on that. I would like the system to be able to support maybe 1.5 to 1.8 megawatts per UPS string. I don't know what interval they're working on. They've moved up, I think, to a 1.1 meg from the 750s. There's a very small market for that, but we've got facilities where we used to operate at one or two megawatts and now they're operating at five or ten megawatts. The power requirements for the new IT systems are a lot higher.
For how long have I used the solution?
In our Denver facility, we installed them in 2014 - 2015.
What do I think about the stability of the solution?
My impressions of the stability of Eaton UPSs has been good.
I've installed six Eaton 9395s. Three of them had failures when they were one week old. The good part about that is all three of them happened before we had critical load on them. It does seem like they need to be tested a little bit more at the factory.
When I did a factory witness test in Raleigh for this last one, the problem was that they were getting fan alarms on the maintenance panels. When I asked about the fan alarms, the factory engineer told me, "Well yeah, that's because they've gone with cheaper fans, and the cheaper fans don't have a high enough speed to keep from having a fan alarm." I said, "Well, this test has failed, and I won't accept this unless you replace all the fans." They did. But the thought of them waiting for a customer to catch that just aggravated me.
I'm a proponent of Eaton. I like their technology, but I think they're trying to cut corners now and they need to stop it.
What do I think about the scalability of the solution?
They scale well. We use the 750 interval.
I do have considerable information for the smaller modular systems which are great systems. I haven't found an application for them, but when I do I would love to have the opportunity to implement one. That's a valuable aspect, where you can actually do plug-and-play. You could expand a UPS system without taking down your primary load.
How are customer service and technical support?
I've contacted first- second-, and third-tier, tech support - all of them. I have had some issues with their first tier. I've actually had facilities managers contact them to put in four-hour-response service calls and they've been asked for a credit card so that they could charge the $800 service fee upfront. I didn't take it very well and I did escalate that and it did get taken care of. But that's a little bit of a slap in the face when you're spending, five or six million dollars a year with a company, even though we're a little fish.
There all those kinds of issues. We've got 13 locations and I've got facilities managers who may call in for support and in normal situations, their tech support is very good.
I'm having a little bit of issue because there's a little gap between warranty and annual support. I've got systems that are coming off of a three-year warranty and I'm trying to get Eaton to start putting us in a rotation for annual maintenance. Eaton doesn't do that on their own, which seems odd to me because that's money for them. So we're still trying to get those transitions to happen. It's a little quirky thing, but moving from a warranty period over to putting the machines on regular, paid, annual maintenance is not smooth.
Which solution did I use previously and why did I switch?
I still have two Toshiba 9000 series UPSs in our Phoenix data center.
Eaton UPSs are inherently efficient. I replaced Liebert 500 and 600 series UPSs with these. The Liebert UPSs were about 40 percent efficient at the load level that we were at. These Eatons, even worst case, moved it to 90 percent. Toshiba would have saved me money. Any of the newer UPSs would have saved similar amounts. Just moving to a newer UPS saved me lots of money.
How was the initial setup?
The setup is complicated, more than it needs to be, by electricians who are charging you more than they need to charge you.
However, from a manufacturing standpoint, the actual setup and install of the UPS is fairly simplistic. It's a box you need to plug into power, and then you plug critical load into it, and you plug batteries into it. When I install a UPS, I judge by the cost to install versus the product cost. That's kind of a double-edge sword because I try to get my product costs down real low. But when the cost to install starts ballooning over the cost of the product, then to me, the install is too complicated. In a greenfield situation with an Eaton UPS, it's probably 25 to 30 percent of the cost of the UPS. But where I'm installing and retrofitting existing data centers, it's probably 100 percent to 125 percent of the UPS. And that's because I'm mandating that, in the process of installing a UPS, they have to keep my critical-load up on something. So that does increase the cost. I don't think Eaton has much control over that, unless you're buying - and we do - the installation from Eaton. Even in that instance, it's still back to the electricians. They're just passing through the electrician's cost.
Deployment generally takes three to four months from the order to installation, if no additional switchgear is required. Switchgear has something like a ten-month waiting period. Once we have the Eaton unit it can usually be installed over a two- to three-week period, if the prep work has been done beforehand.
Our deployment strategy is specific to each location. Looking at Denver, each time we've replaced one in Denver we put the unit on order and we then sit down and design the way the load will be actively switched over to a different UPS. We may or may not require reconfiguration of switchgear. Then, when the units arrive, we use Eaton CSEs who come out and unpack and install the unit itself. At the same time, we've got electricians replacing or building the battery distribution breakers. Once Eaton does a field-start on the unit, then we will plan a third-party commissioner to come out and commission the unit under load.
In terms of the number of people involved in a setup, I've seen groups of eight to ten electricians at the peaks. Then it drops down and you're going to have one controls-person, Eaton, and four to six electricians. And there is the H5 person watching over them, whether they're an electrical manager, a facilities manager, or a project manager. It will just be one body from an H5 standpoint, usually, who's controlling what's happening.
Once it's deployed, it will usually be one person taking care of and scheduling maintenance. A number of our buildings have building engineering staff, so these are fed into a BMS (building management system). So the UPS would actually be feeding signals to a BMS, whether it be JCI, Metasys, or Honeywell EBI. The building engineers will get alarms based on that and, usually, we run a staff of one to two building engineers, 24/7 at our facilities. Those building engineers are not dedicated to the UPS, they're dedicated to the whole building. The mechanical portions of the building take some 80 percent of their time. If we get an alarm from a UPS it's usually a failure of the UPS or the utility.
What about the implementation team?
Our third-party commissioner for Denver is called Primary Integration. We've used a couple of other services but only one time. Primary Integration is the only one that we continue to use, that we're happy with their performance.
What was our ROI?
The UPSs are an expense that I've got to have, but I'm gaining the return in the revenue for colocation.
What's my experience with pricing, setup cost, and licensing?
The cost per unit, as a rough estimate, is $300,000 per UPS 750-block.
The only additional services would be the maintenance we do. I don't know the exact costs of that at each location, but at Cleveland or Denver I believe we're paying about 35,000 a year per maintenance contract.
Which other solutions did I evaluate?
We actually looked at the market and we liked the Eaton UPS from an efficiency standpoint. They were really close to the Mitsubishi or Toshiba UPSs. We talked with both manufacturers' support teams and we like the closeness of Eaton's manufacturing to their technical team - the way they're entwined - so that they experience lessons-learned a lot quicker than an outside, third-party team that would be doing the support and feeding back to the factory. Eaton's service and engineers were one of the deciding factors for us.
I believe that Eaton is starting to lose some of its expert field-team components, so I am a little concerned currently, but that's just been in the last year. All of their key, field support team engineers and technicians are leaving. When they leave, it's kind of a big gap.
Toshiba was a little less expensive. The Toshiba had, I thought, a little better meantime between failure but, again, my concern was in maintenance and keeping the Toshiba current. That is a con, but on the other side is the fact that the Toshiba could readily be supported by a third-party, so maintenance going forward had a tendency to be a little cheaper.
Quality of maintenance is always a concern. With a third-party, it's a larger concern. Eaton really took care of all my concerns early on with the quality of their maintenance. That was good since I couldn't go anywhere else for maintenance.
The fact that I'm now seeing some of their field engineering diminish in quality - and don't get me wrong, I've not experienced any impact, but I'm seeing that there's a risk - creates a concern because there is no third-party I can go to. Where, with Toshiba, if I don't like what's happening to a third-party maintenance vendor, I can go to another third-party maintenance vendor who's supporting Toshiba.
What other advice do I have?
You've got to design it well into your system, as with any UPS. The UPS can't do things that the switchgear doesn't give it the availability to do. Also, it's only as good as the last maintenance that was performed on it.
The biggest thing, as far as the UPS units themselves goes, is that people have a tendency to forget about the UPS units. They're always on standby and nobody thinks about them until there's actually an outage. The good part about the UPS is it's an active component in the data center stream, in what we deliver to our colocation customers. The big component is the flexibility of a UPS to switch load from module to module; seamlessly is the biggest positive. We forget that sometimes when it's just sitting there humming.
The lessons I've learned are around attention to detail. Make sure, when you open your door to let somebody in to work on batteries for you, that you know who that person is, that the person has not been working on batteries for only six months and before that they were working at the McDonald's. Most of the issues are caused by lack of process, which is aggravated by lack of expertise in people.
I'm not using Eaton's lithium-ion batteries. I'm waiting for the next battery model. I'm concerned on a number of levels with lithium-ion, more from a regulatory standpoint. I believe that once lithium-ion proliferates within the industry, that the fire code is going to change terribly. Unfortunately, I believe it's going to affect me whether I go with lithium-ion or not. But if we slow the distribution of lithium-ion, it'll at least slow down the regulatory requirements for all batteries in data centers. I always try to make a five-year investment in batteries. The battery technology is changing quickly and in another five years there'll be something that either rivals lithium-ion, at least, or that is a lot better. Lithium-ion is going to be a lot more controlled in the future. Thermal runaway has always been a problem in data centers, even though lead-acid combatted it a lot better. Lithium-ion still has the problems, and that's what causes them to burn up.
Eaton has a product called Foreseer. Foreseer is a wonderful product that can be purchased as an add-on. Eaton doesn't communicate about or advertise that product very well. That product does supply an unlimited amount of information about the electrical systems tied into the UPS. That information is vital to me, but you don't know about any of that information when you purchase an Eaton UPS. I found out more about it from a third-party distributor. You buy Foreseer from Eaton and then they integrate Foreseer into your system. At that time I thought, "That's a wonderful system and it's probably one of the top two in the market," but Eaton doesn't sell it well.
As a data center operator, my highest efficiency is running the UPS as heavily loaded as I can get it. The problem today is that I've got an A and a B side, and I need 100 percent failover capability. So the best I can achieve is about 50 percent load, if I don't oversubscribe. Am I thinking about oversubscribing my UPS? The answer would be is "yes," I always think about it, but customers don't like me to do that. We're looking to implement DSM over the portfolio. Right now it's spotty. With the implementation of DSM, we can comfortably oversubscribe and monitor that utilization more closely. With that implementation, I would increase the percentage of critical load on my UPS or the percentage of utilization. I would take it up to about 70 percent, but I wouldn't be comfortable taking it any higher than that.
I would rate Eaton UPSs at eight out of ten. A couple of items that I spoke about above are what brings it back to an eight. I would probably rate Eaton, as a whole, a little bit lower.
I'm rating it at eight because of the fact that none of the "infant mortality" happened under critical load. If it had happened under critical load, I probably would have rated it lower, but they've been lucky - or we've been lucky - so it gets an eight right now.
I do believe that the folks within Eaton all work extremely well in supporting me in servicing our company. That's what keeps me with Eaton. All UPSs are advancing now at about the same rate. An Eaton or a Toshiba or a Mitsubishi or even Schneider all offer similar qualities. It's really the Eaton teams that keep me with Eaton right now.