Another 15 minutes

Got the new battery pack for my UPS now and the internal temperature seems to be low and stable so far, it’s been charging now for an hour. I’m glad this solved the problem because getting a new UPS would’ve cost alot more money since it was out of warranty.

When I got home from work I was expecting the network to be down, like described in my previous post today. But to my horror the entire rack was dead. The UPS had shutdown completely, I had to reconnect my bad batteries for it to start again. I contacted APC to check if this is normal behaviour when no batteries are connected for a longer period of time. The servers are now checking data integrity on the disks but I expect everything to be running again soon. I don’t like this situation with the UPS though, I sure hope APC will come through for me on this one.

Lately I had been starting to smell a strange smell of rotten eggs in my apartment that seemed to originate from where I have my rack of servers. I didn’t think too much about it, but today I got an automated warning email from my APC UPS that the internal temperature exceeded the warning range. It was over 65 degrees Celsius, normal temperature is about 30 degrees. Looking in the log I could see that it had been increasing slowly over the past few days. I’m guessing either the fan is broken or the thermostat is, hopefully it’s just the fan, because I have just ordered a new replacement battery pack that needed to be changed since the current battery is over three years old.

I decided to remove the current battery pack, just to see if there was something wrong with them and if I could see the fan behind the pack. When I tried to remove the batteries I noticed that two of the four batteries had expanded, not looking so good anymore. I don’t know if they expanded and then broke the UPS somehow, or if the fan failed and then the batteries expanded from the heat. Anyway, I will go without battery backup now until I get the new pack and also take a look at the fan and see if that fixes the problem.

Molten Lead-Acid UPS batteries smells like rotten eggs, who knew? :)