Wednesday, January 28, 2009

A trip I didn't enjoy.

A paradox. That's what it is. 

As a library systems person, I have always had the conviction that the library's computers must be kept running at all times. When I worked at the Thunder Bay Public Library many years ago, I sometimes suddenly woke up in the early hours of the morning, usually from the rumble of distant thunder. If lightning accompanied it, I would instinctively feel around for my clothes, and before the storm hit, I would be dressed and out the door, on my way to the workplace.

In those days, electric power supplies were not as sophisticated (for the money) as are today's versions. Even though our computer room circuits were protected by a massive Uninterruptible Power Supply, they were not certain to survive everything nature could throw at them. And in any case, the batteries on that UPS could only keep things running for perhaps a half hour at best.  Today's versions have software that can phone a designated person in the event of a failure. (Ever notice that phone service rarely goes down, while power is more vulnerable?) 

Our service probably could have included that extra cost feature, but did not, but in any case, it didn't matter. My instincts served me well. I was like a wolf in an earthquake zone

After the storms passed, I would be there to restart any gear that was down, and assess damages if any. In those cases when the event was prolonged, I might have to restore files from a backup tape.

Today, I work in a system that has a very large UPS in a separate computer room with environmental conditioning. There are about a dozen servers, and some other computers drawing power from the room's own service panel. Power interruptions are, thankfully, rare, and when they occur, the UPS generally gives us an hour to shut things down gracefully.

So why did the entire room shut down instantly yesterday, servers, services, UPS and all?

We decided to set up a brand new server that will take over the duties of one of the ones due to retire. I crawled under the counter to remove a small power supply (also known as a "wall wart") that powered a set of computer speakers, to make room for a power strip with six receptacles on it. I plugged it in, and then plugged in the power supply. As I pushed it home, I noticed a small blue spark. Instantly the room went dark and quiet, except for the emergency lighting and the light from the windows. HUH?

This was a case of the 25-cent engine part bringing down the plane. Presumably, a faulty power strip shorted out when the plug was inserted, and the main circuit breaker tripped. It was electrifying.

It took my boss and me a few minutes to find the solution, because, unlike many breakers whose handles flip to the half-open position, everything seemed unchanged. Finally, he toggled the main switch, and the lights came back. Half an hour later, we had the servers back up.

Fortunately, the incident occurred on a day when the first heavy snowstorm of winter had rolled over the area. Not many people were making it to the library in any case, so the main complaint was that users couldn't get on the internet for a while.

I'm no electrician, but I would suspect that the UPS basically protects the circuits from incoming events. When the wall wart shorted out the circuit, it did was not a power failure or a surge from the main supply. It was a problem from the circuit itself, flying back to the main breaker, which did its job. Don't try this at home, if you can.