W3WN
03-15-2016, 10:23 AM
So I come back to my desk from lunch yesterday, a little early after hearing from someone that "the whole 3rd floor, all of the computers are down" to find my boss and KB3ERQ in our server room, door open, doors off the rack cabinets and out leaning on the wall, and a server out of the rack and leaning by my desk.
Not a good sign.
I then see that there's a wet spot, slowly getting bigger, at the edge of the server that's against the carpet.
Definitely not a good sign.
Yeah, we had an outage for a few hours yesterday, lost a switch and a server, at least for the moment. The server has a redundant backup that's picked up the slack, but the new switch configuration wasn't in place yet...
What happened? Well, let this be a warning to anyone of you guys with a server room that you're at least partially responsible for...
We have an AC unit in the server room; it's designed to fit between some of the racks and "blend in." It has a drain for excess water condensation, basically a small hose with a pump attached. In our case, this hose exits from the top of the AC unit, into the ceiling and from there is piped to a drain.
Haven't had an issue with it for years. Until yesterday.
Seems that last week, someone was in doing the semi-annual fire safety inspection. Normally no big deal. But, this time, when whomever it was lifted the ceiling tiles to inspect the "dead space" between the drop ceiling & real ceiling, he dislodged the drain tube.
So, water starts pooling, and looking for an exit. Preferably down. Finds some of the cables routed up that way, starts heading down... and at some point, some of it enters the back of the switch, some of it enters the back of the server. (We figure that where the cables had an change in angle, the water simply dripped... dripped.... dripped down to whatever device lay below it)
Now, the good news is, no permanent damage. Well, the switch & server may not come back, but they're being replaced. In terms of functionality, a minor disruption, but for the most part, most of the network users didn't miss a beat. (Just the majority of the one floor, and some cable rerouting took care of that). But it could have been worse, much worse. Nothing overtly shorted out, for one. And ownership gave the boss the green light to order whatever he needed, even if it turns out to be unnecessary -- they'd rather have another layer of redundancy, if that turns out to be the case.
New server & switch were supposed to be overnighted, but the nearest CDW warehouse didn't have them in stock, so we may not get them until tomorrow.
Of course, the switch that remained working... no one's checked it to see if anything is floating in it... yet... :shhh:
Not a good sign.
I then see that there's a wet spot, slowly getting bigger, at the edge of the server that's against the carpet.
Definitely not a good sign.
Yeah, we had an outage for a few hours yesterday, lost a switch and a server, at least for the moment. The server has a redundant backup that's picked up the slack, but the new switch configuration wasn't in place yet...
What happened? Well, let this be a warning to anyone of you guys with a server room that you're at least partially responsible for...
We have an AC unit in the server room; it's designed to fit between some of the racks and "blend in." It has a drain for excess water condensation, basically a small hose with a pump attached. In our case, this hose exits from the top of the AC unit, into the ceiling and from there is piped to a drain.
Haven't had an issue with it for years. Until yesterday.
Seems that last week, someone was in doing the semi-annual fire safety inspection. Normally no big deal. But, this time, when whomever it was lifted the ceiling tiles to inspect the "dead space" between the drop ceiling & real ceiling, he dislodged the drain tube.
So, water starts pooling, and looking for an exit. Preferably down. Finds some of the cables routed up that way, starts heading down... and at some point, some of it enters the back of the switch, some of it enters the back of the server. (We figure that where the cables had an change in angle, the water simply dripped... dripped.... dripped down to whatever device lay below it)
Now, the good news is, no permanent damage. Well, the switch & server may not come back, but they're being replaced. In terms of functionality, a minor disruption, but for the most part, most of the network users didn't miss a beat. (Just the majority of the one floor, and some cable rerouting took care of that). But it could have been worse, much worse. Nothing overtly shorted out, for one. And ownership gave the boss the green light to order whatever he needed, even if it turns out to be unnecessary -- they'd rather have another layer of redundancy, if that turns out to be the case.
New server & switch were supposed to be overnighted, but the nearest CDW warehouse didn't have them in stock, so we may not get them until tomorrow.
Of course, the switch that remained working... no one's checked it to see if anything is floating in it... yet... :shhh: