Drip, Drip, Drip...

[W3WN]

So I come back to my desk from lunch yesterday, a little early after hearing from someone that "the whole 3rd floor, all of the computers are down" to find my boss and KB3ERQ in our server room, door open, doors off the rack cabinets and out leaning on the wall, and a server out of the rack and leaning by my desk.

Not a good sign.

I then see that there's a wet spot, slowly getting bigger, at the edge of the server that's against the carpet.

Definitely not a good sign.

Yeah, we had an outage for a few hours yesterday, lost a switch and a server, at least for the moment. The server has a redundant backup that's picked up the slack, but the new switch configuration wasn't in place yet...

What happened? Well, let this be a warning to anyone of you guys with a server room that you're at least partially responsible for...

We have an AC unit in the server room; it's designed to fit between some of the racks and "blend in." It has a drain for excess water condensation, basically a small hose with a pump attached. In our case, this hose exits from the top of the AC unit, into the ceiling and from there is piped to a drain.

Haven't had an issue with it for years. Until yesterday.

Seems that last week, someone was in doing the semi-annual fire safety inspection. Normally no big deal. But, this time, when whomever it was lifted the ceiling tiles to inspect the "dead space" between the drop ceiling & real ceiling, he dislodged the drain tube.

So, water starts pooling, and looking for an exit. Preferably down. Finds some of the cables routed up that way, starts heading down... and at some point, some of it enters the back of the switch, some of it enters the back of the server. (We figure that where the cables had an change in angle, the water simply dripped... dripped.... dripped down to whatever device lay below it)

Now, the good news is, no permanent damage. Well, the switch & server may not come back, but they're being replaced. In terms of functionality, a minor disruption, but for the most part, most of the network users didn't miss a beat. (Just the majority of the one floor, and some cable rerouting took care of that). But it could have been worse, much worse. Nothing overtly shorted out, for one. And ownership gave the boss the green light to order whatever he needed, even if it turns out to be unnecessary -- they'd rather have another layer of redundancy, if that turns out to be the case.

New server & switch were supposed to be overnighted, but the nearest CDW warehouse didn't have them in stock, so we may not get them until tomorrow.

Of course, the switch that remained working... no one's checked it to see if anything is floating in it... yet...

[K4PIH]

Kind of the same situation where I work. The roof leaks due to all the penetration by pipes, hoses, vets. The building management says it's no problem they have drain pains above the suspended ceiling to catch the drips. Well that's all fine and dandy but someone has to empty the drain pans right? We know when they overflow because we can see the expanding yellow stain on the ceiling tiles. Since the building in managed by a commercial firm and we are not allowed to touch anything, we have to call them when we see the stains. They send Homer up, dumps the drain pans, installs new ceiling tiles, goes home. Last year it was bad enough that it leaked into the radio room and behind the surface of the sheetrock.

[kb2vxa]

Not far from here is an unattended remote packet node/BBS in a hangar loft jammed with radio equipment for the Navy, government alphabet agencies and the town's fire repeater. Every so often the packet station goes down and always for the same reason, some tech tripped over the power cord to the rack and pulled the plug out of the wall. Then the club has to send the techs through all the post 9-11 security and escorted to the rack to plug it back in. The problem could be easily fixed by changing the plug and socket to a twist lock type, but nothing could be that simple at Lakehurst Naval Air Station.
"Since the building in managed by a commercial firm and we are not allowed to touch anything..."
Close, the socket is Navy property and tampering with government property is a federal crime.
"It burst into flames! It burst into flames, and it's falling, it's crashing! Oh, the humanity....... This is the worst thing I've ever witnessed."
That's right Herb, trying to untangle the Gordian Knot of miles and miles of government red tape is the worst thing I have ever witnessed when a company I worked for had to get some test parameters changed when I found the error that caused 70% red tag returns for "an unknown electrical problem".

Don't fix the problem, just put pans under it. Once upon a time there was an upstairs enclosed porch with a roof that had a big hole in it, below was an enclosed porch with the landlord's office in it. Upstairs lived a friend who put buckets under the hole. I asked him if he would like to see the hole get fixed, he said yes, I said get rid of the buckets. He did and before long there was a storm, one hell of a downpour that went on for hours. I don't have to fill in all the details, a week later a crew ripped everything off down to the rafters and put on a new roof.

[KG4CGC]

Kind of the same situation where I work. The roof leaks due to all the penetration by pipes, hoses, vets. The building management says it's no problem they have drain pains above the suspended ceiling to catch the drips. Well that's all fine and dandy but someone has to empty the drain pans right? We know when they overflow because we can see the expanding yellow stain on the ceiling tiles. Since the building in managed by a commercial firm and we are not allowed to touch anything, we have to call them when we see the stains. They send Homer up, dumps the drain pans, installs new ceiling tiles, goes home. Last year it was bad enough that it leaked into the radio room and behind the surface of the sheetrock.

Tell'im e's breeding mold and mosquitoes.

[NM5TF]

Tell'im e's breeding mold and mosquitoes.

make sure you tell 'em it makes a fine breeding area for Zika Virus mosquitoes.....and then call the CDC.....

[K4PIH]

Tell'im e's breeding mold and mosquitoes.

Good idea! We have anther small shed 8x8 that houses some of our repeater gear. The roof leaks down into the walls so we told the owners (Navy) to fix it due to the mold growing on the walls. Their solution was to nail up plastic wallboard. like you see in gas station bathrooms. The mold is growing so thick it's starting to push the wallboard out! I went in the shed back in October and got sick. Not ever going back in again.

I like the Zika virus thing too!

[kb2vxa]

I see you don't understand the difference between getting rid of buckets and getting rid of mold so here it is in a nut shell. The landlord had a nice apartment, the rest of the building was a slum and he didn't fix anything that didn't affect him directly. When rain poured down destroying his computer and filing cabinets full of records he fixed the roof. The leaky roof where K4PIH works doesn't affect management, they don't live there. The leaky roof over the repeater gear doesn't affect the Navy, they don't live there either.

[n6hcm]

this sort of nonsense affected about 1/3 of my time at the VA ... no joke. leaky roof, shitty inadequate ac, shitty adequate ac that leaked everywhere. do not miss this at all.

[W3WN]

Well, since the thread has shifted to the Mold & Mildew of the Month Club...

My club last fall made a major investment to one of the repeater shacks. Currently owned by a club member (but heavily used by the club, so we chip in on power and maintenance costs), the previous (original) owner had built the shack out of shipping crates. No, really. A 2x4 frame, shipping crate wood for the walls, no outside weatherproofing beyond paint, and what looked like shingles left over from a couple of roofing jobs on the roof.

To say that it had a mold issue... well, it wasn't out of hand, but it was getting worse. Plus some rotting one, and since the floor was wood, the critters loved to chew threw it...

The good news is that the shack (and it really was a shack) got torn down last fall. New concrete base poured, new shack built on top of it... looks pretty nice. And no mold.