Server Problem

[n2ize]

I have a Linux box (server) down in the basement that performs a multitude of services, i.,e. router, ftp, web, nameserver, proxy, database (MySQL) and lots lots more.... Earlier I noticed that it was refusing connections on most, if not all, of its services. I couldn't even gain remote access so I went into the cellar and noticed that it seemed to have crashed. So, I hit the big switch and tried a reboot. Nothing, the screen stayed blank and a repetitive clicking noise came from inside the machine. I said, "oh shit the hard drive just bought the farm". I shut it and again hit the switch, same problem. I shut it again, waited a few seconds and this time noticed some stuff came on screen but then again it just sat there clicking. I shut it again, waited another minute or two. This time the bios messages came up, it took longer to start clicking...then stopped and just froze. I let it sit for a few more minutes and then restarted. BINGO, no strange sounds, it did a clean and full boot and so far is still running fine.

So what is the verdict ? Overheating ? Seems like the more I let it rest the closer and closer it came to performing a normal boot. I am going to be working in the cellar tomorrow. I'm going to take it off line, give it a thorough cleaning, make sure the fans are running properly, do a full backup and install a new hard drive. The server basically chugs along 24/7/365 and frequently is up > 365 days with nary a reboot. It may be time for some preventative maintenance.

Oh, I did back up my important stuff, such as my database (.sql) files, etc.

[PA5COR]

Running all day long it might be dust clogging the heatsinks and fans causing overheating.
I clean out the pc's here every 3 months or so and you will be surprised aboout the dust that gathers in them from just a few hours use every day.
Hard disks that run continuously don't die that easy, most of them die by the spin up and cold warm cycles from switching the puters on/off, continious use doesn't cause that much thermal shock.

Keeping a good backup ready is a must, i do a backup every week with a USB H.D.
My laptop is on 12 to 16 hours a day and needs a good cleanout including opening up to get to the fans every 3 months minimum.

Hope you get it sorted, check connectors while you are at it, reseat memory etc.

[n2ize]

Yeah. So far its still running. But I'm going to pull it out of line later and give it a cleaning. It's been running constantly for about 2 years so it might be charged up with dust.

[KC2UGV]

Clicking?

That hard drive is on it's last leg. Go, buy a new HD, and clone the old to the new.

[n2ize]

Clicking?

That hard drive is on it's last leg. Go, buy a new HD, and clone the old to the new.
Any particular utilities that you would recommend for a full disc clone ? In the past I would usually use "tar" or "cp", or "dd".

[KC2UGV]

GnuPartEd, or Norton Ghost.

If you have another machine to slave them onto, then dd, although you might have issues with different drive sizes.

[n2ize]

GnuPartEd, or Norton Ghost.

If you have another machine to slave them onto, then dd, although you might have issues with different drive sizes.

Can't I simply mount the new HD in a usb adapter and once plugged into the server do a

# dd if=/dev/hda of=/dev/sdb

where /dev/hda is the old drive being cloned and /dev/sdb is the new drive mounted as a usb. Then once "dd" is finished doing its work unplug and remove the drive from the usb adapter and mount it directly in the computer and boot up ?? Since there is only 1 drive in the computer and it's a Linux only machine dd should effectively copy everything to the new drive including the mbr. Does this sound like it would work ?

As far as different drive sizes "dd" will be copying from a smaller drive (about 60 Gb) onto a larger drive (120 Gb) so I don;t think there shouild be any problems related to disk size ??

[KC2UGV]

Can't I simply mount the new HD in a usb adapter and once plugged into the server do a

# dd if=/dev/hda of=/dev/sdb

where /dev/hda is the old drive being cloned and /dev/sdb is the new drive mounted as a usb. Then once "dd" is finished doing its work unplug and remove the drive from the usb adapter and mount it directly in the computer and boot up ?? Since there is only 1 drive in the computer and it's a Linux only machine dd should effectively copy everything to the new drive including the mbr. Does this sound like it would work ?


Maybe. Sometimes, I've had issues doing it with live systems.



As far as different drive sizes "dd" will be copying from a smaller drive (about 60 Gb) onto a larger drive (120 Gb) so I don;t think there shouild be any problems related to disk size ??

Maybe. Just giving you a heads up, that's all. GPartEd resizes everything, and makes a new partition table for the new drive.

[n2ize]

Maybe. Sometimes, I've had issues doing it with live systems.



Maybe. Just giving you a heads up, that's all. GPartEd resizes everything, and makes a new partition table for the new drive.

Okay, I guess I can try GPartEd. If it makes life easier I am up for it. Does GParted do the entire cloning process ? Or do I need to use any additional utilities. I was under the impression that GPartEd was just for re sizing existing partitions ?

[n2ize]

Also, what about "Clonezilla" ?

[WØTKX]

Depending on the drive and it's firmware, you may be able to get it to stop using the bad spots. My laptop had an issue after it was dropped while running when I was in the hospital. I lost 3 MP3 files that I had backed up elsewhere. Scary, but it worked out. I had to boot via USB stick to run the utility, and it has been fine ever since, by setting the damaged area as "bad". The utility was manufacturer specific, Western Digital in this case. I did get a spare drive, but have not needed it... and check the running drive every now and then.

The damage happened as I was writing MP3 files from my Sansa player. Luckily, it caused no physical harm, and if I ever cared, I could use these areas again with a low level format. Decided that was not worth the effort.

But sans any thermal or shock issues, your drive may be dying.

[KC2UGV]

Okay, I guess I can try GPartEd. If it makes life easier I am up for it. Does GParted do the entire cloning process ? Or do I need to use any additional utilities. I was under the impression that GPartEd was just for re sizing existing partitions ?

I guess I should clarify. The GParted LiveCD. It has a cloning utility on it. It may even be...


Also, what about "Clonezilla" ?

...this. Not positive, as it's been a while since I've had to clone a drive. I try to keep /home and /var on separate drives.

[KA9MOT]

Depending on the drive and it's firmware, you may be able to get it to stop using the bad spots. My laptop had an issue after it was dropped while running when I was in the hospital. I lost 3 MP3 files that I had backed up elsewhere. Scary, but it worked out. I had to boot via USB stick to run the utility, and it has been fine ever since, by setting the damaged area as "bad". The utility was manufacturer specific, Western Digital in this case. I did get a spare drive, but have not needed it... and check the running drive every now and then.

The damage happened as I was writing MP3 files from my Sansa player. Luckily, it caused no physical harm, and if I ever cared, I could use these areas again with a low level format. Decided that was not worth the effort.

But sans any thermal or shock issues, your drive may be dying.

Usually when you hear a drive clicking, it has noting to do with bad sectors. It is mechanical.

[n2ize]

As a precaution the disc is being cloned as I speak. I decided to try "Clonezilla". So far so good. When it is finished I should, hopefully, have a working copy of my existing drive on a new drive. Let's face it. This drive was way overdue for a cloning/backup. It has been chugging along for many years and may be running on borrowed time.

[n2ize]

Keeping a good backup ready is a must, i do a backup every week with a USB H.D.
My laptop is on 12 to 16 hours a day and needs a good cleanout including opening up to get to the fans every 3 months minimum.

Most important thing to backup on that particular machine are the database files. Replacing those would be next to impossible. Lot of invaluable research/study materials in those files. Those get backed up regularly.

[WØTKX]

Usually when you hear a drive clicking, it has noting to do with bad sectors. It is mechanical.

But, if it's working furiously to read bad data, it will make repetitive noises that sound very similar. Truth.

In my case the clicking would only happen when trying to work with the damaged files.
Marked that area bad, and it's been humming along for another two years.

I was able to test the drive at a low level, and it said that area was fine, but the sector data in that spot was corrupt.

[n2ize]

In this case the clicking I heard happened when i tried to boot the machine. And after letting it rest for a few minutes it stopped. I suspect something overheated. It was very humid last night, the air was still and the cellar can get quite warm with the furnace and boiler nearby. In the fall, winter, spring there usually a breeze and the cellar gets better ventilation. I may have to consider installing an exhaust fan.

[KC2UGV]

In this case the clicking I heard happened when i tried to boot the machine. And after letting it rest for a few minutes it stopped. I suspect something overheated. It was very humid last night, the air was still and the cellar can get quite warm with the furnace and boiler nearby. In the fall, winter, spring there usually a breeze and the cellar gets better ventilation. I may have to consider installing an exhaust fan.

Might want to consider "not in the basement"...

[n2ize]

Might want to consider "not in the basement"...
Yeah, I might have to rethink the location. For the most part the cellar was a convenient location. Out of the way, usually dry and not much dust, mild even in the coldest winter and generally not to hot in the summer. Plus it is right where the FiOS, phone and electrical service enters the building. A couple of years ago a new furnace and boiler was installed and I think as a result there is a greater "heat factor" than in the past. So yes, I might consider relocating or eventually discontinuing and decommissioning the server and outsourcing my services. Might even be more productive and useful in the long run. I can understand why many businesses chose to outsource their services these days. Far less maintenance, better redundancy, more robust and dedicated staff.

[n2ize]

Okay, so far thwe following partitions cloned ok, /dev/sda1, sda2. sda3, sda4, sda5. On /dev/sda6 we couldn't get a clone because there were some surface errors on the disk. It is not trying to clone /dev/sda7, /dev/sda8 and /dev/sda9 (yeah, i partition the crap outta discs). I guess if there are no other issues I will see what is stored on /dev/sda6. If it is nothing too critical I may be able to restore what is missing from scratch. At least some of the important stuff has been copied so it will help make life a little easier. Indeed, the hard drive was dying.

[n2ize]

Aside from some surface errors it looks like the important partitions were cloned ok. Now, if I get a chance today I put in the clone and see it the system will boot and run without issue.

[ad4mg]

Any particular utilities that you would recommend for a full disc clone ? In the past I would usually use "tar" or "cp", or "dd".

Download the latest Clonezilla iso and burn a CD. Do a disk to disk, it just can't possibly get any easier. It actually utilizes "dd" for the copy, with an easy to use front end for the parameters.

Drive to drive, or drive to image, nothing is easier than Clonezilla. It's even amazingly simple to clone a drive to an image across a network.



ETA: I see you made the wise choice!

[n2ize]

Download the latest Clonezilla iso and burn a CD. Do a disk to disk, it just can't possibly get any easier. It actually utilizes "dd" for the copy, with an easy to use front end for the parameters.

Drive to drive, or drive to image, nothing is easier than Clonezilla. It's even amazingly simple to clone a drive to an image across a network.



ETA: I see you made the wise choice!

dd is pretty easy to use in and of itself. But I decided to use Clonezilla this time around. It did a great job. Except for some disc surface errors on one partition it cloned the entire disk and, in particular all my important stuff, like my daemon configurations, binaries, database files, etc. etc. I put the new disc in today and viola' it booted up no problem and the system is up and running normally. Clonezilla did a super job.

A few small problems...

1) I think I am still going to have to use GParted to resize the partitions to take advantage of the larger disc space.

2) Certain problems occurred where Clonezilla could not fully clone partition /dev/sda6 which contains /usr/local. This is not to bad since none of my really critical stuff is stored in or running from /usr/local. For example an experimental web app won't run from /usr/local because a library file was either corrupted or wasn't copied. But that is no problem because the configs are all there and all I really need is to "diff" the directories and re-install whatever was missing. The great part was that Clonezilla handled the errors gracefully and left me with a functional copy of my original drive.

So, all in all Clonzilla did a top job. And I still have the old disc drive should any issues arise over the next few days.

On a side note, when I opened the system to replace the drive and clean it I was surprised at the lack of dust inside. Aside from a small spider web and some collected dust near the bottom, the rest of the system was immaculately clean inside. And this is after years upon years of constant running.24/7/365. Apparently the basement is a remarkably clean dust free environment. I wish the same was true for upstairs. My desktop system is usually caked with dust after a few months.