Saving a dying server
-
server we are on is a DS710i which was launched back in late 2011 it seems - from what I can see it should have come with dual 1tb drives but for some reason we seem to just have 1.
so far the backup seems to have completed. a 7GB tar took about 1 day to complete. luckily they were taking regular backups of the db so not much to replay after the last backup on that gets restored.
-
@larsen161 said:
server we are on is a DS710i which was launched back in late 2011 it seems - from what I can see it should have come with dual 1tb drives but for some reason we seem to just have 1.
How are you determining that you only have one?
-
well, well, well....
Looks like when someone said there was only drive on there what was actually meant was only one drive was setup and in use. Another guy has been working on this but I just decided to jump on it and take another look at what is going on.# iostat -x 1 Linux 3.2.0-99-generic (server88-208-204-138.live-servers.net) 09/03/16 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 4.05 1.77 1.31 46.05 0.00 46.82 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 3.53 20.35 21.25 10.70 1363.54 368.05 108.39 25.83 808.39 105.71 2204.51 28.08 89.73 sdb 0.00 0.00 0.00 0.00 0.01 0.00 9.32 0.00 0.52 0.52 0.00 0.52 0.00 dm-0 0.00 0.00 22.84 18.10 1355.16 320.84 81.86 29.63 723.51 156.72 1438.68 21.91 89.72 # fdisk -l Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0008e4c8 Device Boot Start End Blocks Id System /dev/sda1 * 2048 499711 248832 83 Linux /dev/sda2 499712 8499199 3999744 82 Linux swap / Solaris /dev/sda3 8501246 1953523711 972511233 5 Extended /dev/sda5 8501248 1953523711 972511232 8e Linux LVM Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdb doesn't contain a valid partition table Disk /dev/mapper/VolGroup00-LogVol00: 995.8 GB, 995849404416 bytes 255 heads, 63 sectors/track, 121071 cylinders, total 1945018368 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/VolGroup00-LogVol00 doesn't contain a valid partition table -
@larsen161 said:
well, well, well....
Looks like when someone said there was only drive on there what was actually meant was only one drive was setup and in use. Another guy has been working on this but I just decided to jump on it and take another look at what is going on.# iostat -x 1 Linux 3.2.0-99-generic (server88-208-204-138.live-servers.net) 09/03/16 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 4.05 1.77 1.31 46.05 0.00 46.82 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 3.53 20.35 21.25 10.70 1363.54 368.05 108.39 25.83 808.39 105.71 2204.51 28.08 89.73 sdb 0.00 0.00 0.00 0.00 0.01 0.00 9.32 0.00 0.52 0.52 0.00 0.52 0.00 dm-0 0.00 0.00 22.84 18.10 1355.16 320.84 81.86 29.63 723.51 156.72 1438.68 21.91 89.72 # fdisk -l Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0008e4c8 Device Boot Start End Blocks Id System /dev/sda1 * 2048 499711 248832 83 Linux /dev/sda2 499712 8499199 3999744 82 Linux swap / Solaris /dev/sda3 8501246 1953523711 972511233 5 Extended /dev/sda5 8501248 1953523711 972511232 8e Linux LVM Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdb doesn't contain a valid partition table Disk /dev/mapper/VolGroup00-LogVol00: 995.8 GB, 995849404416 bytes 255 heads, 63 sectors/track, 121071 cylinders, total 1945018368 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/VolGroup00-LogVol00 doesn't contain a valid partition tableOh geeze. Who set that up? Are you expected to set the RAID up yourself?
-
They forgot to RAID it!!!
-
I saw this thread ages ago, but when I saw forgot to raid it, I re-read the OP and then saw Fasthosts.
Yeah....they play VERY fast and loose with their customers. RUN AWAY screaming your head off if you can. This won't help you immediately but get anything you have with them into another provider as quickly as possible.
-
it's now moving over to AWS where the rest of the companies servers are. this started with one small local project many years back outside of the main dev/r&d team at hq and grew to host more systems for not only our local uk team but those in all our other offices in the us, ru, & il.
Fasthosts are being quite responsive and helpful. This is the latest from them once I said why wasn't this configured with hardware RAID from the start.
"A DS710i will have had raid configured, unless the server is of certain flavours of Ubuntu, which don't support software raid in a stable manner. Your older server is Ubuntu 12.04, which is one of those affected builds.
Your other server has hardware raid configured, I believe.
Based on your screenshot, only one drive was configured at all - /sdb contains no partitions. I assume that this means the performance issues are purely due to issues on /sda. As this contains all of your data and system files, replacing this will require a rebuild of the server. Following that, we can connect the existing disk for data recovery"
-
@larsen161 said:
"A DS710i will have had raid configured, unless the server is of certain flavours of Ubuntu, which don't support software raid in a stable manner. Your older server is Ubuntu 12.04, which is one of those affected builds.
That makes no sense. You get hardware RAID... unless you need it. You clearly needed it, so you didn't get it?
-
@scottalanmiller exactly
myself. moving on, so not worrying about trying understand logic in that one -
@scottalanmiller said:
@larsen161 said:
"A DS710i will have had raid configured, unless the server is of certain flavours of Ubuntu, which don't support software raid in a stable manner. Your older server is Ubuntu 12.04, which is one of those affected builds.
That makes no sense. You get hardware RAID... unless you need it. You clearly needed it, so you didn't get it?
Oh geeze, I glossed over that one too. Idk what's wrong with me today. That doesn't make any sense at all.
