xenserver error fsck

DustinB3403

@travisdh1 said in xenserver error fsck:

@DustinB3403 said in xenserver error fsck:
@travisdh1 said in xenserver error fsck:
Hrm, maybe try remounting root? I think this should do it:
mount -o remount,rw /
/devb/sdb1 read-write is write-protected.

So how do I remove that?
If it's write protected to the point that it won't remount in read/write mode (what you have here), it's time to assume the storage is not recoverable.

The good news (if such a thing exists in this situation), is that restoring from XO backups is quite quick. This is what I got to deal with 3 weeks ago. XS Root drive went into a read only state, and was stuck that way even between reboots. Replaced the root drive and imported the XO backups... I used XenCenter to import the XO image itself, and then used XO to import the other backups.

We've got dc's on this host...... which means I have to do a SPX restore (since XO) isn't backing up.

travisdh1

@DustinB3403 said in xenserver error fsck:

@travisdh1 said in xenserver error fsck:
@DustinB3403 said in xenserver error fsck:
@travisdh1 said in xenserver error fsck:
Hrm, maybe try remounting root? I think this should do it:
mount -o remount,rw /
/devb/sdb1 read-write is write-protected.

So how do I remove that?
If it's write protected to the point that it won't remount in read/write mode (what you have here), it's time to assume the storage is not recoverable.

The good news (if such a thing exists in this situation), is that restoring from XO backups is quite quick. This is what I got to deal with 3 weeks ago. XS Root drive went into a read only state, and was stuck that way even between reboots. Replaced the root drive and imported the XO backups... I used XenCenter to import the XO image itself, and then used XO to import the other backups.
We've got dc's on this host...... which means I have to do a SPX restore (since XO) isn't backing up.

DustinB3403

And a f***ing massive file server.

Danp

Any chance you previously saved the metadata to your SR?

BRRABill

Sorry you are going through this.

Especially on a Friday night.

DustinB3403

Im going to start an restore from the 21'st to my second host, which is running fine. And then we'll have to do file level restores from there....

For the DC going to have to bring in our MSP.....

F**** [moderated] XenServer why do you hate me so much.

Why has my original XS host run fine (literally like a f***** [moderated] dream for 2 years) and these two hosts are giving me issues....

what the f**** [moderated]!

DustinB3403

@Danp said in xenserver error fsck:

Any chance you previously saved the metadata to your SR?

No.

Dashrender

@DustinB3403 said in xenserver error fsck:

I dont even think I can migrate the VM's off.

I was afraid of this.

DustinB3403

@Danp said in xenserver error fsck:

Any chance you previously saved the metadata to your SR?

I can however restore my backups from 6 days ago to my remote host, and inside of the VM's boot from SPX recovery environment and restore from there.

Danp

@DustinB3403 Or a clone / image of the USB?

DustinB3403

@Danp said in xenserver error fsck:

@DustinB3403 Or a clone / image of the USB?

Nothing recent enough to try I don't think. I do have a clone of this USB though.

DustinB3403

@Danp from 6-21-2016.

f****ing bullpucky[moderated]... cause "we can never have these servers down"

Well how the f*** *[moderated] do you expect to do maintenance or anything on them?!

Dashrender

Hold on dude..
your data is still probably just fine..

As long as XC still works, pull the data i mentioned before - find all the uuid's from all drives and their mounted order to each VM and write it down.
then you'll shut down the server, replace the USB drive, then install XS then install updates,
then attach SR
then start remaking your VMs by hand and attach your VDIs to the VMs and boot

Danp

@DustinB3403 said in xenserver error fsck:

@Danp said in xenserver error fsck:

@DustinB3403 Or a clone / image of the USB?

Nothing recent enough to try I don't think. I do have a clone of this USB though.

Make another copy and then boot from that. What's the worst that could happen?

BRRABill

Yeah I would definitely try...

booting from your cloned USB
reinstalling XS and recreating your VMs by hand as Dash has suggested
doing a restore

In that order.

I played with XS a lot moving the boot drive and stuff and it was pretty resilient.

DustinB3403

How do I get the disk UUIDs for the Vm's?

momurda

@DustinB3403
xe vm-list
or
xe vdi-list for disks

BRRABill

@momurda said in xenserver error fsck:

@DustinB3403
xe vm-list
or
xe vdi-list for disks

I wonder if that will work for him?

Seems like anything the host has to do fails...

momurda

If those wont work in cli, you still can use XC to get this info. Or XO i imagine has this info listed on its web interface.

DustinB3403

we're probably going to migrate everything over (but guess when.... next flipping weekend...)