RHEL 4 not seeing ext3 label

JaredBusch

So back home, and I have the files backed up in like 4 places.

I recombined the .img files and then unzipped them.

Getting ready to setup a new VM on Proxmox, but I poked around dmesg on the running system first.

SCSI subsystem initialized
Fusion MPT base driver 3.02.73rh
Copyright (c) 1999-2006 LSI Logic Corporation
Fusion MPT SPI Host driver 3.02.73rh
ACPI: PCI Interrupt 0000:02:05.0[A] -> GSI 34 (level, low) -> IRQ 201
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator,Target}
scsi0 : ioc0: LSI53C1030, FwRev=01032300h, Ports=1, MaxQ=255, IRQ=201
ACPI: PCI Interrupt 0000:02:05.1[B] -> GSI 33 (level, low) -> IRQ 209
mptbase: Initiating ioc1 bringup
ioc1: 53C1030: Capabilities={Initiator,Target}
scsi1 : ioc1: LSI53C1030, FwRev=01032300h, Ports=1, MaxQ=255, IRQ=209
Fusion MPT SAS Host driver 3.02.73rh
megaraid cmm: 2.20.2.6rh (Release Date: Tue Jan 16 12:35:06 PST 2007)
megaraid: 2.20.4.6-rh2 (Release Date: Wed Jun 28 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x1960:0x1028:0x0518: bus 9:slot 4:func 0
ACPI: PCI Interrupt 0000:09:04.0[A] -> GSI 106 (level, low) -> IRQ 233
megaraid: fw version:[351S] bios version:[1.10]
scsi2 : LSI Logic MegaRAID driver
scsi[2]: scanning scsi channel 0 [Phy 0] for non-raid devices
  Vendor: PE/PV     Model: 1x6 SCSI BP       Rev: 1.0 
  Type:   Processor                          ANSI SCSI revision: 02
scsi[2]: scanning scsi channel 1 [Phy 1] for non-raid devices
scsi[2]: scanning scsi channel 2 [virtual] for logical drives
  Vendor: MegaRAID  Model: LD 0 RAID1   69G  Rev: 351S
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sda: 143114240 512-byte hdwr sectors (73274 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
SCSI device sda: 143114240 512-byte hdwr sectors (73274 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
 sda: sda1 sda2 sda3
Attached scsi disk sda at scsi2, channel 2, id 0, lun 0
  Vendor: MegaRAID  Model: LD 1 RAID5  139G  Rev: 351S
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sdb: 286228480 512-byte hdwr sectors (146549 MB)
sdb: asking for cache data failed
sdb: assuming drive cache: write through
SCSI device sdb: 286228480 512-byte hdwr sectors (146549 MB)
sdb: asking for cache data failed
sdb: assuming drive cache: write through
 sdb: sdb1
Attached scsi disk sdb at scsi2, channel 2, id 1, lun 0

I think this tells me that I should try the megaRAID controller this time. I swaer I already tried. But I have slept since then. Tuesday and Wednesday were crazy stressed getting data..

JaredBusch

Well damnit. It does not see the second disk..

Looks like an error during boot

Dashrender

can you boot from a live image and see both disks?

I did a d2vm of a windows 2003 server and I had to run checkdisk like 10 times before it finally worked.. don't ask my why I tried it so many times... I think there is a thread around here somewhere about it.

JaredBusch

@Dashrender said in RHEL 4 not seeing ext3 label:

can you boot from a live image and see both disks?

I did a d2vm of a windows 2003 server and I had to run checkdisk like 10 times before it finally worked.. don't ask my why I tried it so many times... I think there is a thread around here somewhere about it.

The restored drives are fine. Can be mounted as previously noted and the label reports correctly.

The issue seems to be that the kernel, as built, is not loading the drives correctly. Potentially because the VM is using a SCSI driver method the old ass kernel does not understand.

jt1001001

Didn't Dell "back in the day" use or require their own megaraid driver's on Linux?? Can't remember as its been ages since I delt with a 28XX series with a PERC raid card.

JaredBusch

Using VirtIO SCSI (the default selection) the drives are not even seen by tthe recovery boot image. The onyl thing shown is the USB drive holding the data to restore.

JaredBusch

Using VMWare PVSCSI the system won't even boot.

JaredBusch

VirtIO SCSI Single is the same as VirtIO SCSI

JaredBusch

The LSI 53C895A shows the drives, so attempting a restore..
Note: Even though it says "Default" this is not the default choice when you go through the wizard, VirtIO SCSI is the default selection.

JaredBusch

Here is a video of the physical server booting. You can see the drives coming online at the 1:56 mark

Youtube Video

JaredBusch

@JaredBusch said in RHEL 4 not seeing ext3 label:

The LSI 53C895A shows the drives, so attempting a restore..
Note: Even though it says "Default" this is not the default choice when you go through the wizard, VirtIO SCSI is the default selection.

So the restore completed, but no boot drive detected I guess.

JaredBusch

@JaredBusch or because I am getting cross eyed and did not notice that scsi0 was not selected in the boot order....

Fixed that.. Label not found.

JaredBusch

CentOS 4.8 booting to rescue mode

The labels show as being correct.

JaredBusch

This guy had a similar issue. he was moving from VMWare (old) instead of Physical to KVM.
https://serverfault.com/questions/1040265/cannot-migrate-ancient-vm

But his non-answer does not help.

Also, it appears my scsi_mod is loading first. so not the same issue he had.

JaredBusch

and solved it. finally..

one of the reboots into the CentOS 4 disc was slow or something and I caught it pop this screen (took me 6 reboots to get screenshot).
This is the Kudzu hardware detection thing.

The VM is setup using the LSI 53C895A SCSI Controller.
Booted into rescue mode with the CentOS 4 CD.

chroot /mnt/sysimage
vi /etc/modprobe.conf
# make this the only scsi_hostapadter
alias scsi_hostadapter sym53c8xx
# exit vi !!! omfg how!!!
cd /boot
mkinitrd -v -f initrd-2.6.9-55.EL.img 2.6.9-55.EL
exit
exit

System automatically reboots to come out of rescue mode.
Make sure you remove the ISO at this point.
Then boom..

JaredBusch

@Pete-S said in RHEL 4 not seeing ext3 label:

It feels like I'm watching reality TV.

Restoring the above dd created images now.

I like to dump live as I work on weird things like this, because it will almost certainly help someone else at some point.

1337

@Dashrender said in RHEL 4 not seeing ext3 label:

@JaredBusch said in RHEL 4 not seeing ext3 label:

@Pete-S said in RHEL 4 not seeing ext3 label:

You could potentially try to install centos4, or rhel4 is even better, so you get to a bootable system.
Then just copy the files from the backup over your installation.

It is an option I have thought about. I'll be on site this morning, and I will be shutting down the host and booting to a Fedora Live to run dd in an effort to get a solid disk image.

I tried their built in process on Tuesday and it failed with sector/block read errors. A little digging through the files on the recovery ISO showed that all they were doing was using dd, so I am hoping to use dd with more intelligent options to continue on and such.

What's repair solution for bad blocks in a setup like this? If dd can't read because of bad blocks, I'm hoping 'nix has some tool to fix/recover/replace these bad blocks, assuming the data's recoverable on the hardware, otherwise it's a restore time, right?

ddrescue is the best tool to use to copy from a drive that has errors. It's actually made for data recovery and is much smarter than dd. It will probe the drive and the data that is missing in different ways and several times so it can usually retrieve more data.

1337

@JaredBusch said in RHEL 4 not seeing ext3 label:

@Pete-S said in RHEL 4 not seeing ext3 label:

It feels like I'm watching reality TV.

Restoring the above dd created images now.

I like to dump live as I work on weird things like this, because it will almost certainly help someone else at some point.

Yeah, it's great!

JaredBusch

Booted straight to the CentOS 4 ISO, went into linux rescue, updated the initrd img and bam. working system from the current (as of 4 days ago) manual disk images I made.

Next project to re-learn how they restore data files. Have not done that in almost 10 years. Having no virtual infrastructure to play with, prior to this, made that harder.