Exchange, Unitrends, Veeam and ESXi walk into a bar

[nadnerB]

G'Day Lads n Lasses,

I'm getting the below error when backing up my Exchange 2010 server with Veeam B&R 7 which causes the job to fail.

Source: Disk
Event ID: 15
Description:
The device, \Device\Harddisk2\DR2, is not ready for access yet.

I've checked the ESXi host and the data store, partition and the VMFS seem to be ok (two other servers on the same data store back up perfectly fine), and this has lead me to believe that there is something wrong with the virtual hard drives.

One of the helpful lads at Veeam support sent this as part of a response to my support ticket:
Here is the error output from the logging data provided:

[17.06.2014 15:07:20] < 1816> >> |--tr:Disk read operation has failed. Disk path: [[<Datastore name>] <VM_Folder/VM_HDD1.vmdk]. Read offset: [5,742,002,176]. Requested data size: [4194304]. Disk size: [214,748,364,800]
[17.06.2014 15:07:20] < 1816> >> |--tr:Failed to pre-read data ( first start block '5476', blocks num '4'. Block size: '1048576').
[...]
[17.06.2014 15:07:20] < 1816> >> |Exception from server: VDDK error: 21036749815809.Unknown error

This is suggesting that an error has occurred reading from the storage within a VDDK based disklib operation. This generally suggests that there is either an issue with the virtual hard disks, datastore disks, or the VM is a physical to virtual conversion that is encountering issues.

I think I've narrowed the issue down to one of my mailbox databases as I can back up ALL but one of them with UEB 7 (Free), so this means that it could still be an issue with the virtual hard disk or the data store disks.

Unitrends sends this error message to the GUI FATAL: Incomplete backup received from client. <sarcasm> So helpful>/sarcasm> I can get to the log file location via PuTTY but I am having difficulty getting more detailed information as I can't work out how to get the log file off the virtal appliance. Anyhoo, another issue for another day.

Now the questions:

1. Can I defrag the virtual hdd without taking exchange offline?
2. If I take the Exchange database offline, is it likely that the database won't mount after I run ESEUTIL against it? I have attempted to restore into a Recovery database but it's in a "Dirty Shutdown" state and I can't fix it, so I'm concerned.

I can't move hosts at this point in time as I have not yet configured the new host (awaiting an extra CPU and RAM).

[alexntg]

@nadnerB said:

Source: Disk
Event ID: 15
Description:
The device, \Device\Harddisk2\DR2, is not ready for access yet.

Which machine are you seeing this on?

I've checked the ESXi host and the data store, partition and the VMFS seem to be ok (two other servers on the same data store back up perfectly fine), and this has lead me to believe that there is something wrong with the virtual hard drives.

What kind of storage/RAID type is that VMDK on?

Now the questions:

1. Can I defrag the virtual hdd without taking exchange offline?

If you have a bad disk, one of the last things you ever want to do with it is defrag it. You risk causing further damage.

2. If I take the Exchange database offline, is it likely that the database won't mount after I run ESEUTIL against it? I have attempted to restore into a Recovery database but it's in a "Dirty Shutdown" state and I can't fix it, so I'm concerned.

Not sure. "It depends." If you're keeping it on the same disk, you're repairing a database on a bad disk. If you drop the database, move it to a new disk, then try to bring it up, if you do need to repair it, you'd be repairing it on a new disk.

I can't move hosts at this point in time as I have not yet configured the new host (awaiting an extra CPU and RAM).

Can you set up a new disk with a new database and move the mailboxes over to it? That might be your safest bet, though it is possible the IS would crash when you hit the corrupted part of the disk.

As a side note, check for any residual snapshots and that the Veeam proxy doesn't have any extra mounted disks. It shouldn't have anything to do with the issue, but it doesn't hurt to check.

[nadnerB]

@alexntg

@alexntg said:

Which machine are you seeing this on?

The Mail server

What kind of storage/RAID type is that VMDK on?

Storage Blade

• RAID 5
  -- NOT FOR LONG!!!

If you have a bad disk, one of the last things you ever want to do with it is defrag it. You risk causing further damage.
Righto

Not sure. "It depends." If you're keeping it on the same disk, you're repairing a database on a bad disk. If you drop the database, move it to a new disk, then try to bring it up, if you do need to repair it, you'd be repairing it on a new disk.

Righto, so, you suggest: dismount --> move to new disk --> Repairs? --> Mount

Can you set up a new disk with a new database and move the mailboxes over to it? That might be your safest bet, though it is possible the IS would crash when you hit the corrupted part of the disk.
Hmmmm, I'll have to think about this.

As a side note, check for any residual snapshots and that the Veeam proxy doesn't have any extra mounted disks. It shouldn't have anything to do with the issue, but it doesn't hurt to check.

I checked the Datastore and found many $-ctk.vmdk files. Research indicates that theses are for Change Block Tracking (from when I attempted to get Veeam to backup the server). Can I safely delete these? or is there something to turn off first? (veeam is not currently backing up this server.
Speaking of,
In the above image, the first virtual HDD isn't listing its provisioned size. Does that seem a little off to you?