Xen Orchestra - Permissions Issue - File level restore

travisdh1

@Danp said in Xen Orchestra - Permissions Issue - File level restore:

@DustinB3403 I have experienced the fuser error

Totally off topic: How can you not love Linux/UNIX, especially when we have errors telling us fuser.

DustinB3403

@Danp said in Xen Orchestra - Permissions Issue - File level restore:

@DustinB3403 I haven't experienced the fuser error

Edit: Fixed missing negative

So are you able to restore files?

Danp

@DustinB3403 said in Xen Orchestra - Permissions Issue - File level restore:

So are you able to restore files?

Nope. Either getting errors in the log like this --


Dec 21 11:27:23 xo-5 xo-server[26997]: { Error: Command failed: mount --options=loop,ro,offset=1048576,noload --source=/tmp/tmp-269977KZDUzPQPshd/vhdi1 --target=/tmp/tmp-26997NWHkKW0eSCVg
Dec 21 11:27:23 xo-5 xo-server[26997]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
Dec 21 11:27:23 xo-5 xo-server[26997]:        missing codepage or helper program, or other error
Dec 21 11:27:23 xo-5 xo-server[26997]:        In some cases useful info is found in syslog - try
Dec 21 11:27:23 xo-5 xo-server[26997]:        dmesg | tail or so.
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise.all.then.arr (/opt/xo-server/node_modules/execa/index.js:170:11)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at tryCatcher (/opt/xo-server/node_modules/bluebird/js/release/util.js:16:23)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromiseFromHandler (/opt/xo-server/node_modules/bluebird/js/release/promise.js:510:31)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromise (/opt/xo-server/node_modules/bluebird/js/release/promise.js:567:18)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromise0 (/opt/xo-server/node_modules/bluebird/js/release/promise.js:612:10)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromises (/opt/xo-server/node_modules/bluebird/js/release/promise.js:691:18)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._fulfill (/opt/xo-server/node_modules/bluebird/js/release/promise.js:636:18)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at PromiseArray._resolve (/opt/xo-server/node_modules/bluebird/js/release/promise_array.js:125:19)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at PromiseArray._promiseFulfilled (/opt/xo-server/node_modules/bluebird/js/release/promise_array.js:143:14)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromise (/opt/xo-server/node_modules/bluebird/js/release/promise.js:572:26)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromise0 (/opt/xo-server/node_modules/bluebird/js/release/promise.js:612:10)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Promise._settlePromises (/opt/xo-server/node_modules/bluebird/js/release/promise.js:691:18)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Async._drainQueue (/opt/xo-server/node_modules/bluebird/js/release/async.js:138:16)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Async._drainQueues (/opt/xo-server/node_modules/bluebird/js/release/async.js:148:10)
Dec 21 11:27:23 xo-5 xo-server[26997]:     at Immediate.Async.drainQueues (/opt/xo-server/node_modules/bluebird/js/release/async.js:17:14)
Dec 21 11:27:23 xo-5 xo-server[26997]:   code: 32,
Dec 21 11:27:23 xo-5 xo-server[26997]:   killed: false,
Dec 21 11:27:23 xo-5 xo-server[26997]:   stdout: '',
Dec 21 11:27:23 xo-5 xo-server[26997]:   stderr: 'mount: wrong fs type, bad option, bad superblock on /dev/loop0,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n',
Dec 21 11:27:23 xo-5 xo-server[26997]:   failed: true,
Dec 21 11:27:23 xo-5 xo-server[26997]:   signal: null,
Dec 21 11:27:23 xo-5 xo-server[26997]:   cmd: 'mount --options=loop,ro,offset=1048576,noload --source=/tmp/tmp-269977KZDUzPQPshd/vhdi1 --target=/tmp/tmp-26997NWHkKW0eSCVg' }
Dec 21 11:27:23 xo-5 xo-server[26997]: Wed, 21 Dec 2016 17:27:23 GMT xo:api [email protected] | backup.scanFiles(...) [233ms] =!> Error: Command failed: mount --options=loop,ro,offset=1048576,noload --source=/tmp/tmp-269977KZDUzPQPshd/vhdi1 --target=/tmp/tmp-26997NWHkKW0eSCVg
Dec 21 11:27:23 xo-5 xo-server[26997]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
Dec 21 11:27:23 xo-5 xo-server[26997]:        missing codepage or helper program, or other error
Dec 21 11:27:23 xo-5 xo-server[26997]:        In some cases useful info is found in syslog - try
Dec 21 11:27:23 xo-5 xo-server[26997]:        dmesg | tail or so.

or else this

Dec 21 11:51:40 xo-5 xo-server[26997]: Wed, 21 Dec 2016 17:51:40 GMT xo:api [email protected] | backup.scanDisk(...) [17ms] =!> Error: EROFS: read-only file system, mkdir '/tmp/tmp-26997k3oduVxLfSa8'

Danp

Probably should restart the VM to see if that changes anything with the read-only issue.

DustinB3403

Yeah I'm still getting that fusermount error...

Dec 21 12:59:25 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:25 GMT xo:api [email protected] | remote.getAll(...) [2ms] ==> array
Dec 21 12:59:28 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:28 GMT xo:api [email protected] | resourceSet.getAll(...) [2ms] ==> array
Dec 21 12:59:28 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:28 GMT xo:api [email protected] | acl.getCurrentPermissions(...) [2ms] ==> object
Dec 21 12:59:30 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:30 GMT xo:api [email protected] | remote.getAll(...) [2ms] ==> array
Dec 21 12:59:31 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:31 GMT xen-api [email protected]: event.from(...) [30s] ==> object
Dec 21 12:59:31 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:31 GMT xen-api [email protected]: event.from(...) [14ms] ==> object
Dec 21 12:59:33 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:33 GMT xo:api [email protected] | resourceSet.getAll(...) [3ms] ==> array
Dec 21 12:59:33 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:33 GMT xo:api [email protected] | acl.getCurrentPermissions(...) [3ms] ==> object
Dec 21 12:59:35 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:35 GMT xo:api [email protected] | backup.scanDisk(...) [887ms] =!> Error: spawn fusermount ENOENT
Dec 21 12:59:35 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:35 GMT xo:api [email protected] | remote.getAll(...) [3ms] ==> array
Dec 21 12:59:38 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:38 GMT xo:api [email protected] | resourceSet.getAll(...) [3ms] ==> array
Dec 21 12:59:38 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:38 GMT xo:api [email protected] | acl.getCurrentPermissions(...) [3ms] ==> object
Dec 21 12:59:40 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:40 GMT xen-api [email protected]: event.from(...) [1m] ==> object
Dec 21 12:59:40 xo-test-debian-3 xo-server[5648]: Wed, 21 Dec 2016 17:59:40 GMT xo:api [email protected] | remote.getAll(...) [2ms] ==> array

Danp

There was definitely something wrong with the VM's state. Whoami returned unknown uid. Had to forcibly reboot from within XO.

Danp

Was actually able to pull up a file listing from one of my backups. Here's what showed in the log -

Dec 21 12:07:02 xo-5 ntfs-3g[1857]: Version 2015.3.14AR.1 integrated FUSE 28
Dec 21 12:07:02 xo-5 ntfs-3g[1857]: Mounted /dev/loop1 (Read-Only, label "", NTFS 3.1)
Dec 21 12:07:02 xo-5 ntfs-3g[1857]: Cmdline options: ro
Dec 21 12:07:02 xo-5 ntfs-3g[1857]: Mount options: ro,allow_other,nonempty,relatime,fsname=/dev/loop1,blkdev,blksize=4096
Dec 21 12:07:02 xo-5 ntfs-3g[1857]: Ownership and permissions disabled, configuration type 7
Dec 21 12:07:02 xo-5 ntfs-3g[1857]: Unmounting /dev/loop1 ()

Danp

@DustinB3403 said in Xen Orchestra - Permissions Issue - File level restore:

Yeah I'm still getting that fusermount error...

What is the output from whereis fusermount?

Danp

@DustinB3403 said in Xen Orchestra - Permissions Issue - File level restore:

So are you able to restore files?

Yes, it appears to work in those cases where I can actually get a file listing.

DustinB3403

@Danp said in Xen Orchestra - Permissions Issue - File level restore:

@DustinB3403 said in Xen Orchestra - Permissions Issue - File level restore:

Yeah I'm still getting that fusermount error...

What is the output from whereis fusermount?

xoadmin@xo-test-debian-3:~$ whereis fusermount
fusermount:

DustinB3403

I do have fusermount under

/usr/share/bash-completion/completions/fusermount

Danp

This is what I get

xoadmin@xo-5:~$ whereis fusermount
fusermount: /bin/fusermount /usr/share/man/man1/fusermount.1.gz

Guessing that you either don't have it properly installed or it isn't in a location where it can be found.

DustinB3403

@Danp Do you have SSHFS installed?

DustinB3403

One sec, Fuse not installed on debian by default... FML... repo added installing fuse.

Rebooting and then recompiling..

Danp

@DustinB3403 Thought you did that yesterday. Did you rebuild from scratch?

DustinB3403

@Danp said in Xen Orchestra - Permissions Issue - File level restore:

@DustinB3403 Thought you did that yesterday. Did you rebuild from scratch?

No I added the universal repo yesterday for something else that was needed.

Anyways, now trying to mount a single backup I get

Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: Wed, 21 Dec 2016 19:15:18 GMT xo:api [email protected] | backup.scanDisk(...) [1s] ==> object
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: { [Error: Command failed: mount --options=loop,ro --source=/tmp/tmp-416paluKS1uHkUS/vhdi7 --target=/tmp/tmp-4160yW3G6ma9LZP
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: missing codepage or helper program, or other error
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: In some cases useful info is found in syslog - try
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: dmesg | tail or so.
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: ]
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: code: 32,
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: killed: false,
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: stdout: '',
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: stderr: 'mount: wrong fs type, bad option, bad superblock on /dev/loop0,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n',
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: failed: true,
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: signal: null,
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: cmd: 'mount --options=loop,ro --source=/tmp/tmp-416paluKS1uHkUS/vhdi7 --target=/tmp/tmp-4160yW3G6ma9LZP' }
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: Wed, 21 Dec 2016 19:15:18 GMT xo:api [email protected] | backup.scanFiles(...) [218ms] =!> Error: Command failed: mount --options=loop,ro --source=/tmp/tmp-416paluKS1uHkUS/vhdi7 --target=/tmp/tmp-4160yW3G6ma9LZP
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: missing codepage or helper program, or other error
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: In some cases useful info is found in syslog - try
Dec 21 14:15:18 xo-test-debian-3 xo-server[416]: dmesg | tail or so.

Trying another backup . oh and dmesg | tail contains

10.867971] Adding 392188k swap on /dev/xvda5.  Priority:-1 extents:1 across:392188k SSFS
13.590095] RPC: Registered named UNIX socket transport module.
13.590099] RPC: Registered udp transport module.
13.590100] RPC: Registered tcp transport module.
13.590101] RPC: Registered tcp NFSv4.1 backchannel transport module.
13.633648] FS-Cache: Loaded
13.681056] FS-Cache: Netfs 'nfs' registered for caching
13.943515] Installing knfsd (copyright (C) 1996 [email protected]).
86.204993] fuse init (API version 7.23)
86.635229] loop: module loaded

DustinB3403

Seems to be specific to that backup, as I am able to restore files to my file server now without issue.

So a few things to we should try to compile into the update script, check for fuse, and install it if its not there.

DustinB3403

Another is listing things like this....

398.675872] ntfs: (device loop0): ntfs_readdir(): Directory index record with vcn 0x0 is corrupt.  Corrupt inode 0xe96b9.  Run chkdsk.
398.675890] ntfs: (device loop0): ntfs_readdir(): Directory index record with vcn 0x0 is corrupt.  Corrupt inode 0xe96b9.  Run chkdsk.

So yeah.. gonna run a check disk on my VM here..

Danp

@DustinB3403 said in Xen Orchestra - Permissions Issue - File level restore:

Seems to be specific to that backup, as I am able to restore files to my file server now without issue.

So a few things to we should try to compile into the update script, check for fuse, and install it if its not there.

Also for libvhdi-utils

DustinB3403

@Danp That was the other thing, that I installed yesterday.

libvhdi-utils