Unable to delete KVM snapshot

black3dynamite

Are you trying to delete the snapshot while the VM is still running?

dafyre

Note, in the process below, <diskname> could be vda, sda, or hda.
The first command tells you which to use.

virsh domblklist plex
virsh blockcommit plex <disk name> --verbose --pivot --active

Once the blockcommit command finishes, shutdown plex, and rename the plex_snap disk image.

Start Plex back up and make sure your updates and such are still installed. If all is well, then delete the plex_snap disk image.

stacksofplates

The type of snapshot you created is an external snapshot. The type that you create when you use the Virt-Manager gui is an internal. The internal all exist in the .qcow2 image and are copy on write. The external are AOW/ROW so they have to be block committed to the original backing store before you can delet them since all new writes/reads have been directed to the new image.

stacksofplates

@dafyre said in Unable to delete KVM snapshot:

Note, in the process below, <diskname> could be vda, sda, or hda.
The first command tells you which to use.
virsh domblklist plex
virsh blockcommit plex <disk name> --verbose --pivot --active
Once the blockcommit command finishes, shutdown plex, and rename the plex_snap disk image.

Start Plex back up and make sure your updates and such are still installed. If all is well, then delete the plex_snap disk image.

You shouldn’t have to shutdown the image unless it’s just for the updates. The pivot option points the guest back to the original backing store.

stacksofplates

Just as a side note, if the guest is running a database it's best to install the QEMU guest agent. Then pass --atomic when you do the snapshot. This will quiesce the file system and then unfreeze it when the snapshot is finished.

JaredBusch

@stacksofplates said in Unable to delete KVM snapshot:

Just as a side note, if the guest is running a database it's best to install the QEMU guest agent. Then pass --atomic when you do the snapshot. This will quiesce the file system and then unfreeze it when the snapshot is finished.

Plex has some type of DB backing store, but not horribly worried about it for this one.

Emad R

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

JaredBusch

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

JaredBusch

@stacksofplates said in Unable to delete KVM snapshot:

The type of snapshot you created is an external snapshot. The type that you create when you use the Virt-Manager gui is an internal. The internal all exist in the .qcow2 image and are copy on write. The external are AOW/ROW so they have to be block committed to the original backing store before you can delet them since all new writes/reads have been directed to the new image.

I did external on purpose.

I tested this on a smaller VM weeks ago and did not have a problem. I guess because nothing changed? I just made the snapshot and then deleted.

JaredBusch

@stacksofplates said in Unable to delete KVM snapshot:

@dafyre said in Unable to delete KVM snapshot:
Note, in the process below, <diskname> could be vda, sda, or hda.
The first command tells you which to use.
virsh domblklist plex
virsh blockcommit plex <disk name> --verbose --pivot --active
Once the blockcommit command finishes, shutdown plex, and rename the plex_snap disk image.

Start Plex back up and make sure your updates and such are still installed. If all is well, then delete the plex_snap disk image.
You shouldn’t have to shutdown the image unless it’s just for the updates. The pivot option points the guest back to the original backing store.

But I still need to delete the backup file manually from disk?

JaredBusch

@black3dynamite said in Unable to delete KVM snapshot:

Are you trying to delete the snapshot while the VM is still running?

Of course.

JaredBusch

sudo virsh domblklist plex
Target     Source
------------------------------------------------
hda        -
hdb        /kvm_store/disk_b/plex.plex_snap

[jbusch@kvm ~]$ sudo virsh blockcommit plex hdb --verbose --pivot --active
Block commit: [ 26 %]

Emad R

@jaredbusch said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

Interesting I dont take VM backup frequently, when I do I rsync the whole VM file.

rsync --progress --inplace -h -W /var/lib/libvirt/images/VM/centos7.0-clone.qcow2 [email protected]:/var/lib/libvirt/images/VM/centos7.0-clone.qcow2

This after the initial copy, if you are extra paranoid you can take the file everytime if you dont trust rsync algorithms.

But it is interesting this external snapshot approach, how much are the sizes of external snapshots roughly ? I know it depends but what are we dealing with here. 100 MB ?

stacksofplates

@jaredbusch said in Unable to delete KVM snapshot:

@stacksofplates said in Unable to delete KVM snapshot:
@dafyre said in Unable to delete KVM snapshot:
Note, in the process below, <diskname> could be vda, sda, or hda.
The first command tells you which to use.
virsh domblklist plex
virsh blockcommit plex <disk name> --verbose --pivot --active
Once the blockcommit command finishes, shutdown plex, and rename the plex_snap disk image.

Start Plex back up and make sure your updates and such are still installed. If all is well, then delete the plex_snap disk image.
You shouldn’t have to shutdown the image unless it’s just for the updates. The pivot option points the guest back to the original backing store.
But I still need to delete the backup file manually from disk?

Yes

stacksofplates

@jaredbusch said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

Allocate on Write snapshots are also much faster. It’s the way @scale does their snapshots. You can have thousands before you get a performance hit vs only a few with COW.

stacksofplates

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

Interesting I dont take VM backup frequently, when I do I rsync the whole VM file.

rsync --progress --inplace -h -W /var/lib/libvirt/images/VM/centos7.0-clone.qcow2 [email protected]:/var/lib/libvirt/images/VM/centos7.0-clone.qcow2

This after the initial copy, if you are extra paranoid you can take the file everytime if you dont trust rsync algorithms.

But it is interesting this external snapshot approach, how much are the sizes of external snapshots roughly ? I know it depends but what are we dealing with here. 100 MB ?

You have to shut the VM off to just rsync the drive. This way you can leave the VM on.

You don’t copy the external snapshot. You take a snapshot and copy the backing store. Then blockcommit (merge) the snapshot back into the original image.

Emad R

@stacksofplates said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

Interesting I dont take VM backup frequently, when I do I rsync the whole VM file.

rsync --progress --inplace -h -W /var/lib/libvirt/images/VM/centos7.0-clone.qcow2 [email protected]:/var/lib/libvirt/images/VM/centos7.0-clone.qcow2

This after the initial copy, if you are extra paranoid you can take the file everytime if you dont trust rsync algorithms.

But it is interesting this external snapshot approach, how much are the sizes of external snapshots roughly ? I know it depends but what are we dealing with here. 100 MB ?

You have to shut the VM off to just rsync the drive. This way you can leave the VM on.

You don’t copy the external snapshot. You take a snapshot and copy the backing store. Then blockcommit (merge) the snapshot back into the original image.

No need if you freeze the filesystem first then rsync. but for this I recommend taking each time the whole file with rsync

dafyre

@stacksofplates said in Unable to delete KVM snapshot:

@dafyre said in Unable to delete KVM snapshot:
Note, in the process below, <diskname> could be vda, sda, or hda.
The first command tells you which to use.
virsh domblklist plex
virsh blockcommit plex <disk name> --verbose --pivot --active
Once the blockcommit command finishes, shutdown plex, and rename the plex_snap disk image.

Start Plex back up and make sure your updates and such are still installed. If all is well, then delete the plex_snap disk image.
You shouldn’t have to shutdown the image unless it’s just for the updates. The pivot option points the guest back to the original backing store.

I do this because I've deleted the snapshot file after running blockcommit and had issues, so now I do that as a just in case measure. If it was something in production, I'd just leave the old snapshot file until the next maintenance window.

stacksofplates

@emad-r said in Unable to delete KVM snapshot:

@stacksofplates said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

Interesting I dont take VM backup frequently, when I do I rsync the whole VM file.

rsync --progress --inplace -h -W /var/lib/libvirt/images/VM/centos7.0-clone.qcow2 [email protected]:/var/lib/libvirt/images/VM/centos7.0-clone.qcow2

This after the initial copy, if you are extra paranoid you can take the file everytime if you dont trust rsync algorithms.

But it is interesting this external snapshot approach, how much are the sizes of external snapshots roughly ? I know it depends but what are we dealing with here. 100 MB ?

You have to shut the VM off to just rsync the drive. This way you can leave the VM on.

You don’t copy the external snapshot. You take a snapshot and copy the backing store. Then blockcommit (merge) the snapshot back into the original image.

No need if you freeze the filesystem first then rsync. but for this I recommend taking each time the whole file with rsync

Uh there’s no difference. If you freeze the system it can’t be used. This is to run on live systems so they remain running the whole time.

JaredBusch

@stacksofplates said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@stacksofplates said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch said in Unable to delete KVM snapshot:

@emad-r said in Unable to delete KVM snapshot:

@jaredbusch

Hi,

What is the reasoning for doing external snapshots and going against the default (where everything is saved in one file), I use virt but I love virt-manager and with that it always does the snap internally.

Any benefits of external aside being the ESXi way?

Cause it seems here that what is causing the issue, and I recall somewhere I read the qcow2 expects everything to be in 1 file.

I just read @stacksofplates basically what he says.

because you cannot copy the file off to backup when you use the internal snapshot

Interesting I dont take VM backup frequently, when I do I rsync the whole VM file.

rsync --progress --inplace -h -W /var/lib/libvirt/images/VM/centos7.0-clone.qcow2 [email protected]:/var/lib/libvirt/images/VM/centos7.0-clone.qcow2

This after the initial copy, if you are extra paranoid you can take the file everytime if you dont trust rsync algorithms.

But it is interesting this external snapshot approach, how much are the sizes of external snapshots roughly ? I know it depends but what are we dealing with here. 100 MB ?

You have to shut the VM off to just rsync the drive. This way you can leave the VM on.

You don’t copy the external snapshot. You take a snapshot and copy the backing store. Then blockcommit (merge) the snapshot back into the original image.

No need if you freeze the filesystem first then rsync. but for this I recommend taking each time the whole file with rsync

Uh there’s no difference. If you freeze the system it can’t be used. This is to run on live systems so they remain running the whole time.

I guess I need to look into your backup script thing one of these days.

The systems I want to backup are not stateful, like yours, but it will give me a starting point.