KVM and Back Ups
-
Okay, so after doing some research, I've found some excellent news regarding KVM and incremental backups. But it also comes with some bad news.
It looks like the ability and support to do incremental backups is fully integrated into QEMU (I think as of 2.5).
It uses what I keep seeing referred to as Bitmaps, or more specifically Dirty Bitmaps or Block Dirty Bitmaps.
The bad news, from the little research I've done so far, is that all this stuff is managed using JSON files, which would be done via the QMP (QEMU Machine Protocol). This basically tells me that the only way to do incremental backups with KVM, is if you are a software engineer and know how to build software that uses the QMP, or have the time to figure it all the hell out. It all seems relatively simple...
So, this means we're still stick using something that costs money... because I doubt anyone would build something like this open sourced when there's a lot of money to be made... unless Google or someone does it for internal use and decides to make it open source.
I spent some more time looking at things since writing everything above, and looks like you can get starting issuing QMP commands via telnet to start, and then by CLI after installing socat and rlwrap. Again, no idea what I'm looking here: https://wiki.qemu.org/Documentation/QMP, but perhaps this deserves some more looking into.
Maybe someone else can clear it up for me so I know what I'm looking at. I wish I had more time and programming skills... I'd definitely be into figuring this stuff out and putting it out there.
maybe SodiumSuite devs feel like being nice?
Anyways, here are my references:
https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs/bitmaps.html
https://wiki.qemu.org/Features/IncrementalBackup
https://www.linux-kvm.org/images/e/e6/Kvm-forum2017_backup.pdf
https://events.static.linuxfound.org/sites/events/files/slides/kvm2015_rh_light_44_vfinal.pdf
-
@tim_g
One problem is the dirty bitmapping in the API is push only so anything expecting an interactive API won't work. This means the backup system can't query the API for dirty bitmaps, only create them.(At least it was this way the last time I looked at it)
-
This is quite an old thread, but it discusses exactly what I am researching.
I use KVM (via Fedora) in my home lab, and currently, since I have no production workloads on it, I run file only backups via a couple scripts.
However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.
What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.
Is this still a good option: https://storware.eu/products/vprotect/
Is there something better?
Can something like ProxMox handle this? https://www.proxmox.com/en/proxmox-ve
-
Proxmox backups are always a full backup.
https://pve.proxmox.com/wiki/Backup_and_Restore -
@fuznutz04 said in KVM and Back Ups:
However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.
What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.You come back to the fundamental question of... why are you approaching backups this way?
Backups have never been something that we think of at the hypervisor level for decades, all of a sudden its the hot new "everyone has to do it" way. But it has all kinds of limitations and should only be a consideration for certain workloads, and only one of the options. Backups are always by workload, never by platform. It's fundamentally a flawed way to think of backups because it hides the scope and complexity of getting reliable backups and causes all kinds of problems.
Under no circumstances can anyone answer how you should take backups based on knowing your platform - no matter what hypervisor or OS you use, your backups have to be a decision based on your workloads.
-
@black3dynamite said in KVM and Back Ups:
Proxmox backups are always a full backup.
https://pve.proxmox.com/wiki/Backup_and_RestoreDo you use or have any experience using proxmox? Does/can it just run as a VM on the host?
-
@fuznutz04 said in KVM and Back Ups:
@black3dynamite said in KVM and Back Ups:
Proxmox backups are always a full backup.
https://pve.proxmox.com/wiki/Backup_and_RestoreDo you use or have any experience using proxmox? Does/can it just run as a VM on the host?
PM doesn't do anything special, you still need to deal with the backups on a workload basis.
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.
What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.You come back to the fundamental question of... why are you approaching backups this way?
Backups have never been something that we think of at the hypervisor level for decades, all of a sudden its the hot new "everyone has to do it" way. But it has all kinds of limitations and should only be a consideration for certain workloads, and only one of the options. Backups are always by workload, never by platform. It's fundamentally a flawed way to think of backups because it hides the scope and complexity of getting reliable backups and causes all kinds of problems.
Under no circumstances can anyone answer how you should take backups based on knowing your platform - no matter what hypervisor or OS you use, your backups have to be a decision based on your workloads.
So in this case, Iād have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.
What Iām after is an āinstantā type restore. For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
-
@fuznutz04 said in KVM and Back Ups:
So in this case, Iād have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.
That's all that you want. Just the OS level backups.
-
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
-
For most workloads, like the ones that you have, you can just shut down the VM, and take a totally safe backup. Then you only need to take application level backups and do a restore from them for the latest data. Even ProxMox for their "safe" backups powers down your VMs first, that's how they handle it.
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
-
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
-
To answer this, what I did for my lab is setup UrBackup and just installed the agent into each of my VM's.
It works and is simple enough to restore from.
-
It's for this kind of stuff that ZFS and BtrFS has been made popular, because they handle this very well.
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
-
@DustinB3403 said in KVM and Back Ups:
To answer this, what I did for my lab is setup UrBackup and just installed the agent into each of my VM's.
It works and is simple enough to restore from.
Right, that's often the better option. It's safer than hypervisor level backups, and if set up well, is just as fast and more flexible.
-
@DustinB3403 said in KVM and Back Ups:
To answer this, what I did for my lab is setup UrBackup and just installed the agent into each of my VM's.
It works and is simple enough to restore from.
I have to check that out tonight in my lab
-
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
LVM, ZFS, BtrFS are all fine. I've not used this but here is a script to do LVM backups...
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. Thatās what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
LVM, ZFS, BtrFS are all fine. I've not used this but here is a script to do LVM backups...
Awesome. Iāve used ZFS in the past, but it was on a freeNas box I was testing. Seemed pretty good at the time (zfs)
