Offsite Backup Solution Needed
-
@Sparkum said:
@MattSpeller said:
For ~200GB of changes you're in the butter zone for LTO5/6. Just make sure if you go with 6 you can feed it fast enough as they work best with a full buffer to avoid running out of data mid write.
I personally wondering if the ~200GB number I'm coming up with is more how Backup Exec does its backups, looking closely I cant fathem why certain servers have the growth they are showing.
You can have nearly zero total size change but files themselves could change drastically internally, so that needs to be backed up.
I'm not sure how DB's work, but let's assume if you change even one bit in a DB you have to backup the whole thing - so if it's a 100 GB db, and you change 1 bit, you have 100 GB to backup now.. just a lame example.
-
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
-
@JaredBusch said:
@Sparkum said:
@JaredBusch Does replication do any sort of snapshot?
Of course it does. That is how all backup mechanisms work.
This is what VMWare shows on the source host when Veeam 9 runs a replication job .
and here is what it looks like on the destinaiton side.
Ya thats just where my concern is.
Maybe what happened with the VM filling up and crashing is something stupid that I could fix with a simple phone call to Veeam, but like every company we have certain servers that cant have downtime during business hours, so if something like that happened on one of our critical servers shit will hit the fan so fast.
-
@Dashrender said:
@Sparkum said:
@MattSpeller said:
For ~200GB of changes you're in the butter zone for LTO5/6. Just make sure if you go with 6 you can feed it fast enough as they work best with a full buffer to avoid running out of data mid write.
I personally wondering if the ~200GB number I'm coming up with is more how Backup Exec does its backups, looking closely I cant fathem why certain servers have the growth they are showing.
You can have nearly zero total size change but files themselves could change drastically internally, so that needs to be backed up.
I'm not sure how DB's work, but let's assume if you change even one bit in a DB you have to backup the whole thing - so if it's a 100 GB db, and you change 1 bit, you have 100 GB to backup now.. just a lame example.
Ya for sure, and we can some "kinda silly" things set up that would need to change like our OMG VIP sql database, backs up as well as does a DB dumb onto our backup server which then backs up again, so the change on our backup server is huge (didnt take that into account in the 100-200GB)
-
@Sparkum said:
Ya thats just where my concern is.
Maybe what happened with the VM filling up and crashing is something stupid that I could fix with a simple phone call to Veeam, but like every company we have certain servers that cant have downtime during business hours, so if something like that happened on one of our critical servers shit will hit the fan so fast.
I realize you had a problem with a snapshot, but that is how every single VM backup solution works and is a native function of every single hypervisor out there. You should contact support for either the hypervisor or the backup vendor.
-
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
-
@JaredBusch said:
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
correct, but it's a two step process.
- create backup - a) create snap b) copy data to backup repository c) delete snap
- replicate data from repository to remote location
As long as step 1 is done completely locally, you shouldn't have a problem with your snaps.
What we still don't know - and really is important before providing any advice of real value, is why the snaps caused the server to crash - if it even was really the snap that cause it.
i.e. did it run out of disk space? out of RAM(though that doesn't make sense) CPU overload, etc, etc, etc....
One possible story behind the crash - the snap was taken - the copy process starts but takes forever, the local VM host runs out of disk space - VM Host crashes.
But this is only one of many possible situations. In this situation local NAS for repository would solve the problem.
-
@Dashrender said:
@JaredBusch said:
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
correct, but it's a two step process.
- create backup - a) create snap b) copy data to backup repository c) delete snap
- replicate data from repository to remote location
As long as step 1 is done completely locally, you shouldn't have a problem with your snaps.
What we still don't know - and really is important before providing any advice of real value, is why the snaps caused the server to crash - if it even was really the snap that cause it.
i.e. did it run out of disk space? out of RAM(though that doesn't make sense) CPU overload, etc, etc, etc....
One possible story behind the crash - the snap was taken - the copy process starts but takes forever, the local VM host runs out of disk space - VM Host crashes.
But this is only one of many possible situations. In this situation local NAS for repository would solve the problem.
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
-
@Dashrender said:
@JaredBusch said:
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
correct, but it's a two step process.
- create backup - a) create snap b) copy data to backup repository c) delete snap
- replicate data from repository to remote location
As long as step 1 is done completely locally, you shouldn't have a problem with your snaps.
What we still don't know - and really is important before providing any advice of real value, is why the snaps caused the server to crash - if it even was really the snap that cause it.
i.e. did it run out of disk space? out of RAM(though that doesn't make sense) CPU overload, etc, etc, etc....
One possible story behind the crash - the snap was taken - the copy process starts but takes forever, the local VM host runs out of disk space - VM Host crashes.
But this is only one of many possible situations. In this situation local NAS for repository would solve the problem.
I would agree that the snapshot issue would be the first thing to tackle. I would avoid backupexec like the plague and can't recommend Veeam enough. I would also upgrade the connection at the remote site. Then decide how you want to get the data over there- whether you are using replication or backup copy job (both are features in Veeam).
-
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
-
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
-
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
I thought you said Veeam was over zealous? what did you mean by that?
-
@Dashrender said:
@JaredBusch said:
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
correct, but it's a two step process.
- create backup - a) create snap b) copy data to backup repository c) delete snap
- replicate data from repository to remote location
As long as step 1 is done completely locally, you shouldn't have a problem with your snaps.
What we still don't know - and really is important before providing any advice of real value, is why the snaps caused the server to crash - if it even was really the snap that cause it.
i.e. did it run out of disk space? out of RAM(though that doesn't make sense) CPU overload, etc, etc, etc....
One possible story behind the crash - the snap was taken - the copy process starts but takes forever, the local VM host runs out of disk space - VM Host crashes.
But this is only one of many possible situations. In this situation local NAS for repository would solve the problem.
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. I have no place doing this with a critical shortage of drive space though as is sounds @Sparkum does.
But my understanding of it is that it uses the data collected fromthe backup job to replicate the changes and then does the merge remotely. Just as it does the local merge when you hit the incremental limit set in the normal backup job.
-
A snap works (as I understand it) by creating a new VMDK file and setting the original VMDK to read only. All changes are made to the new VMDK - this new file will grow and grow as needed until you delete it. Deleting it merges all of the changes in the new file into the old file, and sets the old file to read/write again.
-
This is how Veeam replication works - https://www.veeam.com/university-course/backup-replication-how-it-works.html
This is for v7 but I am pretty sure that it is still the same in v9
Edit: you will have to click through several slides to get to the actual replication section.
-
@JaredBusch said:
But my understanding of it is that it uses the data collected fromthe backup job to replicate the changes and then does the merge remotely. Just as it does the local merge when you hit the incremental limit set in the normal backup job.
except that the merge is done on the backup repository storage, not the VM storage in the local non hypervisor replication scenario.
-
@JaredBusch said:
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. I have no place doing this with a critical shortage of drive space though as is sounds @Sparkum does.
Awww - OK now we're seeing the differences. JB is doing replication through the hypervisor,
I'm talking about doing replication through the backup software.The advantage of doing it through the backup software is that the VM host in question is cut out of the loop entirely during the replication process.
-
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
Now, in the case where you are running only 1 backup and you still run out of space - that just shows you how much change your doing. How much free space is there on the datastore of the VM host that crashed?
You might have to start by growing that datastore.
-
@Dashrender I bet it was during the initial full backup...
-
@Dashrender said:
@JaredBusch said:
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. I have no place doing this with a critical shortage of drive space though as is sounds @Sparkum does.
Awww - OK now we're seeing the differences. JB is doing replication through the hypervisor,
I'm talking about doing replication through the backup software.No I was not. I clearly stated I used Veeam to make replicas in the hypervisor. Veeam is the backup software.
