Anyone backing up a file server with 13 million plus files?
-
@stacksofplates said in Anyone backing up a file server with 13 million plus files?:
We use have a mix of Backup Exec and NetBackup. We have ~500 TB backing up through Backup Exec but it's pushing the limits of that software so we are switching to NetBackup (what we use everywhere else). I think we can do around 7TB a night for incrementals. Not sure how long a full backup takes.
How many files?
I'm asking that specifically because I have another server with 14.5 TB and it does a full backup in less than 2 days... very few files.
This other server (the one with 13m files) takes almost a week just to scan the file system before it even starts moving data... and it's half capacity of the one above.
In my case, and OP, it's more the number of files than the total size.
-
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@stacksofplates said in Anyone backing up a file server with 13 million plus files?:
We use have a mix of Backup Exec and NetBackup. We have ~500 TB backing up through Backup Exec but it's pushing the limits of that software so we are switching to NetBackup (what we use everywhere else). I think we can do around 7TB a night for incrementals. Not sure how long a full backup takes.
How many files?
I'm asking that specifically because I have another server with 14.5 TB and it does a full backup in less than 2 days... very few files.
This other server (the one with 13m files) takes almost a week just to scan the file system before it even starts moving data... and it's half capacity of the one above.
In my case, and OP, it's more the number of files than the total size.
I'll have to look. I don't think I've ever actually tried to find out. It's CFD and mechanical/electrical modelling software which creates a ton of files so there should be quite a few.
-
@tim_g for less than 100Tb, you could still use a traditional file server and a CBT backup like Veeam on VMware. Over 100Tb, you should really look into object storage.
-
@francesco-provino said in Anyone backing up a file server with 13 million plus files?:
@tim_g for less than 100Tb, you could still use a traditional file server and a CBT backup like Veeam on VMware. Over 100Tb, you should really look into object storage.
I'm thinking different backup software will resolve the main issue, but I wanted to see if anyone else has a similar file server they are backing up with something that's working well.
I was thinking of giving Veeam a try, even if it's just a one-off, hopefully something that can integrate into a full enterprise backup solution later.
-
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
-
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
7TB isn't bad. 13m files is a bit tough to handle. This is where hypervisor / platform agentless backups have an advantage. If they skip the file list, it's that much easier. But if they read the file list anyway, it is back to square one.
-
Veeam doesn't care about files, it's block level backup isn't it?
-
Assuming the VM is getting backed-up and not files/folders or drives.
-
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
-
Currently, backup consists of bi-weekly full tape backups, and daily incremental tape backup.
This is how it's been since before I started.
We want to do on-prem backups now, and then move the on-prem backups to tape, instead of how it's being done now.
But as it stands, the tape backup software is doing a much better job than the on-prem backup software (Windows Server Backup, because it's included in the OS and free.)
WSB isn't working for this amount of files. I need to use something else. I can't do weekly full on-prem backups if it takes two weeks to back up lol.
I'll just have to start going through them to test. I'll start with Veeam I suppose, unless someone has experienced this using something else.
-
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
Gotcha. . . anything that can do block level, while skipping reading every file is what you're looking for then. While also offering the ability to perform per file recovery (I'm assuming).
-
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
Gotcha. . . anything that can do block level, while skipping reading every file is what you're looking for then. While also offering the ability to perform per file recovery (I'm assuming).
Good point.
I haven't paid much attention to that fact, but which backup softwares do it like that?
-
Veeam Windows Agent does block level backup but if it is being linked to an existing Veeam B&R Repository, there might be search indexes that are built during backup. I remember reading some best-practice documentation about their catalog but only related to VM backups. Not sure if you're poiting Veeam Windows Agent to a B&R or just to storage.
-
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
Gotcha. . . anything that can do block level, while skipping reading every file is what you're looking for then. While also offering the ability to perform per file recovery (I'm assuming).
Good point.
I haven't paid much attention to that fact, but which backup softwares do it like that?
Scale HC3's built in backup will do that without looking at the files
-
@scottalanmiller said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
Gotcha. . . anything that can do block level, while skipping reading every file is what you're looking for then. While also offering the ability to perform per file recovery (I'm assuming).
Good point.
I haven't paid much attention to that fact, but which backup softwares do it like that?
Scale HC3's built in backup will do that without looking at the files
Can it do individual file restores?
-
@scottalanmiller said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
Gotcha. . . anything that can do block level, while skipping reading every file is what you're looking for then. While also offering the ability to perform per file recovery (I'm assuming).
Good point.
I haven't paid much attention to that fact, but which backup softwares do it like that?
Scale HC3's built in backup will do that without looking at the files
Sure, but a $200 Veeam Agent for Windows Server is much more likely than $30k + a migration LOL
-
@dashrender said in Anyone backing up a file server with 13 million plus files?:
@scottalanmiller said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
@tim_g said in Anyone backing up a file server with 13 million plus files?:
@dustinb3403 said in Anyone backing up a file server with 13 million plus files?:
What hypervisor? 7TB and 13 Million files doesn't seem that excessive for pretty much any solution today.
It's a physical server, with an attached MD1000 for on-prem backup.
Gotcha. . . anything that can do block level, while skipping reading every file is what you're looking for then. While also offering the ability to perform per file recovery (I'm assuming).
Good point.
I haven't paid much attention to that fact, but which backup softwares do it like that?
Scale HC3's built in backup will do that without looking at the files
Can it do individual file restores?
https://mangolassi.it/topic/16110/restoring-files-and-folders-out-of-scale-hc3-vm-snapshots
-
Veeam likely would work well here, but I'm not sure if it actually skips reading and creating a file index though. . .
I honestly don't know of any solution that would do this. . . agent-wise.
-
I'm going to test it with the free version just to see how it handles it.
If it works out, we'll buy it. $200 is nothing at this point... and if it proves itself, it may help with going all Veeam later (hopefully).
-
Veeam test is on-going.
I started it over an hour ago, and so far, it's over a week ahead of where the other backup software would be...
It's going to back it all up into a .vbk file. So after it's complete, I'll see what can be done with it tape-wise, file restore wise, etc.
I'm not exactly sure "how" it's doing the backup... but I did see from Resmon it read the Master File Table. Much more efficient than scanning every damn file...
