Need a script to cleanup a Backblaze B2 bucket

JaredBusch

Ok, so I am using B2 for the offsite storage of Veeam backups via the native Cloud Sync app from Synology in their units.

This part has, more or less, been working perfectly. You can see me talk about it in this thread.

Goal:

keep full backups files, *.vbk, for 13 months.
keep incremental backup files, *.vib for 32 days.

But I have a limitation from Backblaze that I need to overcome.
B2 buckets have a function called "Lifecycle Management" that you can use to automatically purge files from a bucket.

This would be great, but you cannot use a wildcarded to purge files. This is a huge limitation.

Support stated that files deleted by the local side would be hidden depending on the sync solution. Well, I don't want that because I want to retain the full backups for a year, but incremental backups for only 32 days on B2, but the local retention policy is different.

So that means I need to create a script to use the B2 API to hide or delete the files I want to remove from B2 in order to let B2's lifecycle rule then cleanup the bucket.

Here is what a typical subfolder in the bucket looks like. Each VM is in a separate folder when Veeam make the original backup, so everything on B2 is in subfolders.

So before I go screw around with this, does someone else know of, or have, a script for something like this?

Curtis

RClone might be able to do this: https://rclone.org/b2/

JaredBusch

@Curtis How? Rclone has no idea what my needed schedule is. Rclone is just a tool to send commands to B2 in this scenario.

Something needs to know/determine what files exist and then mark them hidden appropriately.

Emad R

@JaredBusch

If you can keep the backup locally, then rclone or b2 api can do this, by performing the rules locally then syncing it to the cloud.

If you cant keep them locally due to size, why not create cron rclone dummy rule to sync dummy file to that bucket, this will force it to delete b2 storage cause that dummy file is not existing, you will need to use rclone sync function and not copy.

JaredBusch

Why are you all hung up on using a backup tool like rclone? I have working solid backups. I don't need another backup system to be setup and maintained.

Using the B2 API is not terribly difficult I see no reason to do anything else. A script that pulls down the information, and then runs a simple date logic loop and tells the appropriate things to "delete" does not seem that complicated.

Feel free to convince me I am wrong, but so far I am not feeling it.

JaredBusch

@Emad-R said in Need a script to cleanup a Backblaze B2 bucket:

If you can keep the backup locally,

The scenario was clearly stated. The local and remote retention periods are different.

dbeato

I have had this limitation with B2 for a while and will be interesting to see what the outcome is on this scenario.

DustinB3403

I haven't played a ton with the b2 CLI, but are you able to hide the files you need to keep and then run your cleanup and then unhide those files with wildcards?

b2 hide-file <bucketName> <fileName>

Maybe something like

b2 hide-file Jared synology-5-*

Also this documentation may be useful to building an answer. I assume you've already seen it.

DustinB3403

Or maybe as an alternative, hide the files that are "old" and then remove those so you're only seeing the current backups.

JaredBusch

@DustinB3403 said in Need a script to cleanup a Backblaze B2 bucket:

Also this documentation may be useful to building an answer. I assume you've already seen it.

Yes, tha tis what I was looking at using, yes.

DustinB3403

Here's some additional information, specifically using rclone into b2.

Unsolved Need a script to cleanup a Backblaze B2 bucket