Using Rclone from a server to backup multiple volumes from different SSH hosts

DustinB3403

@black3dynamite said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

Use this to rclone your Mac clients to Fedora Server
https://rclone.org/sftp/

Use this to rclone your Fedora Server to B2
https://rclone.org/b2/

So I've already gone through this, maybe I'm just missing it.. will rereview.

marcinozga

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

DustinB3403

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

That's not at all accurate, I would have to install rclone on every workstation, have custom cron jobs on every workstation and then would have to validate that every workstation sync'd.

DustinB3403

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

DustinB3403

One option I considered is just installing Cyberduck on the workstations and having the user do the backup.

Of course that just won't be done and thus is off the table.

marcinozga

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

That's not at all accurate, I would have to install rclone on every workstation, have custom cron jobs on every workstation and then would have to validate that every workstation sync'd.

And all that you can do with Ansible.
Install rclone? https://docs.ansible.com/ansible/latest/modules/homebrew_module.html#homebrew-module
Setup cron? https://docs.ansible.com/ansible/latest/modules/cron_module.html#cron-module

marcinozga

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

And since when has cron been disabled in macOS?

DustinB3403

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

And since when has cron been disabled in macOS?

For quite some time, it's still there, but you have to enable it, grant it root access etc.

marcinozga

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

And since when has cron been disabled in macOS?

For quite some time, it's still there, but you have to enable it, grant it root access etc.

That's simply not true. You don't have to enable it. And you don't have to give it root access unless you're writing somewhere you shouldn't be in first place.

DustinB3403

Anyways I figured this out.

Having multiple rclone sources like

and running

Works

dafyre

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

Anyways I figured this out.

Having multiple rclone sources like

and running

Works

The /Volumes/<path to external> would change for each system, wouldn't it?

Could you back up to b2-rclone:/<username>/external ?

DustinB3403

@dafyre said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

Anyways I figured this out.

Having multiple rclone sources like

and running

Works

The /Volumes/<path to external> would change for each system, wouldn't it?

Could you back up to b2-rclone:/<username>/external ?

Possibly, but what I was thinking I would do is simply connect to the administrative user on each of these workstations, hit the volume that way. The Volume name may change (as spaces make everything more difficult). But yeah I think this'll work.

DustinB3403

To clarify for anyone else in the future, you need to have multiple apple-# sftp connections.

The b2 connections could also be custom as you can create multiple buckets and simply connect to that bucket via rclone.

DustinB3403

This is what the b2-rclone remote setting looks like for my needs, obviously this could be adjusted with the advanced configuration menu.

And this is the Apple workstation

DustinB3403

Ideally when this is all said and done, what I think would make the most sense is to create multiple buckets for each of these apple workstations.

So apple-bucket-1, apple-bucket-2 and so on.

From there, simply have cron run the sync commands for each of these and have it run hourly.

DustinB3403

Worth noting on this that while sync works, copy is the much safer option.

So

rclone copy source:/path dest:/path is what I'll likely end up using. In any case, the resulting solution still works.

Still need to tweak OSX a bit as it seems to kill the SSH session after some time for sporadic reasons - still testing but I may end up having to add ServerAliveInterval 120 to /etc/ssh/ssh_config to resolve that issue.