ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Using Rclone from a server to backup multiple volumes from different SSH hosts

    Scheduled Pinned Locked Moved Solved IT Discussion
    rcloneb2backupautomationusers
    20 Posts 4 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DustinB3403D
      DustinB3403 @black3dynamite
      last edited by

      @black3dynamite said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

      Use this to rclone your Mac clients to Fedora Server
      https://rclone.org/sftp/

      Use this to rclone your Fedora Server to B2
      https://rclone.org/b2/

      So I've already gone through this, maybe I'm just missing it.. will rereview.

      1 Reply Last reply Reply Quote 0
      • M
        marcinozga @DustinB3403
        last edited by

        @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

        @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

        @DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

        This I considered, but would involve tinkering with each workstation.

        That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

        DustinB3403D 1 Reply Last reply Reply Quote 0
        • DustinB3403D
          DustinB3403 @marcinozga
          last edited by

          @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

          @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

          @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

          @DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

          This I considered, but would involve tinkering with each workstation.

          That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

          That's not at all accurate, I would have to install rclone on every workstation, have custom cron jobs on every workstation and then would have to validate that every workstation sync'd.

          M 1 Reply Last reply Reply Quote 0
          • DustinB3403D
            DustinB3403
            last edited by

            In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

            Centrally managing this makes way more sense rather than the ad hoc approach.

            M 1 Reply Last reply Reply Quote 0
            • DustinB3403D
              DustinB3403
              last edited by

              One option I considered is just installing Cyberduck on the workstations and having the user do the backup.

              Of course that just won't be done and thus is off the table.

              1 Reply Last reply Reply Quote 0
              • M
                marcinozga @DustinB3403
                last edited by marcinozga

                @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                @DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

                This I considered, but would involve tinkering with each workstation.

                That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

                That's not at all accurate, I would have to install rclone on every workstation, have custom cron jobs on every workstation and then would have to validate that every workstation sync'd.

                And all that you can do with Ansible.
                Install rclone? https://docs.ansible.com/ansible/latest/modules/homebrew_module.html#homebrew-module
                Setup cron? https://docs.ansible.com/ansible/latest/modules/cron_module.html#cron-module

                1 Reply Last reply Reply Quote 0
                • M
                  marcinozga @DustinB3403
                  last edited by marcinozga

                  @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                  In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

                  Centrally managing this makes way more sense rather than the ad hoc approach.

                  And since when has cron been disabled in macOS?

                  DustinB3403D 1 Reply Last reply Reply Quote 0
                  • DustinB3403D
                    DustinB3403 @marcinozga
                    last edited by

                    @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                    @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                    In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

                    Centrally managing this makes way more sense rather than the ad hoc approach.

                    And since when has cron been disabled in macOS?

                    For quite some time, it's still there, but you have to enable it, grant it root access etc.

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      marcinozga @DustinB3403
                      last edited by

                      @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                      @marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                      @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                      In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

                      Centrally managing this makes way more sense rather than the ad hoc approach.

                      And since when has cron been disabled in macOS?

                      For quite some time, it's still there, but you have to enable it, grant it root access etc.

                      That's simply not true. You don't have to enable it. And you don't have to give it root access unless you're writing somewhere you shouldn't be in first place.

                      1 Reply Last reply Reply Quote 0
                      • DustinB3403D
                        DustinB3403
                        last edited by

                        Anyways I figured this out.

                        Having multiple rclone sources like
                        SKtlxrr.png

                        and running

                        b9469d45-8af9-4407-b185-7105a79a7557-image.png

                        Works

                        dafyreD 1 Reply Last reply Reply Quote 2
                        • dafyreD
                          dafyre @DustinB3403
                          last edited by

                          @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                          Anyways I figured this out.

                          Having multiple rclone sources like
                          SKtlxrr.png

                          and running

                          b9469d45-8af9-4407-b185-7105a79a7557-image.png

                          Works

                          The /Volumes/<path to external> would change for each system, wouldn't it?

                          Could you back up to b2-rclone:/<username>/external ?

                          DustinB3403D 1 Reply Last reply Reply Quote 0
                          • DustinB3403D
                            DustinB3403 @dafyre
                            last edited by

                            @dafyre said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                            @DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

                            Anyways I figured this out.

                            Having multiple rclone sources like
                            SKtlxrr.png

                            and running

                            b9469d45-8af9-4407-b185-7105a79a7557-image.png

                            Works

                            The /Volumes/<path to external> would change for each system, wouldn't it?

                            Could you back up to b2-rclone:/<username>/external ?

                            Possibly, but what I was thinking I would do is simply connect to the administrative user on each of these workstations, hit the volume that way. The Volume name may change (as spaces make everything more difficult). But yeah I think this'll work.

                            1 Reply Last reply Reply Quote 0
                            • DustinB3403D
                              DustinB3403
                              last edited by

                              To clarify for anyone else in the future, you need to have multiple apple-# sftp connections.

                              The b2 connections could also be custom as you can create multiple buckets and simply connect to that bucket via rclone.

                              1 Reply Last reply Reply Quote 0
                              • DustinB3403D
                                DustinB3403
                                last edited by

                                This is what the b2-rclone remote setting looks like for my needs, obviously this could be adjusted with the advanced configuration menu.

                                sj858rr.png

                                And this is the Apple workstation

                                jGytpzZ.png

                                1 Reply Last reply Reply Quote 0
                                • DustinB3403D
                                  DustinB3403
                                  last edited by

                                  Ideally when this is all said and done, what I think would make the most sense is to create multiple buckets for each of these apple workstations.

                                  So apple-bucket-1, apple-bucket-2 and so on.

                                  From there, simply have cron run the sync commands for each of these and have it run hourly.

                                  1 Reply Last reply Reply Quote 0
                                  • DustinB3403D
                                    DustinB3403
                                    last edited by DustinB3403

                                    Worth noting on this that while sync works, copy is the much safer option.

                                    So

                                    rclone copy source:/path dest:/path is what I'll likely end up using. In any case, the resulting solution still works.

                                    Still need to tweak OSX a bit as it seems to kill the SSH session after some time for sporadic reasons - still testing but I may end up having to add ServerAliveInterval 120 to /etc/ssh/ssh_config to resolve that issue.

                                    1 Reply Last reply Reply Quote 0
                                    • 1 / 1
                                    • First post
                                      Last post