ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Continuous asynch file replication in linux?

    Scheduled Pinned Locked Moved IT Business
    18 Posts 5 Posters 1.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DustinB3403D
      DustinB3403
      last edited by

      DRBD comes to mind.

      scottalanmillerS 1 Reply Last reply Reply Quote 1
      • 1
        1337
        last edited by

        Sorry, posted in the wrong section. Should be under IT discussion.

        1 Reply Last reply Reply Quote 0
        • DustinB3403D
          DustinB3403
          last edited by

          Isn't rsync async by design?

          To ask a second question, are you looking for something that operates at the storage layer (DRBD-esk) or at the software layer like Rsync?

          At the software layer I can't think of anything specific off hand.

          1 scottalanmillerS 2 Replies Last reply Reply Quote 0
          • 1
            1337 @DustinB3403
            last edited by 1337

            @DustinB3403 said in Continuous asynch file replication in linux?:

            Isn't rsync async by design?

            To ask a second question, are you looking for something that operates at the storage layer (DRBD-esk) or at the software layer like Rsync?

            At the software layer I can't think of anything specific off hand.

            I was thinking keeping lots of small files synced between a bunch of servers. Any one could change any file at any time with the expectation that it will show up on the other servers as soon as possible. Writes are low volume though but should be low latency and not take 10 minutes.

            That's why I assumed something similar to rsync but not running on a cron job.

            DustinB3403D 1 Reply Last reply Reply Quote 0
            • DustinB3403D
              DustinB3403 @1337
              last edited by

              @Pete-S said in Continuous asynch file replication in linux?:

              @DustinB3403 said in Continuous asynch file replication in linux?:

              Isn't rsync async by design?

              To ask a second question, are you looking for something that operates at the storage layer (DRBD-esk) or at the software layer like Rsync?

              At the software layer I can't think of anything specific off hand.

              I was thinking keeping lots of small files synced between a bunch of servers. Any one could change any file at any time with the expectation that it will show up on the other servers as soon as possible. Writes are low volume though but should be low latency and not take 10 minutes.

              So all distinct standalone servers?

              You're probably wanting to use Rsync.

              1 1 Reply Last reply Reply Quote 0
              • 1
                1337 @DustinB3403
                last edited by

                @DustinB3403 said in Continuous asynch file replication in linux?:

                @Pete-S said in Continuous asynch file replication in linux?:

                @DustinB3403 said in Continuous asynch file replication in linux?:

                Isn't rsync async by design?

                To ask a second question, are you looking for something that operates at the storage layer (DRBD-esk) or at the software layer like Rsync?

                At the software layer I can't think of anything specific off hand.

                I was thinking keeping lots of small files synced between a bunch of servers. Any one could change any file at any time with the expectation that it will show up on the other servers as soon as possible. Writes are low volume though but should be low latency and not take 10 minutes.

                So all distinct standalone servers?

                You're probably wanting to use Rsync.

                Yes, all distinct standalone servers.

                DustinB3403D 1 Reply Last reply Reply Quote 0
                • DustinB3403D
                  DustinB3403 @1337
                  last edited by

                  @Pete-S said in Continuous asynch file replication in linux?:

                  @DustinB3403 said in Continuous asynch file replication in linux?:

                  @Pete-S said in Continuous asynch file replication in linux?:

                  @DustinB3403 said in Continuous asynch file replication in linux?:

                  Isn't rsync async by design?

                  To ask a second question, are you looking for something that operates at the storage layer (DRBD-esk) or at the software layer like Rsync?

                  At the software layer I can't think of anything specific off hand.

                  I was thinking keeping lots of small files synced between a bunch of servers. Any one could change any file at any time with the expectation that it will show up on the other servers as soon as possible. Writes are low volume though but should be low latency and not take 10 minutes.

                  So all distinct standalone servers?

                  You're probably wanting to use Rsync.

                  Yes, all distinct standalone servers.

                  Yeah, Rsync may be the most direct approach. What I see maybe being an issue is if the second (3rd or 4th) servers say "oh new file" and start to sync that back to the original source.

                  I'm sure it would work fine, generally speaking.

                  1 Reply Last reply Reply Quote 1
                  • black3dynamiteB
                    black3dynamite
                    last edited by

                    Use systemd to monitor directories and files
                    https://www.putorius.net/systemd-path-units.html

                    Syncthing
                    https://syncthing.net/
                    ed5a18ec-b998-4217-946a-9f83c092349f-image.png

                    scottalanmillerS 1 Reply Last reply Reply Quote 4
                    • D
                      dave_c
                      last edited by

                      I would say:
                      lsyncd for user mode, simple, rsync based solution: https://github.com/axkibe/lsyncd
                      csync2 for multi-host synchronization with conflict detection: https://github.com/LINBIT/csync2
                      MARS for a kernel side solution (DRBD alternative): http://schoebel.github.io/mars/

                      1 Reply Last reply Reply Quote 1
                      • scottalanmillerS
                        scottalanmiller @DustinB3403
                        last edited by

                        @DustinB3403 said in Continuous asynch file replication in linux?:

                        DRBD comes to mind.

                        That's the standard.

                        D 1 Reply Last reply Reply Quote 0
                        • scottalanmillerS
                          scottalanmiller @DustinB3403
                          last edited by

                          @DustinB3403 said in Continuous asynch file replication in linux?:

                          Isn't rsync async by design?

                          Yes. Rsync can't do full sync, only async. But it's not automatic.

                          1 Reply Last reply Reply Quote 0
                          • scottalanmillerS
                            scottalanmiller @black3dynamite
                            last edited by

                            @black3dynamite said in Continuous asynch file replication in linux?:

                            Use systemd to monitor directories and files
                            https://www.putorius.net/systemd-path-units.html

                            Syncthing
                            https://syncthing.net/
                            ed5a18ec-b998-4217-946a-9f83c092349f-image.png

                            This is probably the best option.

                            1 Reply Last reply Reply Quote 0
                            • D
                              dave_c @scottalanmiller
                              last edited by

                              @scottalanmiller
                              Is it for asynchronous replication? I know it is for synchronous.

                              scottalanmillerS 1 Reply Last reply Reply Quote 0
                              • scottalanmillerS
                                scottalanmiller @dave_c
                                last edited by

                                @dave_c said in Continuous asynch file replication in linux?:

                                @scottalanmiller
                                Is it for asynchronous replication? I know it is for synchronous.

                                Which tool?

                                D 1 Reply Last reply Reply Quote 0
                                • D
                                  dave_c @scottalanmiller
                                  last edited by

                                  @scottalanmiller said in Continuous asynch file replication in linux?:

                                  @dave_c said in Continuous asynch file replication in linux?:

                                  @scottalanmiller
                                  Is it for asynchronous replication? I know it is for synchronous.

                                  Which tool?
                                  DRBD as the standard asynchronous replication option for Linux

                                  scottalanmillerS 1 Reply Last reply Reply Quote 0
                                  • scottalanmillerS
                                    scottalanmiller @dave_c
                                    last edited by

                                    @dave_c said in Continuous asynch file replication in linux?:

                                    @scottalanmiller said in Continuous asynch file replication in linux?:

                                    @dave_c said in Continuous asynch file replication in linux?:

                                    @scottalanmiller
                                    Is it for asynchronous replication? I know it is for synchronous.

                                    Which tool?
                                    DRBD as the standard asynchronous replication option for Linux

                                    DRBD is both sync and async. Most uses of it are async.

                                    D 1 Reply Last reply Reply Quote 0
                                    • D
                                      dave_c @scottalanmiller
                                      last edited by

                                      @scottalanmiller
                                      Thanks for the clarification

                                      1 Reply Last reply Reply Quote 0
                                      • 1 / 1
                                      • First post
                                        Last post