ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Storage and Data Locality

    IT Discussion
    drbd gluster storage infrastructure san data locality
    10
    30
    3.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dafyre
      last edited by

      So I've been piddling about with an idea in my head and I want holes poked in it.

      Essentially what I want to build is a storage setup between my Home (60 Mbit download, 5 Meg Upload) and my hosted server in France (100 Mbit both ways). To start with, this would be used for keeping files synchronized between Home and France, but eventually I'd like to use it to keep my Home VMs replicated to France, and my France VMs replicated to my home.

      I'm not looking or "Instant replication" or anything like that. I just want the replication to happen...eventually. I know this could be done with rsync for static files, but I want to do VMs as well. I was just curious as to what other methods could be done.

      I thought about Gluster or DRBD, but then data locality becomes an issue, right?

      My VMs done with KVM on both systems.

      Ideas, comments, tell me I'm crazy... Just looking for input and thoughts or other ideas I can try. This is a lab environment, so I'm up for pretty much anything within reason.

      1 Reply Last reply Reply Quote 0
      • S
        scottalanmiller
        last edited by

        If your latency is low enough, @StarWind_Software can do this now. It's a newer feature of theirs. They no longer require that their cluster nodes be at local speeds.

        1 Reply Last reply Reply Quote 1
        • S
          scottalanmiller
          last edited by

          Pinging @kooler and @Stuka

          1 Reply Last reply Reply Quote 0
          • D
            dafyre
            last edited by

            Current latency is between 200 and 250ms over ZT.

            S 1 Reply Last reply Reply Quote 0
            • S
              scottalanmiller @dafyre
              last edited by

              @dafyre said in Storage and Data Locality:

              Current latency is between 200 and 250ms over ZT.

              That's a lot of latency, that'll make things hard. But an async replica is an option.

              D 1 Reply Last reply Reply Quote 0
              • D
                dbeato
                last edited by

                If I can offer any input, please do not use DRBD, it can corrupt your data pretty easily.

                D 1 Reply Last reply Reply Quote 2
                • D
                  dafyre @scottalanmiller
                  last edited by

                  @scottalanmiller said in Storage and Data Locality:

                  @dafyre said in Storage and Data Locality:

                  Current latency is between 200 and 250ms over ZT.

                  That's a lot of latency, that'll make things hard. But an async replica is an option.

                  That's the term I was looking for. Async replica -- that's what I'm going for... In the event that my home network burns down, the goal is to be able to spin up anything I need in France.

                  1 Reply Last reply Reply Quote 1
                  • D
                    dafyre @dbeato
                    last edited by

                    @dbeato said in Storage and Data Locality:

                    If I can offer any input, please do not use DRBD, it can corrupt your data pretty easily.

                    Thanks for the forewarning -- that was one of the things I was considering.

                    1 Reply Last reply Reply Quote 0
                    • N
                      NerdyDad
                      last edited by

                      I was thinking Veeam B&R but don't think you want to go to either Hyper-V or ESXi and dont have piles of cash just laying around, do you?

                      D 1 Reply Last reply Reply Quote 0
                      • D
                        dafyre @NerdyDad
                        last edited by

                        @nerdydad said in Storage and Data Locality:

                        I was thinking Veeam B&R but don't think you want to go to either Hyper-V or ESXi and dont have piles of cash just laying around, do you?

                        Yeah, this is a personal lab, so I don't have piles of cash lying around. Since my hosted server is limited in OS choices, I'm sticking with KVM. (ESXi and Hyper-V are not offered).

                        1 Reply Last reply Reply Quote 0
                        • A
                          Alex Sage
                          last edited by

                          rsync?

                          D 1 Reply Last reply Reply Quote -1
                          • T
                            travisdh1
                            last edited by

                            I think it's just assumed that if your using KVM, that your also competent enough to write your own backup scripts. If you already have a backup script setup, just create a local backup and then rsync those files to the other box.

                            If you need a backup script still, well, I've been meaning to work on one, you just might be the motivation to get it done.

                            J 1 Reply Last reply Reply Quote 0
                            • J
                              JaredBusch @travisdh1
                              last edited by

                              @travisdh1 said in Storage and Data Locality:

                              I think it's just assumed that if your using KVM, that your also competent enough to write your own backup scripts. If you already have a backup script setup, just create a local backup and then rsync those files to the other box.

                              This is just a steaming pile of shit.

                              Anyone can backup any system easy enough. KVM has nothing to do with it.

                              What you cannot do with a simple script is do differential or incremental backups such as Veeam and Unitrends do.
                              That is why backups are hard, and why there are not bazillions of backup solutions out there.

                              S 1 Reply Last reply Reply Quote 0
                              • S
                                scottalanmiller @JaredBusch
                                last edited by

                                @jaredbusch said in Storage and Data Locality:

                                @travisdh1 said in Storage and Data Locality:

                                I think it's just assumed that if your using KVM, that your also competent enough to write your own backup scripts. If you already have a backup script setup, just create a local backup and then rsync those files to the other box.

                                This is just a steaming pile of shit.

                                Anyone can backup any system easy enough. KVM has nothing to do with it.

                                What you cannot do with a simple script is do differential or incremental backups such as Veeam and Unitrends do.
                                That is why backups are hard, and why there are not bazillions of backup solutions out there.

                                Also, usable automation. It's easy to automate a straight backup copy job; it's very hard to automate a system for restoring a specific file from a specific point in time.

                                1 Reply Last reply Reply Quote 0
                                • stacksofplatesS
                                  stacksofplates
                                  last edited by

                                  I’ve used rsnapshot before and it works really well. Uses hard links to save space. Won’t help with your VM disks but will help with files.

                                  1 Reply Last reply Reply Quote 1
                                  • ObsolesceO
                                    Obsolesce
                                    last edited by

                                    ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.

                                    ReaR works great on stateless VMs, really easy and quick to recover.

                                    stacksofplatesS 1 Reply Last reply Reply Quote 1
                                    • stacksofplatesS
                                      stacksofplates @Obsolesce
                                      last edited by

                                      @tim_g said in Storage and Data Locality:

                                      ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.

                                      ReaR works great on stateless VMs, really easy and quick to recover.

                                      Why are you backing up stateless machines?

                                      ObsolesceO 1 Reply Last reply Reply Quote 2
                                      • ObsolesceO
                                        Obsolesce @stacksofplates
                                        last edited by

                                        @stacksofplates said in Storage and Data Locality:

                                        @tim_g said in Storage and Data Locality:

                                        ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.

                                        ReaR works great on stateless VMs, really easy and quick to recover.

                                        Why are you backing up stateless machines?

                                        Policy and DR.

                                        S black3dynamiteB 2 Replies Last reply Reply Quote 0
                                        • S
                                          scottalanmiller @Obsolesce
                                          last edited by

                                          @tim_g said in Storage and Data Locality:

                                          @stacksofplates said in Storage and Data Locality:

                                          @tim_g said in Storage and Data Locality:

                                          ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.

                                          ReaR works great on stateless VMs, really easy and quick to recover.

                                          Why are you backing up stateless machines?

                                          Policy and DR.

                                          Don't need it for DR. Then the question becomes, why did someone make that policy?

                                          ObsolesceO 1 Reply Last reply Reply Quote 0
                                          • black3dynamiteB
                                            black3dynamite @Obsolesce
                                            last edited by

                                            @tim_g said in Storage and Data Locality:

                                            @stacksofplates said in Storage and Data Locality:

                                            @tim_g said in Storage and Data Locality:

                                            ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.

                                            ReaR works great on stateless VMs, really easy and quick to recover.

                                            Why are you backing up stateless machines?

                                            Policy and DR.

                                            With stateless machines, wouldn't it be easier to automate the deployment, especially if your data is stored on another virtual disk or storage appliance?

                                            S 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post