BitTorrent/P2P technology for distributed file transfer of large files?
-
Is it possible to use bittorrent or other P2P technology to transfer large files between a number of servers in different locations? Not file sharing for everyone but as a means of file distribution.
What software is needed? Can the files be secured from unauthorized access?
-
you could setup a nextcloud server and install the sync clients on the machines in question.. then place the files into the NC, and they would sync to the servers.
-
@Dashrender said in BitTorrent/P2P technology for distributed file transfer of large files?:
you could setup a nextcloud server and install the sync clients on the machines in question.. then place the files into the NC, and they would sync to the servers.
But wouldn't that be just like using rsync?
You don't get the advantage of pulling files (or fragments of files) from multiple servers at the same time.
-
You certain can do this.
Use whatever your preferred BitTorrent client is.
Put your files where you want them in system one.
Crate a new torrent in the client with all of the files. Or a torrent per file or whatever.
Save the torrent file and start seeding.
Distribute the torrent file to systems two through X and start downloading.
The process will speed up as each system gets pieces.
-
https://en.wikipedia.org/wiki/Syncthing looks like an interesting option. Not sure how it compares for performance.
-
There used to be one called BTSync, but my googlefu is failing me.
I did stumble across this one: https://librevault.com/ on AlternativeTo.net
-
@JaredBusch said in BitTorrent/P2P technology for distributed file transfer of large files?:
You certain can do this.
Use whatever your preferred BitTorrent client is.
Put your files where you want them in system one.
Crate a new torrent in the client with all of the files. Or a torrent per file or whatever.
Save the torrent file and start seeding.
Distribute the torrent file to systems two through X and start downloading.
The process will speed up as each system gets pieces.
You are suggesting going without having a tracker? I guess I have to look into how the torrent file is defined. If it works without a tracker then it is great because that means you only have to install the bittorrent "client" (which works both as a client and a server).
-
@Pete-S said in BitTorrent/P2P technology for distributed file transfer of large files?:
Is it possible to use bittorrent or other P2P technology to transfer large files between a number of servers in different locations? Not file sharing for everyone but as a means of file distribution.
What software is needed? Can the files be secured from unauthorized access?
Sure, I don't see why a torrent system would not work for this. Generally doesn't make sense for intra-corporate transfers but if you have a really unique multi-site situation it might work out.
-
@Pete-S said in BitTorrent/P2P technology for distributed file transfer of large files?:
Is it possible to use bittorrent or other P2P technology to transfer large files between a number of servers in different locations? Not file sharing for everyone but as a means of file distribution.
What software is needed? Can the files be secured from unauthorized access?
Dfsr would be easier. Secure and distributed, mesh by default.
-
@Obsolesce said in BitTorrent/P2P technology for distributed file transfer of large files?:
@Pete-S said in BitTorrent/P2P technology for distributed file transfer of large files?:
Is it possible to use bittorrent or other P2P technology to transfer large files between a number of servers in different locations? Not file sharing for everyone but as a means of file distribution.
What software is needed? Can the files be secured from unauthorized access?
Dfsr would be easier. Secure and distributed, mesh by default.
I would add the caveat to test first... tried DFSR to sync our primary file-servers and it just wasn't up to the task...
Also have to consider the possibility that the OP isn't running Windows (lucky bugger if that's the case)
-
@Pete-S said in BitTorrent/P2P technology for distributed file transfer of large files?:
@Dashrender said in BitTorrent/P2P technology for distributed file transfer of large files?:
you could setup a nextcloud server and install the sync clients on the machines in question.. then place the files into the NC, and they would sync to the servers.
But wouldn't that be just like using rsync?
You don't get the advantage of pulling files (or fragments of files) from multiple servers at the same time.
That's correct, would not address what you are looking for.
-
@notverypunny said in BitTorrent/P2P technology for distributed file transfer of large files?:
Also have to consider the possibility that the OP isn't running Windows (lucky bugger if that's the case)
He didn't say, so I'm allowed to assume whatever I want
-
@notverypunny said in BitTorrent/P2P technology for distributed file transfer of large files?:
I would add the caveat to test first...
That always goes without saying, it's part of the development process. I take it testing things first is a new concept for you?
