Large file sharing to, from and within mainland China
-
Picking RHEL 7 is probably fine.
-
@scottalanmiller said in Large file sharing to, from and within mainland China:
Picking RHEL 7 is probably fine.
I tried Other Linux but it only had the E1000 NIC so I tried RHEL 7 and it had the VMXNET 3 NIC so I went with that and am doing the updates on the Fedora installation now.
-
This post is deleted! -
@scottalanmiller I am looking at your post on installing via saltstack and I am not sure where to begin. I posted on that thread to keep it related.
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller I am looking at your post on installing via saltstack and I am not sure where to begin. I posted on that thread to keep it related.
I just responded. I'm on my phone in bed. But I have Salt guides on here separate from this that show how to set up both pieces. All pieces can be on one box. Although that makes it a bit silly.
-
@scottalanmiller Thanks for the reply.
-
So it looks like there are essentially 2 speeds when dealing with China- The inside China speed and the International speed. Their inside speed is 100Mbps but their average international speed (they refer to it as the International Corridor) is 1Mbps. Yeah... 1!
I have yet to find an optimal way of doing this.
I have asked them to find out how to increase their international bandwidth. I was doing some Googling over the weekend and found that larger companies have this capability but I don't know how they are doing it and how much it costs (probably more than we can spend).
Also, if I can work with a Chinese host for a nextcloud server, setting up some public-facing websites, which nextcloud may or may not be classified as, may or may not require some sort of license from the Chinese government, which could take a month or more or get denied. If I can get that setup, then we are still dealing with an issue where we would need to replicate the data across to a second nextcloud server here. From what I was reading on the nextcloud forums, there is no native way of doing this yet and even if we could, we are still constrained by the 1Mbps average international speed.
The user proposed syncing two Synology NAS devices (1 in China and 1 here), but that introduces all sorts of other headaches on how to get the data shared out to our users here and backed up consistently. As we all know, syncing does not equal a backup.
-
@wrx7m said in Large file sharing to, from and within mainland China:
So it looks like there are essentially 2 speeds when dealing with China- The inside China speed and the International speed. Their inside speed is 100Mbps but their average international speed (they refer to it as the International Corridor) is 1Mbps. Yeah... 1!
I have yet to find an optimal way of doing this.
I have asked them to find out how to increase their international bandwidth. I was doing some Googling over the weekend and found that larger companies have this capability but I don't know how they are doing it and how much it costs (probably more than we can spend).
Also, if I can work with a Chinese host for a nextcloud server, setting up some public-facing websites, which nextcloud may or may not be classified as, may or may not require some sort of license from the Chinese government, which could take a month or more or get denied. If I can get that setup, then we are still dealing with an issue where we would need to replicate the data across to a second nextcloud server here. From what I was reading on the nextcloud forums, there is no native way of doing this yet and even if we could, we are still constrained by the 1Mbps average international speed.
The user proposed syncing two Synology NAS devices (1 in China and 1 here), but that introduces all sorts of other headaches on how to get the data shared out to our users here and backed up consistently. As we all know, syncing does not equal a backup.
Would something like RDS or VDI work in this instance?
-
@coliver said in Large file sharing to, from and within mainland China:
@wrx7m said in Large file sharing to, from and within mainland China:
So it looks like there are essentially 2 speeds when dealing with China- The inside China speed and the International speed. Their inside speed is 100Mbps but their average international speed (they refer to it as the International Corridor) is 1Mbps. Yeah... 1!
I have yet to find an optimal way of doing this.
I have asked them to find out how to increase their international bandwidth. I was doing some Googling over the weekend and found that larger companies have this capability but I don't know how they are doing it and how much it costs (probably more than we can spend).
Also, if I can work with a Chinese host for a nextcloud server, setting up some public-facing websites, which nextcloud may or may not be classified as, may or may not require some sort of license from the Chinese government, which could take a month or more or get denied. If I can get that setup, then we are still dealing with an issue where we would need to replicate the data across to a second nextcloud server here. From what I was reading on the nextcloud forums, there is no native way of doing this yet and even if we could, we are still constrained by the 1Mbps average international speed.
The user proposed syncing two Synology NAS devices (1 in China and 1 here), but that introduces all sorts of other headaches on how to get the data shared out to our users here and backed up consistently. As we all know, syncing does not equal a backup.
Would something like RDS or VDI work in this instance?
I have a server, here, running RDS via RDGateway but they are still needing to transfer the large files to, from and within China.
-
@wrx7m said in Large file sharing to, from and within mainland China:
The user proposed syncing two Synology NAS devices (1 in China and 1 here), but that introduces all sorts of other headaches on how to get the data shared out to our users here and backed up consistently. As we all know, syncing does not equal a backup.
That basically never works.
-
@coliver said in Large file sharing to, from and within mainland China:
Would something like RDS or VDI work in this instance?
1Mb/s with high latency... I'd guess no.
-
@scottalanmiller said in Large file sharing to, from and within mainland China:
@coliver said in Large file sharing to, from and within mainland China:
Would something like RDS or VDI work in this instance?
1Mb/s with high latency... I'd guess no.
You're probably right. There doesn't really seem to be a good solution in this event.
-
@scottalanmiller I figured as much. I have a couple synology NAS devices for some scratch space and even then, it is a datastore for VMware. The other is my Veeam backup repo.
-
@wrx7m said in Large file sharing to, from and within mainland China:
@coliver said in Large file sharing to, from and within mainland China:
@wrx7m said in Large file sharing to, from and within mainland China:
So it looks like there are essentially 2 speeds when dealing with China- The inside China speed and the International speed. Their inside speed is 100Mbps but their average international speed (they refer to it as the International Corridor) is 1Mbps. Yeah... 1!
I have yet to find an optimal way of doing this.
I have asked them to find out how to increase their international bandwidth. I was doing some Googling over the weekend and found that larger companies have this capability but I don't know how they are doing it and how much it costs (probably more than we can spend).
Also, if I can work with a Chinese host for a nextcloud server, setting up some public-facing websites, which nextcloud may or may not be classified as, may or may not require some sort of license from the Chinese government, which could take a month or more or get denied. If I can get that setup, then we are still dealing with an issue where we would need to replicate the data across to a second nextcloud server here. From what I was reading on the nextcloud forums, there is no native way of doing this yet and even if we could, we are still constrained by the 1Mbps average international speed.
The user proposed syncing two Synology NAS devices (1 in China and 1 here), but that introduces all sorts of other headaches on how to get the data shared out to our users here and backed up consistently. As we all know, syncing does not equal a backup.
Would something like RDS or VDI work in this instance?
I have a server, here, running RDS via RDGateway but they are still needing to transfer the large files to, from and within China.
What about a more manual process? Like a dedicated machine inside China that has a NextCloud client? Then they manually manage files inside of China, rather than having them be automatically handled elsewhere? I don't know the end user use case, but if there aren't a lot of users, this is likely pretty simple.
-
@scottalanmiller The problems with that, is that we still need to give and receive the large files from them and also get them backed up.
-
WAN Optimization can help, but with users not in a central location in China, it doesn't help much. There are some serious WAN optimization products out there, but this isn't a great use case for them.
Is there anything that you can do to compress these large files? How big are they, how many and what is the change rate on them? And how much is changed in China vs, in the US?
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller The problems with that, is that we still need to give and receive the large files from them and also get them backed up.
I know and NextCloud will do that.
-
So here is my thought.... You have one machine IN China that syncs to the NextCloud server in the US (or vice versa, but fewer hassles this way.) The Chinese staff grab the file from there and copy it around however they need. They always save changes back to that location, though. So the latest copy is always on the machine syncing to NextCloud. This machine will start sending the updated copy to the US the moment that it is saved, but the Chinese staff get to keep working.
Then the NextCloud server just has to be backed up as usual in the US.
-
@scottalanmiller I had suggested that we can upload the files to a nextcloud server here and send our Chinese users the links so that they can share the data with the factories/vendors but he said that would only work for some of the situations.
EDIT- That would cut out the downloading twice and place the burden on the factories. The reason this isn't ideal is because they sometimes manipulate/edit the files before sending them onto the factories/vendors
The large files are compressed in a zip format.
We also have video, docs and other files that are used for QC of our products that need to be accessed and stored for our R&D and Execs here .
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller I had suggested that we can upload the files to a nextcloud server here and send our Chinese users the links so that they can share the data with the factories/vendors but he said that would only work for some of the situations.
EDIT- That would cut out the downloading twice and place the burden on the factories. The reason this isn't ideal is because they sometimes manipulate/edit the files before sending them onto the factories/vendors
The large files are compressed in a zip format.
We also have video, docs and other files that are used for QC of our products that need to be accessed and stored for our R&D and Execs here .
Right, my suggestion sounds like it would work fine then since it solves all of those problems. It's your thought that you could share links from Nextcloud that isn't going to work. Take that out and re-read mine again. I think you will find that it will work.
Also, don't use zip it's VERY inefficient. There are much more powerful compression mechanisms that might shave a lot of time off of your transfers.
