ZeroTier Site-To-Site

dafyre

ZeroTier Site-To-Site Setup

ASSUMPTIONS:

Site A is on 192.168.10.0/24
Site B is on 192.168.122.0/24
Site A's VM is 192.168.10.2 for the Local Network
Site A's VM is 10.0.0.107 on the ZT Network
Site B is 192.168.122.1 on the Local Network
Site B is 10.0.0.129 on the ZT Network.

Step 1: Build a Private Network on https://my.zerotier.com

Step 2: Spin up a Linux VM at each site. Connect and authorize them to the ZT Network and note their IP address. For instance:

Some folks have reported SIGNIFICANT performance improvement when using 2 cores / 2 vcpus for the Linux VMs.

Step 2B. Enable IP_Forward:

Follow your distribution's instructions to enable ip_forward and make it a permanent change... On most distros, this should work:

sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

You can then sysctl -p /etc/sysctl.conf to reload the configuration or reboot.

sysctl net.ipv4.ip_forward should return

net.ipv4.ip_forward = 1

if everything is going to work correctly.

Step 3: From either of the Linux VMs, ensure that they can ping one another on the ZT Subnet.

Step 4: Set up the Routes inside on https://my.zerotier.com

*Once you set up the routes in ZeroTier Central, you do not have to manually add them to your Linux VMs.

Step 5: Set up the Site Routes at the Routers for Site A and Site B

SITE A Main Router:

You'll notice for the router at Site A that I am using the INTERNAL network address of my Linux VM.

SITE A Linux Router VM:

root@deb-ztrouter /root # ip route
default via 192.168.10.1 dev eth0 onlink
10.0.0.0/24 dev zt1  proto kernel  scope link  src 10.0.0.107
192.168.10.0/24 dev eth0 proto kernel  scope link  src 192.168.10.2
192.168.20.0/24 via 10.0.0.116 dev zt1
192.168.122.0/24 via 10.0.0.129 dev zt1

SITE B, KVM Server, no need for separate VM:

root@france:/root# ip route
default via <my public ip> dev eth0 onlink
10.0.0.0/24 dev zt1 scope link  #ZT Subnet
192.168.10.0/24 via 10.0.0.107 dev zt1 #SiteA, 10.0.0.107 is the ZT IP for the Linux VM at Site A
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1  #This server is Site B

**This was done on systems that do not have UFW or firewall-cmd enabled. You may have to set them up to allow traffic between your sites.

Okay... I think that's it. Mine is working.

If you have any questions or comments, ask away!

lionelb

@dafyre Problem with this config : we lose 90% (or more) of the flow. It's not nothing !

Iperf from a ZT client in Azure to a ZT router on my datacenter (1 GB)
[ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec receiver

Iperf from a ZT client in Azure to a lan machine via the same ZT router my datacenter (1 GB)
[ 4] 0.00-10.00 sec 112 MBytes 9.5 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 9.4 Mbits/sec receiver

My ZT routers are Debian on ESX VM

Or can be the problem?

JasGot

Great write up! Thanks! I looked at ZeroTier a while ago, and because I couldn't get a site-to-site going in under ten minutes, I moved on.

My only (professional) need for VPNs these days is site to site.

I'll give it another go and follow your write up.

Thanks again!

dafyre

@lionelb said in ZeroTier Site-To-Site:

@dafyre Problem with this config : we lose 90% (or more) of the flow. It's not nothing !

Iperf from a ZT client in Azure to a ZT router on my datacenter (1 GB)
[ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec receiver

Iperf from a ZT client in Azure to a lan machine via the same ZT router my datacenter (1 GB)
[ 4] 0.00-10.00 sec 112 MBytes 9.5 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 9.4 Mbits/sec receiver

My ZT routers are Debian on ESX VM

Or can be the problem?

@lionelb said in ZeroTier Site-To-Site:

@dafyre Problem with this config : we lose 90% (or more) of the flow. It's not nothing !

Iperf from a ZT client in Azure to a ZT router on my datacenter (1 GB)
[ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec receiver

Iperf from a ZT client in Azure to a lan machine via the same ZT router my datacenter (1 GB)
[ 4] 0.00-10.00 sec 112 MBytes 9.5 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 9.4 Mbits/sec receiver

My ZT routers are Debian on ESX VM

Or can be the problem?

@lionelb -- What happens if you go from a client in your LAN to a client in your Azure setup?

You could also try to tracert from the Azure Client to your LAN client and make sure it's not doing anything strange.

dafyre

@JasGot said in ZeroTier Site-To-Site:

Great write up! Thanks! I looked at ZeroTier a while ago, and because I couldn't get a site-to-site going in under ten minutes, I moved on.

My only (professional) need for VPNs these days is site to site.

I'll give it another go and follow your write up.

Thanks again!

If you have any questions, do let me know!

This is the set up that I am still using today.

lionelb

@dafyre I think I found the problem, my VM Debian 9 which serves as router had only 1 GB of RAM and 1 vCPU, I made x 2 and it is much better !

Here we see the difference before and after ...

dafyre

@lionelb said in ZeroTier Site-To-Site:

@dafyre I think I found the problem, my VM Debian 9 which serves as router had only 1 GB of RAM and 1 vCPU, I made x 2 and it is much better !

Here we see the difference before and after ...

Thanks for the heads up. I made a note of that in Step 2 for making the Linux router VMs.

lionelb

@dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day

dafyre

@lionelb said in ZeroTier Site-To-Site:

@dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day

I'd recommend always using the current release, so you get security updates and such. Has Buster been released yet?

black3dynamite

@dafyre said in ZeroTier Site-To-Site:

@lionelb said in ZeroTier Site-To-Site:

@dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day

I'd recommend always using the current release, so you get security updates and such. Has Buster been released yet?

Its been out for awhile now.

dafyre

@black3dynamite said in ZeroTier Site-To-Site:

@dafyre said in ZeroTier Site-To-Site:

@lionelb said in ZeroTier Site-To-Site:

@dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day

I'd recommend always using the current release, so you get security updates and such. Has Buster been released yet?

Its been out for awhile now.

Thanks. The last month or so, I have time to eat, breathe, work, sleep, and help kiddo with his homework, lol... and work has been stupid busy for the last 3 or 4 months and I have no idea why.

lionelb

@dafyre My small addition in french : https://www.canaletto.fr/post/zerotier-site-to-site

VoIP_n00b

This post is deleted!

mukky

Bro @dafyre,
You make my life much easier...
Thank you !!

After soo much hassle to achieved opnsense site2site, i found this posting solve the problems with 2 essential modification as follows:

Two essential step:

Enable IP_Forward:
in free BSD we have to edit /etc/defaults/rc.conf
change from gateway_enable="NO" to gateway_enable="YES"
Set up the Site Routes at the Routers for Site A and Site B
it has configured and implemented in opnsense router section

@dafyre, since no body cover this on opnsense, I think it will wonderful, if you could made this video on youtube as well

Good Luck !!

mukky

@mukky said in ZeroTier Site-To-Site:

Bro @dafyre,
You make my life much easier...
Thank you !!

After soo much hassle to achieved opnsense site2site, i found this posting solve the problems with 2 essential modification as follows:

Two essential step:

Enable IP_Forward:
in free BSD we have to edit /etc/defaults/rc.conf
change from gateway_enable="NO" to gateway_enable="YES"

Set up the Site Routes at the Routers for Site A and Site B
it has configured and implemented in opnsense router section

@dafyre, since no body cover this on opnsense, I think it will wonderful, if you could made this video on youtube as well

Good Luck !!

I was struggeling for a month to figure it out, not much info on internet nor tutorial regarding zerotier for site2site. Eventually i succeed to make it work.

The key point to setting on opnsense are:

you have to install zerotier plugin

you have to make your own network on your zerotier account

you have to enable zerotier on your opnsense and adding zerotier connection in it to join your own network.

you have to assign network for zerotier - dont forget to "check" Enable Interface and Prevent interface removal. Also you have to put static ip with is the same ip address as shown on your zerotier joined network.

you have to put firewall rule for zerotier to accept any incoming traffic

you have to put firewall rule for WAN/ISP to accept any incoming traffic from specific source "Ztier.net"

in some cases it requires booting/restart your opnsense to take effect.

setting above will allow any incoming connection from any remote device via zerotier towards your opnsense ip address. (Ref: opnsense ip address = ip address of WAN/ISP). In result, you can remote access your opnsense via laptop from another city / ISP (laptop must have zerotier connection and joint the same network too). On your laptop you will be able to access your opnsense by its ip address assigned by zerotier.

in the case, for example, there is a NAS behind the opnsense that you want to access remotely,....... then you only have to open your zerotier account and put a route rule there

assumed:

your NAS local ip address: 192.168.5.10

NAS local Network on opnsense: LAN-1

your opnsense ip address assigned by Zerotier: 10.188.22.10

then you have to put firewall rule for LAN-1 to accept any incoming traffic from specific source "Ztier.net"

then you have to add "route" on your zerotier account dashboard:

192.168.5.10/32 via 10.188.22.10

in result from remote laptop you can remote access:

a. opnsense by pointing to 10.188.22.10

b. NAS by pointing to 192.168.5.10

(laptop must have zerotier connection and joint the same network too)

Thats it, good luck !