Co-lo + 5 (or more) sites....connect 'em all

1337

Here are some benchmarks on IPsec with some different edgerouters.
https://www.simonmott.co.uk/2018/08/ubiquiti-edgerouter-ipsec-performance/
From the link it says the more powerful ER-4 will top out at about 450 Mbps of IPsec using AES-128.

FATeknollogee

Hmmm...is this an option...? https://www.tnsr.com/

scottalanmiller

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

FATeknollogee

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

One would have to switch to pfSense if TNSR is a viable option.

scottalanmiller

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

One would have to switch to pfSense if TNSR is a viable option.

I guess the real question I'd have is... why? What about TNSR makes it interesting in any way? Aren't you just looking at replacing tried and true, built in IPSec implementations with this complicated package that is just repacking OpenSwan?

I'm confused what you are trying to achieve. Connecting 5+ sites is the absolute clear use case for normal everyday IPSec on your outside hardware router. This is as "by the textbook" as it gets.

Can you use other VPN tech for this like OpenVPN, yes. Should you? Not really, it has no benefits to you. IPSec is best for this for speed, support, ease of use.

This is not a case where ZT has applicability unless you have needs that haven't been mentioned. Same with TNSR, what would this do other than make simple IPSec really hard and complicated for no reason?

This feels like one of those Aaron threads where he's captivated by all kinds of shiny product pages and misses that he's trying to do something very straightforward that is handled best by the tools that everyone uses for this every day. I'm missing what is driving the attempt to research new, hip, flashy products as none of them seem to bring anything to this particular table.

scottalanmiller

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

What options are available today?
VPN, ZeroTier??

Keep in mind anything that does this is a VPN. ZT and others are not VPN alternatives, they are just VPNs. VPN (or leased lines) are the only possible options at the end of the day.

FATeknollogee

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

One would have to switch to pfSense if TNSR is a viable option.

I guess the real question I'd have is... why? What about TNSR makes it interesting in any way? Aren't you just looking at replacing tried and true, built in IPSec implementations with this complicated package that is just repacking OpenSwan?

I'm confused what you are trying to achieve. Connecting 5+ sites is the absolute clear use case for normal everyday IPSec on your outside hardware router. This is as "by the textbook" as it gets.

Can you use other VPN tech for this like OpenVPN, yes. Should you? Not really, it has no benefits to you. IPSec is best for this for speed, support, ease of use.

This is not a case where ZT has applicability unless you have needs that haven't been mentioned. Same with TNSR, what would this do other than make simple IPSec really hard and complicated for no reason?

This feels like one of those Aaron threads where he's captivated by all kinds of shiny product pages and misses that he's trying to do something very straightforward that is handled best by the tools that everyone uses for this every day. I'm missing what is driving the attempt to research new, hip, flashy products as none of them seem to bring anything to this particular table.

The claimed speeds is what caught my attention.
TNSR "claims" they can do High Speed Site-to-Site IPsec VPN
"TNSR provides secure high-speed routing solutions at 1, 10, 40, 100 Gbps, and beyond - at a fraction of the price of alternatives."

JaredBusch

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

One would have to switch to pfSense if TNSR is a viable option.

I guess the real question I'd have is... why? What about TNSR makes it interesting in any way? Aren't you just looking at replacing tried and true, built in IPSec implementations with this complicated package that is just repacking OpenSwan?

I'm confused what you are trying to achieve. Connecting 5+ sites is the absolute clear use case for normal everyday IPSec on your outside hardware router. This is as "by the textbook" as it gets.

Can you use other VPN tech for this like OpenVPN, yes. Should you? Not really, it has no benefits to you. IPSec is best for this for speed, support, ease of use.

This is not a case where ZT has applicability unless you have needs that haven't been mentioned. Same with TNSR, what would this do other than make simple IPSec really hard and complicated for no reason?

This feels like one of those Aaron threads where he's captivated by all kinds of shiny product pages and misses that he's trying to do something very straightforward that is handled best by the tools that everyone uses for this every day. I'm missing what is driving the attempt to research new, hip, flashy products as none of them seem to bring anything to this particular table.

The claimed speeds is what caught my attention.
TNSR "claims" they can do High Speed Site-to-Site IPsec VPN
"TNSR provides secure high-speed routing solutions at 1, 10, 40, 100 Gbps, and beyond - at a fraction of the price of alternatives."

Any vRouter should be able to do this though. All you need for any IPSEC solution is enough offloaded processing power to handle the chosen encryption level.

1337

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

One would have to switch to pfSense if TNSR is a viable option.

I guess the real question I'd have is... why? What about TNSR makes it interesting in any way? Aren't you just looking at replacing tried and true, built in IPSec implementations with this complicated package that is just repacking OpenSwan?

I'm confused what you are trying to achieve. Connecting 5+ sites is the absolute clear use case for normal everyday IPSec on your outside hardware router. This is as "by the textbook" as it gets.

Can you use other VPN tech for this like OpenVPN, yes. Should you? Not really, it has no benefits to you. IPSec is best for this for speed, support, ease of use.

This is not a case where ZT has applicability unless you have needs that haven't been mentioned. Same with TNSR, what would this do other than make simple IPSec really hard and complicated for no reason?

This feels like one of those Aaron threads where he's captivated by all kinds of shiny product pages and misses that he's trying to do something very straightforward that is handled best by the tools that everyone uses for this every day. I'm missing what is driving the attempt to research new, hip, flashy products as none of them seem to bring anything to this particular table.

The claimed speeds is what caught my attention.
TNSR "claims" they can do High Speed Site-to-Site IPsec VPN
"TNSR provides secure high-speed routing solutions at 1, 10, 40, 100 Gbps, and beyond - at a fraction of the price of alternatives."

I've had my eye on TNSR since it was new. It's more of a pure router than pfSense and built primarily for performance. They use DPDK, same as StarWind for instance, to get the high I/O performance. As such it's not suitable for low powered devices.

We run pfSense in our colo on standard xeons but our I/O requirements aren't high enough with only 2 gigabit WAN connection to need TNSR. I haven't tested what the limit is but a saturated 100 Mbps OpenVPN link for instance will barely register any CPU movement at all.

Intel has done some test a long time ago (2010) to show what AES-NI can do and what kind of performance you get on regular hardware using standard linux kernel.
As you can see the test below is running 6 VPN tunnels at the same time and the 10Gbps interface becomes saturated.

They use single CPU servers with Xeon E5645 - that CPU is many generations old today. IPsec tunnels are running AES-128-GCM.
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/aes-ipsec-performance-linux-paper.pdf

scottalanmiller

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

The claimed speeds is what caught my attention.
TNSR "claims" they can do High Speed Site-to-Site IPsec VPN
"TNSR provides secure high-speed routing solutions at 1, 10, 40, 100 Gbps, and beyond - at a fraction of the price of alternatives

All they are doing is IPSec on the CPU. Anyone doing IPSec there gets the same. TNSR isn't doing anything here at all, it's not even doing special IPSec, it's the same generic one that any Linux desktop will use. Which is good, but generic.

scottalanmiller

@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

Hmmm...is this an option...? https://www.tnsr.com/

An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.

One would have to switch to pfSense if TNSR is a viable option.

I guess the real question I'd have is... why? What about TNSR makes it interesting in any way? Aren't you just looking at replacing tried and true, built in IPSec implementations with this complicated package that is just repacking OpenSwan?

I'm confused what you are trying to achieve. Connecting 5+ sites is the absolute clear use case for normal everyday IPSec on your outside hardware router. This is as "by the textbook" as it gets.

Can you use other VPN tech for this like OpenVPN, yes. Should you? Not really, it has no benefits to you. IPSec is best for this for speed, support, ease of use.

This is not a case where ZT has applicability unless you have needs that haven't been mentioned. Same with TNSR, what would this do other than make simple IPSec really hard and complicated for no reason?

This feels like one of those Aaron threads where he's captivated by all kinds of shiny product pages and misses that he's trying to do something very straightforward that is handled best by the tools that everyone uses for this every day. I'm missing what is driving the attempt to research new, hip, flashy products as none of them seem to bring anything to this particular table.

The claimed speeds is what caught my attention.
TNSR "claims" they can do High Speed Site-to-Site IPsec VPN
"TNSR provides secure high-speed routing solutions at 1, 10, 40, 100 Gbps, and beyond - at a fraction of the price of alternatives."

Any vRouter should be able to do this though. All you need for any IPSEC solution is enough offloaded processing power to handle the chosen encryption level.

Yeah, this is 100% about selecting the CPU, nothing else.

1337

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

Any vRouter should be able to do this though. All you need for any IPSEC solution is enough offloaded processing power to handle the chosen encryption level.

Yeah, this is 100% about selecting the CPU, nothing else.

That's not entirely true. The problem with high speed I/O is that the kernel eventually becomes a bottleneck. So to get the performance that the CPU is truly capable of you have to basically bypass the kernel.

That's why routers that are using DPDK can get much higher performance.

1337

Just to show how much DPDK can improve things when you have lots of packets and fast interfaces. This is a performance tests using 4x10GbE.

FATeknollogee

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

Yeah, this is 100% about selecting the CPU, nothing else.

If that's the case, there should be some "better/more" choices than the ER4?

scottalanmiller

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

Yeah, this is 100% about selecting the CPU, nothing else.

If that's the case, there should be some "better/more" choices than the ER4?

Your basic choices are....

ER4 is you want cheap, small hardware.
Bigger Ubiquiti if you want the same but even faster.
Whitebox with larger than Ubiquiti scale hardware.

There are loads of vendors out there, but you are pretty much replicating these three underlying choices in some way. Small hardware, big hardware, white box. In all cases, IPSec is the choice for the fastest option on the given platform.

FATeknollogee

@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:

Your basic choices are....

ER4 is you want cheap, small hardware.
Bigger Ubiquiti if you want the same but even faster.
Whitebox with larger than Ubiquiti scale hardware.

Cheap: ER4/ER6
Bigger Ubiquiti: ER Infinity
Whitebox: pfSense (insert fav brand) w own hardware - bigger/faster cpu, more RAM, SSD, Intel NICs etc

1337

Shouldn't the first question be - how big are your pipes?

Then - how much of that will run over IPsec?

And - what features do you need?

FATeknollogee

@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:

Shouldn't the first question be - how big are your pipes?

Then - how much of that will run over IPsec?

And - what features do you need?

That's a reasonable question(s)

Pipe size: 1x 400/400 (AT&T), 3x 500/500 (Frontier) & 1x 1000/40 (Spectrum). Colo pipe will be adjusted as needed.
How much over IPsec: as much as I can get!
Features: mainly Site to Site VPN

1337

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:

Shouldn't the first question be - how big are your pipes?

Then - how much of that will run over IPsec?

And - what features do you need?

That's a reasonable question(s)

Pipe size: 1x 400/400 (AT&T), 3x 500/500 (Frontier) & 1x 1000/40 (Spectrum). Colo pipe will be adjusted as needed.

How much over IPsec: as much as I can get!

Features: mainly Site to Site VPN

Well, you have peak 1900 Mbps in one direction and 940 in the other. But you never get that all the way so 1000/1000 in the colo will likely be more than you need. If it's all going to be IPsec traffic then ER4/ER6 is too small. Do you need HA as well?

FATeknollogee

@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:

@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:

@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:

Shouldn't the first question be - how big are your pipes?

Then - how much of that will run over IPsec?

And - what features do you need?

That's a reasonable question(s)

Pipe size: 1x 400/400 (AT&T), 3x 500/500 (Frontier) & 1x 1000/40 (Spectrum). Colo pipe will be adjusted as needed.

How much over IPsec: as much as I can get!

Features: mainly Site to Site VPN

Well, you have peak 1900 Mbps in one direction and 940 in the other. But you never get that all the way so 1000/1000 in the colo will likely be more than you need. If it's all going to be IPsec traffic then ER4/ER6 is too small. Do you need HA as well?

HA would be a nice "luxury" to have!

If the ER4/6 is too small, what other choice(s) are available?