Small colo infrastructure for SaaS
-
@dustinb3403 said in Small colo infrastructure for SaaS:
Should Host A go down, everything is on Host B with a quick startup and you're off to the races.
Why quick startup, rather than both running in parallel? There is no licensing cost to running both, there is no obvious benefit unless the only goal is to avoid a load balancer.
What am I missing?
-
@scottalanmiller said in Small colo infrastructure for SaaS:
@pete-s said in Small colo infrastructure for SaaS:
And I want to have the backup locally available so restores are fast - even if I will also backup to offsite at times.
Colo is 30 minute drive from our office.
Hence my design, failover is instantaneous and simple.
It is most certainly is not. It is definitely more complex.
He has hypervisor A with 20 web apps running everything.
Hypervisor A shits.
Now, every single one of those 20 VM's on hypervisor 2 must be logged into and the IP addresses updated, or the router must be logged into and all the inbound traffic must be rerouted. or the added complexity of some kind of magic pool of proxy servers must be previously setup.
Either way, once hypervisor a comes back online, you have to spend more time and effort figuring out how to reverse the replication process.
On the other hand, you setup hypervisor replication from host to host and something to heartbeat it to turn on the VM's on the second host on a failure.
Then when host A comes back, you simply reverse the replication and host 2 goes back to being standby.
-
@dustinb3403 said in Small colo infrastructure for SaaS:
Sure, its easy enough to move a database in a planned move, during a disaster you have to pay the host, setup the DNS records, move the data, let all of the changes sync.
Still not following. Since this would be planned, even in the event of a disaster, it's still free and easy. What am I missing? there is no DNS, no data to move, no syncing... nothing.
Both nodes have the DB all of the time. None of your concerns would exist until yet another layer of disaster were to strike. At which point, you'd be recovering from backup rather than load balancing to the other node.
-
@jaredbusch said in Small colo infrastructure for SaaS:
@scottalanmiller said in Small colo infrastructure for SaaS:
@pete-s said in Small colo infrastructure for SaaS:
And I want to have the backup locally available so restores are fast - even if I will also backup to offsite at times.
Colo is 30 minute drive from our office.
Hence my design, failover is instantaneous and simple.
It is most certainly is not. It is definitely more complex.
He has hypervisor A with 20 web apps running everything.
Hypervisor A shits.
Now, every single one of those 20 VM's on hypervisor 2 must be logged into and the IP addresses updated, or the router must be logged into and all the inbound traffic must be rerouted. or the added complexity of some kind of magic pool of proxy servers must be previously setup.
I'm confused. MY design has none of that. 20 VMs, zero interaction. Just fails over, immediately. No effort, no mapping. Failover can be done safely anytime, all the time. In fact, with normal load balancing, you effectively failover with every incoming new connection.
-
@scottalanmiller said in Small colo infrastructure for SaaS:
@dustinb3403 said in Small colo infrastructure for SaaS:
Should Host A go down, everything is on Host B with a quick startup and you're off to the races.
Why quick startup, rather than both running in parallel? There is no licensing cost to running both, there is no obvious benefit unless the only goal is to avoid a load balancer.
and which host is that going to run on?
and what are you going to do when that host fails?No. You are wrong. The entire word is not stateless. VM replication is easier and simpler and cheaper.
-
@jaredbusch said in Small colo infrastructure for SaaS:
Either way, once hypervisor a comes back online, you have to spend more time and effort figuring out how to reverse the replication process.
Not in my design, none of this either. None. 100% avoids all this complexity.
-
@jaredbusch said in Small colo infrastructure for SaaS:
@scottalanmiller said in Small colo infrastructure for SaaS:
@dustinb3403 said in Small colo infrastructure for SaaS:
Should Host A go down, everything is on Host B with a quick startup and you're off to the races.
Why quick startup, rather than both running in parallel? There is no licensing cost to running both, there is no obvious benefit unless the only goal is to avoid a load balancer.
and which host is that going to run on?
You normally buy a load balancer and put it out front. Or you get one from your colo and they handle that for you. That's extremely common. I've done this in the past in colo, the colo provided it.
You can do this with something like HAProxy if you want to do it for free. If you do that, you need only fail over the HAProxy, nothing else.
-
@jaredbusch said in Small colo infrastructure for SaaS:
No. You are wrong. The entire word is not stateless. VM replication is easier and simpler and cheaper.
That's why I pointed out that this was specifically the standard for well made bespoke apps, which is anything but the whole world.
-
@scottalanmiller said in Small colo infrastructure for SaaS:
@jaredbusch said in Small colo infrastructure for SaaS:
No. You are wrong. The entire word is not stateless. VM replication is easier and simpler and cheaper.
That's why I pointed out that this was specifically the standard for well made bespoke apps, which is anything but the whole world.
I see how both can work.
I think replication of the DB would be faster than continuous replication of the VM because there are less data to be changed.
Also if you have a VM of the application running on both servers and load balance you would get a higher performance and scalable solution. -
If your datacenter doesn't provide load balancing, or you just want to do it on your own, and you don't want to buy a load balancer (or load balancer cluster), you can build your own for free.
Load balancers are cheap, small, and stateless. HAProxy is one of a dozen or so decent open source options. You can make it a live/live cluster, but this takes some work that you might not want to do. If you do want to, great, you've got serious HA and load balancing without a crazy amount of effort.
If you don't, you've got lots of options. You can use VM replication only for the load balancer of course. Or you can manually flip the virtual IP when necessary and have only a single change needed for everything to fail over.
This design is very nice because you get load balancing so that you leverage both systems at the same time, and you "test" both nodes and the complete stacks on both all day, every day. You know for certain that they are both healthy. So if one fails, you aren't in doubt about the other one, you were using it just seconds ago.
Plus failover can be instant, regardless of workload size. No waiting for VMs to spin up, not even to reboot. And no worries that something has changed since the last reboot because they are already up and running.
-
@pete-s said in Small colo infrastructure for SaaS:
Also if you have a VM of the application running on both servers and load balance you would get a higher performance and scalable solution.
Exactly. When things are healthy, you have two (or more) healthy nodes ready at all times, able to split the load. Replication is only the changes as they occur, so typically super tiny. And you can just scale and scale without too much extra effort. Depending on your scaling designs it can be as simple as adding another node to your overall setup.
And you only need to backup from one node, ever, as they are all mirrors of the original.
Now if you get really big and need to shard your data, then things get more complex. But in that case, this is the only design that really works.
-
Alright, just to recap.
We have two options here and we could use either option depending on the VM and the application itself.
- Continuous VM replication (VM host to VM host replication?)
- Database replication (VM guest to VM guest replication)
And neither of them would require shared storage.
Is that correct?
And in the case of database replication we could also set up a load balancer and have the application running on multiple VMs (hosts) for higher performance and automatic failover.
-
@pete-s said in Small colo infrastructure for SaaS:
Alright, just to recap.
We have two options here and we could use either option depending on the VM and the application itself.
- Continuous VM replication (VM host to VM host replication?)
- Database replication (VM guest to VM guest replication)
And neither of them would require shared storage.
Is that correct?
Correct. Shared storage never comes into the equation here.
-
@pete-s said in Small colo infrastructure for SaaS:
And in the case of database replication we could also set up a load balancer and have the application running on multiple hosts for higher performance and automatic failover.
Correct. Higher performance, faster failover. And you are hypervisor agnostic. So XenServer, KVM, LXD, doesn't matter, they don't even have to match.
-
Looking at the hardware for the VM hosts.
VM host nodes will have 1 CPU, 10 cores, 64GB RAM, 500GB SSD storage. In my mind that would allow us to run 8 VMs per node with 4 vCPU, 8GB RAM and 50GB storage per VM. Every VM will be linux/bsd.
Nodes are also easily expandable to 2 CPU, 20 cores, 128GB RAM and whatever storage that is needed (6x3.5" bays).Does the CPU / memory /storage balance seem right for these type of applications?
Also would it be better to have VM setup with just the database or to have each type of application have their own DB inside their VM?
So which one?
- 4 VMs with different apps and one VM with the DB
- 4 VMs with different apps and DB
-
@pete-s said in Small colo infrastructure for SaaS:
Alright, just to recap.
We have two options here and we could use either option depending on the VM and the application itself.
- Continuous VM replication (VM host to VM host replication?)
- Database replication (VM guest to VM guest replication)
And neither of them would require shared storage.
Is that correct?
That is correct. In either case, nothing is shared.
With the guest replication though, nothing has to be changed. Just turn on the VM on Host B.
With the DB replication, you have a lot of items that may need to be changed.
-
@pete-s said in Small colo infrastructure for SaaS:
Looking at the hardware for the VM hosts.
VM host nodes will have 1 CPU, 10 cores, 64GB RAM, 500GB SSD storage. In my mind that would allow us to run 8 VMs per node with 4 vCPU, 8GB RAM and 50GB storage per VM. Every VM will be linux/bsd.
Nodes are also easily expandable to 2 CPU, 20 cores, 128GB RAM and whatever storage that is needed (6x3.5" bays).Does the CPU / memory /storage balance seem right for these type of applications?
Also would it be better to have VM setup with just the database or to have each type of application have their own DB inside their VM?
So which one?
- 4 VMs with different apps and one VM with the DB
- 4 VMs with different apps and DB
It's going to vary, but typically for this, if the DB is really the same across the board, I'd normally do a single DB VM that handles all of the databases. Then you only have one DB cluster to configure.
However, if the uptime and maintenance requirements of the databases are completely different, then maybe I would keep them separate.
What will cause you issues is trying to move to a new database version with schema changes if they are all merged and you can't coordinate maintenance time.
-
@dustinb3403 said in Small colo infrastructure for SaaS:
I would recommend an approach like Host A - Master, Host B slave - NLS backup target
Host A performs continuous replications to Host B, as well as Host A backs up to the NLS host on a different schedule.
Should Host A go down, everything is on Host B with a quick startup and you're off to the races.
You'd still have your separate backups to recover from should something even worse occur.
I like this solution too but I don't like to have one host just in standby because we get better performance if half the VMs are on Host 1 and they can share CPU and storage resources there and the other half on Host 2.
For the VMs that need replication, we might just as well have replication going in both directions between the hosts, instead of just one direction. Don't you agree?
BTW, can you setup continuous replication without XO, just using xencenter or with xen itself?
-
This is how @scottalanmiller suggested HA in the database layer.
Webserver files themselves are static and the data in the database is what is changing.
-
What are you using to create those diagrams?
