Small colo infrastructure for SaaS
-
@scottalanmiller said in Small colo infrastructure for SaaS:
@jaredbusch said in Small colo infrastructure for SaaS:
No. You are wrong. The entire word is not stateless. VM replication is easier and simpler and cheaper.
That's why I pointed out that this was specifically the standard for well made bespoke apps, which is anything but the whole world.
I see how both can work.
I think replication of the DB would be faster than continuous replication of the VM because there are less data to be changed.
Also if you have a VM of the application running on both servers and load balance you would get a higher performance and scalable solution. -
If your datacenter doesn't provide load balancing, or you just want to do it on your own, and you don't want to buy a load balancer (or load balancer cluster), you can build your own for free.
Load balancers are cheap, small, and stateless. HAProxy is one of a dozen or so decent open source options. You can make it a live/live cluster, but this takes some work that you might not want to do. If you do want to, great, you've got serious HA and load balancing without a crazy amount of effort.
If you don't, you've got lots of options. You can use VM replication only for the load balancer of course. Or you can manually flip the virtual IP when necessary and have only a single change needed for everything to fail over.
This design is very nice because you get load balancing so that you leverage both systems at the same time, and you "test" both nodes and the complete stacks on both all day, every day. You know for certain that they are both healthy. So if one fails, you aren't in doubt about the other one, you were using it just seconds ago.
Plus failover can be instant, regardless of workload size. No waiting for VMs to spin up, not even to reboot. And no worries that something has changed since the last reboot because they are already up and running.
-
@pete-s said in Small colo infrastructure for SaaS:
Also if you have a VM of the application running on both servers and load balance you would get a higher performance and scalable solution.
Exactly. When things are healthy, you have two (or more) healthy nodes ready at all times, able to split the load. Replication is only the changes as they occur, so typically super tiny. And you can just scale and scale without too much extra effort. Depending on your scaling designs it can be as simple as adding another node to your overall setup.
And you only need to backup from one node, ever, as they are all mirrors of the original.
Now if you get really big and need to shard your data, then things get more complex. But in that case, this is the only design that really works.
-
Alright, just to recap.
We have two options here and we could use either option depending on the VM and the application itself.
- Continuous VM replication (VM host to VM host replication?)
- Database replication (VM guest to VM guest replication)
And neither of them would require shared storage.
Is that correct?
And in the case of database replication we could also set up a load balancer and have the application running on multiple VMs (hosts) for higher performance and automatic failover.
-
@pete-s said in Small colo infrastructure for SaaS:
Alright, just to recap.
We have two options here and we could use either option depending on the VM and the application itself.
- Continuous VM replication (VM host to VM host replication?)
- Database replication (VM guest to VM guest replication)
And neither of them would require shared storage.
Is that correct?
Correct. Shared storage never comes into the equation here.
-
@pete-s said in Small colo infrastructure for SaaS:
And in the case of database replication we could also set up a load balancer and have the application running on multiple hosts for higher performance and automatic failover.
Correct. Higher performance, faster failover. And you are hypervisor agnostic. So XenServer, KVM, LXD, doesn't matter, they don't even have to match.
-
Looking at the hardware for the VM hosts.
VM host nodes will have 1 CPU, 10 cores, 64GB RAM, 500GB SSD storage. In my mind that would allow us to run 8 VMs per node with 4 vCPU, 8GB RAM and 50GB storage per VM. Every VM will be linux/bsd.
Nodes are also easily expandable to 2 CPU, 20 cores, 128GB RAM and whatever storage that is needed (6x3.5" bays).Does the CPU / memory /storage balance seem right for these type of applications?
Also would it be better to have VM setup with just the database or to have each type of application have their own DB inside their VM?
So which one?
- 4 VMs with different apps and one VM with the DB
- 4 VMs with different apps and DB
-
@pete-s said in Small colo infrastructure for SaaS:
Alright, just to recap.
We have two options here and we could use either option depending on the VM and the application itself.
- Continuous VM replication (VM host to VM host replication?)
- Database replication (VM guest to VM guest replication)
And neither of them would require shared storage.
Is that correct?
That is correct. In either case, nothing is shared.
With the guest replication though, nothing has to be changed. Just turn on the VM on Host B.
With the DB replication, you have a lot of items that may need to be changed.
-
@pete-s said in Small colo infrastructure for SaaS:
Looking at the hardware for the VM hosts.
VM host nodes will have 1 CPU, 10 cores, 64GB RAM, 500GB SSD storage. In my mind that would allow us to run 8 VMs per node with 4 vCPU, 8GB RAM and 50GB storage per VM. Every VM will be linux/bsd.
Nodes are also easily expandable to 2 CPU, 20 cores, 128GB RAM and whatever storage that is needed (6x3.5" bays).Does the CPU / memory /storage balance seem right for these type of applications?
Also would it be better to have VM setup with just the database or to have each type of application have their own DB inside their VM?
So which one?
- 4 VMs with different apps and one VM with the DB
- 4 VMs with different apps and DB
It's going to vary, but typically for this, if the DB is really the same across the board, I'd normally do a single DB VM that handles all of the databases. Then you only have one DB cluster to configure.
However, if the uptime and maintenance requirements of the databases are completely different, then maybe I would keep them separate.
What will cause you issues is trying to move to a new database version with schema changes if they are all merged and you can't coordinate maintenance time.
-
@dustinb3403 said in Small colo infrastructure for SaaS:
I would recommend an approach like Host A - Master, Host B slave - NLS backup target
Host A performs continuous replications to Host B, as well as Host A backs up to the NLS host on a different schedule.
Should Host A go down, everything is on Host B with a quick startup and you're off to the races.
You'd still have your separate backups to recover from should something even worse occur.
I like this solution too but I don't like to have one host just in standby because we get better performance if half the VMs are on Host 1 and they can share CPU and storage resources there and the other half on Host 2.
For the VMs that need replication, we might just as well have replication going in both directions between the hosts, instead of just one direction. Don't you agree?
BTW, can you setup continuous replication without XO, just using xencenter or with xen itself?
-
This is how @scottalanmiller suggested HA in the database layer.
Webserver files themselves are static and the data in the database is what is changing.
-
What are you using to create those diagrams?
-
This is how @dustinb3403 suggested replicating VMs between hosts:
-
@black3dynamite said in Small colo infrastructure for SaaS:
What are you using to create those diagrams?
Microsoft Visio Pro.
-
It does a nice job of making those look good.
-
@pete-s said in Small colo infrastructure for SaaS:
For the VMs that need replication, we might just as well have replication going in both directions between the hosts, instead of just one direction. Don't you agree?
If you don't want Host 2 sitting there waiting for something to fail... then I would have the 4 Production VMs on Host 1 Replicate to Host 2...
And the 4 Production VMs on Host 2 Replicate to Host 1... That would work, but I see the potential for confusion there.
-
I'd also recommend using a tool like Consul for service discovery and health checking.
Fabio (load balancer) natively works with Consul so as systems (LXC or VMs) are brought up they will be auto added to the pool. If they become unreachable they will be automatically removed.
-
-
@pete-s said in Small colo infrastructure for SaaS:
BTW, can you setup continuous replication without XO, just using xencenter or with xen itself?
While possible, it's not easily doable.
For the VMs that need replication, we might just as well have replication going in both directions between the hosts, instead of just one direction. Don't you agree?
I disagree here, the host performance in this case means you need your hosts to be able to cover the workload of your entire clientbase. So having slightly more powerful hosts here isn't an issue, and replication is easily be changed around.
And the reason being is you'd be replicating VM's between two hosts and only having half of the workload protected from a failed host in this case anyways.
The entire conversation revolves around protecting from a host failing. Which if you have the worst case scenario you have your backups to restore from.
-
@pete-s said in Small colo infrastructure for SaaS:
This is how @dustinb3403 suggested replicating VMs between hosts:
While this is accurate, it also misses on the fact that he would still have his NLS server sitting, collecting backups on whatever schedule.
Other than that it is accurate. In Scott's proposal you are making the shift from migrating the entire workload (which is essentially instant) to migrating the database only.
In his case, the load balancer is the weak link in the chain. Granted these don't fail often but it isn't something you have control over either unless you provide your own for the COLO.
