New Infrastructure to Replace Scale Cluster
-
I haven't read the thread, so apologies if I repeat anyone else's words.
Here are some points:
- Central storage is not an SPOF, if done right, it will have redundant parts that can keep it going in case of a component failure, and it can be cloned. I've never seen a well built SAN go completely down in over 20 years of working with them.
- On the other hand, hyperconvergence is a resource drain, with systems like gluster and ceph eating up resources they share with the hypervisor, with neither being aware of each other, and VMs end up murdered by OOM, or just stalled due to CPU overcommitment.
- Gluster and other regular network based storage systems are going to be the bottleneck for the VM performance. So unless you don't care about everything being sluggish, you should think about getting a separate fabric for the storage comms, even if you hyperconverge.
- oVirt can be really nice, but you have to understand what it was built for, and not try to bend it out of shape with ridiculous requirements. A well built and pretty much zero maintenance oVirt setup will have a central storage, proper power management (you do have DRACs, right?) and doesn't use Hosted Engine. That will require more than 3 hosts.
- How many and how powerful will the VMs be? I would really go with a two node cluster, and use the third as a NAS and a standalone libvirt VM for the engine. This is the usual approach for a budget setup, where you can't afford something better.
-
@dyasny Wow, wow....no Hosted Engine? How come everyone keeps pushing HE?
Why no HE? -
@FATeknollogee because it doesn't scale. For a small setup it will work (because you don't want to waste a machine on it), but at scale you will keep getting hit by problems. Remember, the engine runs two postgres databases, both under stress, as well as a java based engine, which is also a resource hog (it's java after all). Add the fact it's doing a lot of network traffic polling all those hypervisors and getting a lot of data about everything they do every 2 seconds, and you have a VM that is doing a LOT.
For a few hypervisors, it will not be a huge issue, but drive that up to a point and you end up in a world of hurt. So for anything large-ish and where reliability is important, just avoid HE.
-
@dyasny The 300 host install you mentioned in the other thread is non-HE?
-
@FATeknollogee absolutely. Pretty much every setup with over 20 hosts I've ever built, wasn't using HE.
-
@dyasny What do you lose without HE?
Without HE does it become a manual setup where one can't use Cockpit to setup? -
@FATeknollogee Yes, it's a simple setup where you run ovirt-engine-setup and it asks you a few questions in the command line. For ease of management, I usually deploy it in a standalone VM on a separate machine. This way, if I need more resources, I can stop the machine, give it some more cores/ram or move it's disk to a faster storage, and start it up again. BAcking it all up is as simple as copying the VM disk.
-
@dyasny What are you deploying as a standalone VM? I thought you said no HE?
-
@FATeknollogee just a regular libvirt/KVM usually. If there is a multivendor virt environment, I install the engine in the second setup (vmware/hyper-v) and often the vCenter is installed in RHV
-
@dyasny Next time I do HE, I think I'll install it as a separate VM instead of the vm inside a vm approach.
-
@FATeknollogee that really depends on your cluster size. If you can afford to dedicate a separate host to it, then why not. Besides scalability, your main benefit will be not having to deal with all the hosted-engine clustering overhead. It really makes life simpler
-
@dyasny said in New Infrastructure to Replace Scale Cluster:
@FATeknollogee that really depends on your cluster size. If you can afford to dedicate a separate host to it, then why not.
You mean a separate host where the HE vm lives on?
-
@FATeknollogee said in New Infrastructure to Replace Scale Cluster:
@dyasny said in New Infrastructure to Replace Scale Cluster:
@FATeknollogee that really depends on your cluster size. If you can afford to dedicate a separate host to it, then why not.
You mean a separate host where the HE vm lives on?
Yes, it's your choice whether to do it in a VM though, it can be on baremetal
-
@dyasny said in New Infrastructure to Replace Scale Cluster:
@FATeknollogee said in New Infrastructure to Replace Scale Cluster:
@dyasny said in New Infrastructure to Replace Scale Cluster:
@FATeknollogee that really depends on your cluster size. If you can afford to dedicate a separate host to it, then why not.
You mean a separate host where the HE vm lives on?
Yes, it's your choice whether to do it in a VM though, it can be on baremetal
Is doing it in a vm bad? That would be my choice unless there is some compelling reason to do it baremetal.
-
@FATeknollogee doing it in a VM is convenient. You can always move that VM to another host, you can easily back it up by copying it's disk and domxml, you can even easily set it up as an HA cluster with pacemaker protecting the libvirt service. Databases though, feel more convenient on baremetal, so if you're going to build something with hundreds of hosts, I'd suggest you invest in the engine host as well.
-
@dyasny so to clarify for me, as i'm fighting a headache.
This design is similar to that of ESXi with vsphere.In that you should have 3 physical hosts, and 1 of which is installed with the vSphere service.
Correct?
-
@DustinB3403 no, in this particular setup, you have two options. The original one would be to go hyperconverged, installing both the storage and hypervisors services on all 3 hosts, and to also deploy the engine (vsphere equivalent) as a VM in the setup (that's called self hosted engine).
The better option, IMO, is to use two hosts as hypervisors, and the third - pack with disks, and use as the storage device (NFS or iSCSI). And also install the engine on it, as a VM or on baremetal - doesn't matter.
You will have less hypervisors, true, but having a storage service on the hypervisors is a resource drain, so you don't actually lose as much in terms of resources. And you gain a proper storage server, less management headache, and a setup that can scale nicely if you decide to add hypervisors or buy a real SAN. Performance will also be better, and you might even end up with more available disk space, because you will not have to keep 3 replicas of every byte like gluster/ceph require you to do.
-
@dyasny said in New Infrastructure to Replace Scale Cluster:
@DustinB3403 no, in this particular setup, you have two options. The original one would be to go hyperconverged, installing both the storage and hypervisors services on all 3 hosts, and to also deploy the engine (vsphere equivalent) as a VM in the setup (that's called self hosted engine).
The better option, IMO, is to use two hosts as hypervisors, and the third - pack with disks, and use as the storage device (NFS or iSCSI). And also install the engine on it, as a VM or on baremetal - doesn't matter.
You will have less hypervisors, true, but having a storage service on the hypervisors is a resource drain, so you don't actually lose as much in terms of resources. And you gain a proper storage server, less management headache, and a setup that can scale nicely if you decide to add hypervisors or buy a real SAN. Performance will also be better, and you might even end up with more available disk space, because you will not have to keep 3 replicas of every byte like gluster/ceph require you to do.
Isn't that an IPOD though?
-
@Dashrender said in New Infrastructure to Replace Scale Cluster:
Isn't that an IPOD though?
It's a server, not an ipod. If you mean SPOF, then yes, if the entire server just dies, you lose the cluster. Obviously, the server itself can be backed up, clustered, and installed with redundant components to avoid that. It's all a matter of balancing between budget, the admin's level or paranoia, and the desire to have a reliable setup without working too hard. I really like the latter, and in the long run this approach has always served me and my customers very well.
These 3 factors are up to the OP of course. All I'm trying to say with my suggestion is that you're not saving anything by hyperconverging, because you ARE making the setup much more complex, with many more moving parts that require configuration, tuning, updates and server resources than you would have by just sticking to the KISS principle.
-
IPOD = Inverted Pyramid of Doom.
This a term that Scott Allen Miller coined ages ago.1 - SAN
2 - Switches
2+ - serversThe general belief is that the SAN is 'so good' it will fail less frequently than the other components. Now - in your setup it might be as good as the servers, since it's not one of those manufactured typical SANs, but it's still just a server.
The general idea around these parts you don't go to centralized storage until you have at least 4 hypervisor hosts, otherwise putting storage locally is often much less expensive and less risky.
And of course, we haven't even touched on HA.
