ESXi cluster, advice needed
-
@pmoncho said in ESXi cluster, advice needed:
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
And I can spin up a new KVM based hypervisor in less than 5 minutes of my time. Plus I save time on the back end by not having to enable basic features for each VM (CPU and RAM dynamic allocation, blanking on what ESXi calls it.) It even includes a web based management platform (Cockpit).
I agree that Hyper-V is such a pain to setup and manage that paying for ESXi could make sense if they were the only two solutions available.
-
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
And I can spin up a new KVM based hypervisor in less than 5 minutes of my time. Plus I save time on the back end by not having to enable basic features for each VM (CPU and RAM dynamic allocation, blanking on what ESXi calls it.) It even includes a web based management platform (Cockpit).
I agree that Hyper-V is such a pain to setup and manage that paying for ESXi could make sense if they were the only two solutions available.
even includes a web based management platform (Cockpit).
This is not acceptable for 80 VMs. You can't clone through it. You're going to manually install 80 systems without a base template? Virt-Manager is still the best way to manage.
However you don't have a REST API for KVM (or Hyper-V). That's a big advantage that vSphere/vCenter has. That opens up a multitude of tools to automate everything you're doing.
Terraform does work with libvirt (through a community provider), but because of the way libvirt works your templates and disks are downloaded from where your terraform instance is running (see here: https://github.com/dmacvicar/terraform-provider-libvirt/issues/299). So you're now limited to only running over localhost unless you want to stream your disks to the remote end every time you build a system.
I still love KVM but I'm very sad that it's this many years later and they don't have anything to compete with this.
-
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
And I can spin up a new KVM based hypervisor in less than 5 minutes of my time. Plus I save time on the back end by not having to enable basic features for each VM (CPU and RAM dynamic allocation, blanking on what ESXi calls it.) It even includes a web based management platform (Cockpit).
I agree that Hyper-V is such a pain to setup and manage that paying for ESXi could make sense if they were the only two solutions available.
Thanks for the info. Based on most of what I read here, only a few use KVM or XCP-NG in production. I don't have an official poll or anything so I could be mistaken. It just seem if someone wants free, they go Hyper-V.
-
@pmoncho said in ESXi cluster, advice needed:
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
Well $500 cheaper. Often faster. More flexibility. Easier to manage. Less to install. No licenses to manage or keep paying for.
Easier, more powerful, cheaper. It's a pretty big improvement.
-
@pmoncho said in ESXi cluster, advice needed:
Not so much in a single server situation.
The same goes both ways. ALL the benefits of ESXi are gone in a single server environment. All. 100% of them. Whereas the benefits of the others are just not as dramatic.
In a tiny environment, all benefits across the board are less. But ESXi gets hit the most, because everything beneficial about the ESXi ecosystem is dependent on a minimum of the Essentials Plus license and multiple machines.
-
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
-
@pmoncho said in ESXi cluster, advice needed:
I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
No cost savings here, not sure where you are finding that savings. Hyper-V doesn't cost $200 more than ESXi to deploy.
And ESXi is quite expensive once you get to any size. It just keeps costing more and more. And you pay for every little feature.
Bottom line, it costs more, and it doesn't bring benefits. It's not faster, it's not easier, it doesn't do anything beneficial.
You are arguing from the position of "it's not THAT bad", but you haven't said anything that's better, only "not that bad." That you have to use "not that bad" as the position tells us that to you, too, it doesn't feel like a good product, but you aren't ready to articulate it yet. But we are hearing you agree with us because stating that it's only "less bad" is very clear that it's still "worse".
-
@travisdh1 said in ESXi cluster, advice needed:
It even includes a web based management platform (Cockpit).
THis is huge. Whether it's Cockpit for small deployments, virt-manager, or Proxmox or whatever, VMware is so far behind in good management tools.
The majority of our deployments are single server shops, and Vmware is far and away the worst option. Hyper-V is awful compared to Xen and KVM, and still worlds ahead of VMware. The ease of being able to quickly and easily manage these systems remotely is a killer feature in the tiniest of shops.
-
@stacksofplates said in ESXi cluster, advice needed:
This is not acceptable for 80 VMs. You can't clone through it. You're going to manually install 80 systems without a base template? Virt-Manager is still the best way to manage.
Not for 80, no. I think he was saying that when you get to the tiny end of the scale, KVM offers advantages in a different way.
-
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
In a single server situation, what flexibility or feature exists? I'm only seeing the benefits of free hypervisors kick in in multi-server situations.
-
@scottalanmiller said in ESXi cluster, advice needed:
@stacksofplates said in ESXi cluster, advice needed:
This is not acceptable for 80 VMs. You can't clone through it. You're going to manually install 80 systems without a base template? Virt-Manager is still the best way to manage.
Not for 80, no. I think he was saying that when you get to the tiny end of the scale, KVM offers advantages in a different way.
Not sure why he would use that case. The case we are talking about is 80 VMs on a single node.
-
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
In a single server situation, what flexibility or feature exists? I'm only seeing the benefits of free hypervisors kick in in multi-server situations.
Not as many, surely. And it varies by the product, but...
- No licensing or license center or logins needed to get downloads and updates (this is a killer feature IMHO, as an MSP we deal with companies that got screwed by this constantly - often when IT pros move on to another company and no one remembers or knows what licenses to manage.)
- Easier remote access, generally without needing to deploy and manage another virtual machine just for that.
- Simpler overall management. VMware is the only platform we are regularly called in to manage because it is "too hard" for the IT teams. We manage everything because people are busy, but VMware is the only one that is regularly "too hard" and people are having issues with it.
- More reliable updates. We see more issues per-VMware install than all other platforms combined. It's not bad, it's just not on par with the market.
- Broader hardware support and options. Software RAID being the massive feature here in the single server space (things like VSANs in the bigger spaces.)
- Included features like built in backups are common in other packages.
- Even in stand alone environments, ability to move workloads to another machine are a huge deal. Stand alone doesn't imply a lack of hardware to send workloads to. It might be that way, but not necessarily.
- Preparation for growth. The single server environment today is the two server environment tomorrow. Flexibility to protect against the unknown always matters.
-
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
In a single server situation, what flexibility or feature exists? I'm only seeing the benefits of free hypervisors kick in in multi-server situations.
Not as many, surely. And it varies by the product, but...
- No licensing or license center or logins needed to get downloads and updates (this is a killer feature IMHO, as an MSP we deal with companies that got screwed by this constantly - often when IT pros move on to another company and no one remembers or knows what licenses to manage.)
- Easier remote access, generally without needing to deploy and manage another virtual machine just for that.
- Simpler overall management. VMware is the only platform we are regularly called in to manage because it is "too hard" for the IT teams. We manage everything because people are busy, but VMware is the only one that is regularly "too hard" and people are having issues with it.
- More reliable updates. We see more issues per-VMware install than all other platforms combined. It's not bad, it's just not on par with the market.
- Broader hardware support and options. Software RAID being the massive feature here in the single server space (things like VSANs in the bigger spaces.)
- Included features like built in backups are common in other packages.
- Even in stand alone environments, ability to move workloads to another machine are a huge deal. Stand alone doesn't imply a lack of hardware to send workloads to. It might be that way, but not necessarily.
- Preparation for growth. The single server environment today is the two server environment tomorrow. Flexibility to protect against the unknown always matters.
Thanks a lot for this. Will keep these in mind.
-
hi everybody!
first of all thank you for your contribution. Keep it simple...
However, i did not mention (intentionally, no ofence i will explain myself) the following facts:- we already have these VMs hosted in a 4-node flexpod environment (if vmware enterprise plus is an overkill for us, then how would you judge flexpod???).
- our organization is rich in terms of money but poor in terms of IT intellectual capital. Therefore we need outsourced support. in our place it is hard to find that, so we usually address ourselves to certified solutions.
- we have invested time and money on vmware hypervisor and our poor IT would not like to throw this away.
- the initial question was supposed to refer to a DRS solution based on vmware SRM (VM based replication, not array based), however years after studying your recommendations i would like to try something more simple.
i understand that all above mentioned arguments are usually trivial in competitive environments, but unfortunately this is not our case.
sorry for wasting your time. i am thankful to you for your recommendations.
BTW what happens if a single node, or a node with local storage is lost? isn't that a potential cause for filesystem corruption?
Moreover, how do i put the host in maintenance mode (Hmmm, and why should i do that if i only have one host, especially with let's say free esxi?)? -
@rtfm said in ESXi cluster, advice needed:
hi everybody!
first of all thank you for your contribution. Keep it simple...
However, i did not mention (intentionally, no ofence i will explain myself) the following facts:- we already have these VMs hosted in a 4-node flexpod environment (if vmware enterprise plus is an overkill for us, then how would you judge flexpod???).
- our organization is rich in terms of money but poor in terms of IT intellectual capital. Therefore we need outsourced support. in our place it is hard to find that, so we usually address ourselves to certified solutions.
- we have invested time and money on vmware hypervisor and our poor IT would not like to throw this away.
- the initial question was supposed to refer to a DRS solution based on vmware SRM (VM based replication, not array based), however years after studying your recommendations i would like to try something more simple.
sorry for wasting your time. i am thankful to you for your recommendations. BTW what happens if a single node, or a node with local storage is lost? isn't that a potential cause for filesystem corruption?
- Just a waste of money
- What region of the world are you located in?
- This is just plain bad thinking. IT by it's nature is always changing. Learning something different should be very quick, resisting change just because you already know something is just the opposite of what IT should be doing.
- Also just a waste of money
Your statement about competitive environments doesn't make any sense. Many of the solutions mentioned are open source and available to anyone with an internet connection.
Assuming you are running with at least the 3-node minimum and a single node is lost, nothing happens. Once the node is put back online, everything is automatically handled in the background for you (Starwind, Gluster, Ceph, Scale).
No need for a "maintenance mode". Updates are handled without the need of a reboot, but we still recommend power cycling everything on a regular basis.
-
First of all, you need move every service you can to SaaS and Pass solution. Get rid of your database servers on prem and put them on PaaS solution. Why do you want to manage infrastructure especially if you are short staffed in IT?
-
@rtfm said in ESXi cluster, advice needed:
hi everybody!
first of all thank you for your contribution. Keep it simple...
However, i did not mention (intentionally, no ofence i will explain myself) the following facts:- we already have these VMs hosted in a 4-node flexpod environment (if vmware enterprise plus is an overkill for us, then how would you judge flexpod???).
- our organization is rich in terms of money but poor in terms of IT intellectual capital. Therefore we need outsourced support. in our place it is hard to find that, so we usually address ourselves to certified solutions.
- we have invested time and money on vmware hypervisor and our poor IT would not like to throw this away.
- the initial question was supposed to refer to a DRS solution based on vmware SRM (VM based replication, not array based), however years after studying your recommendations i would like to try something more simple.
i understand that all above mentioned arguments are usually trivial in competitive environments, but unfortunately this is not our case.
sorry for wasting your time. i am thankful to you for your recommendations.
BTW what happens if a single node, or a node with local storage is lost? isn't that a potential cause for filesystem corruption?
Moreover, how do i put the host in maintenance mode (Hmmm, and why should i do that if i only have one host, especially with let's say free esxi?)?- ?
- Why is your organization poor on IT capital? Why not hire consultants to do it? No reason to have them be on staff, is there?
- This is the sunk cost fallacy. That money is already spent, consider it gone and move forward with a most cost effective solution - that said, sometimes where you already are is the most cost effective when all aspects are considered - at least until a full overhaul is required.
- ?
-
@travisdh1 said in ESXi cluster, advice needed:
@rtfm said in ESXi cluster, advice needed:
hi everybody!
first of all thank you for your contribution. Keep it simple...
However, i did not mention (intentionally, no ofence i will explain myself) the following facts:- we already have these VMs hosted in a 4-node flexpod environment (if vmware enterprise plus is an overkill for us, then how would you judge flexpod???).
- our organization is rich in terms of money but poor in terms of IT intellectual capital. Therefore we need outsourced support. in our place it is hard to find that, so we usually address ourselves to certified solutions.
- we have invested time and money on vmware hypervisor and our poor IT would not like to throw this away.
- the initial question was supposed to refer to a DRS solution based on vmware SRM (VM based replication, not array based), however years after studying your recommendations i would like to try something more simple.
sorry for wasting your time. i am thankful to you for your recommendations. BTW what happens if a single node, or a node with local storage is lost? isn't that a potential cause for filesystem corruption?
- Just a waste of money
- What region of the world are you located in?
- This is just plain bad thinking. IT by it's nature is always changing. Learning something different should be very quick, resisting change just because you already know something is just the opposite of what IT should be doing.
- Also just a waste of money
Your statement about competitive environments doesn't make any sense. Many of the solutions mentioned are open source and available to anyone with an internet connection.
Assuming you are running with at least the 3-node minimum and a single node is lost, nothing happens. Once the node is put back online, everything is automatically handled in the background for you (Starwind, Gluster, Ceph, Scale).
No need for a "maintenance mode". Updates are handled without the need of a reboot, but we still recommend power cycling everything on a regular basis.
You're comparing three different things there. Starwind is a VSA, Gluster and Ceph are replicated storage, and Scale is a hyperconverged product.
Scale isn't open source and is more expensive than the VMware licensing he has mentioned (and a ton less features).
I'm guessing you haven't managed Gluster with KVM. If you have any sizeable images at all it will take forever to sync when you lose a node. And it's going to be a full manual setup with whatever KVM management tool you're using (libvirt, ProxMox, etc).
Starwind is also not open source and no longer supported for KVM or Hyper-V (https://www.starwindsoftware.com/resource-library/starwind-virtual-storage-appliance-installation-guide-with-kvm/) . So why even mention it
The only one that makes any sense is Ceph because it has somehwat of an automated setup in ProxMox. And we've had people in the community say things like this:
CEPH is so slow that there are whole products, like from Starwind, built just to make it fast enough to use for virtualization.
And I agree. I don't think Ceph is a great option for that.
IMO the only way that KVM makes sense is if you use local storage on different servers and have the replication between the VMs themselves and not the hosts. But this takes a lot of work. You'll want to automate the deployments and creation of each VM so no cockpit. And you won't want to do that by hand.
My point is, you can't just say "this is available through open source tools" and expect people to be able to do a setup like that from scratch with no experience.
-
@rtfm said in ESXi cluster, advice needed:
our organization is rich in terms of money but poor in terms of IT intellectual capital. Therefore we need outsourced support. in our place it is hard to find that, so we usually address ourselves to certified solutions.
This is a misunderstanding of markets. There is no such thing as a place with hard to get IT. IT has no location and there are essentially unlimited numbers of available excellent resources ready to assist any business. Businesses simply choose not to look for or hire them and instead hire sales people who screw them and hide their costs in "products" rather than honest or qualified advice. Every business should have outsources support, almost no company is big enough to have all the right people internally. But no business is in a location or situation that it can't get good people.
Certified solutions is really just a way to say "expensive products that are focused on resellers" or, another way, bad solutions that cost you far more to operate. They are channel products designed beginning to end to take advantage of this mindset and to get as much money out of companies that believe this as possible. It's an extremely common and effective game that they play.
If your company doesn't know how to achieve this, then the first thing that they need is a real outsourced CIO. A good CIO will save you a fortune in hours. Running without one is financially reckless.
-
@rtfm said in ESXi cluster, advice needed:
we have invested time and money on vmware hypervisor and our poor IT would not like to throw this away.
This is a business fallacy called the "sunk cost fallacy." Learning a hypervisor is a trivial matter. And moving to simple solutions is a long term investment. IT shouldn't "want" to do anything, they should simply be focused on what's best for the business. That, alone, defines what IT's job is.
https://en.wikipedia.org/wiki/Sunk_cost
Learning another product is just a few hours for those that don't know virtualization, and often "zero" time because it is so easy.
