Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance
-
12 TB of storage and 128GB or RAM isn't much, I have 24TB raw in my lab alone, the RAM I don't need and don't have.
You could purchase a cheap server and get that much capacity for cheap.
-
So you have dual socket CPU with 128 GB of RAM. Which CPUs do you have?
In XenCenter you can select the host and then it's under the General tab and the CPUs section.And then 16TB disks in 8x2TB config with 7200 rpm SAS drives. So 8TB usable space and your using close to 50% now?
Do you know how much cache your H730 RAID controller has?
VM's are thin provisioned and it's a mix of Windows and Linux VM's. How many VMs do you have running and how much of the RAM is provisioned now?
To get some info on performance, in XenCenter select the host and go to the Performance tab. Right click on the graph and select Actions>New graph... Make a graph with IOwait on your SR. Post a screen shot of it together with CPU performance when you are having sluggish performance.
-
@DustinB3403 yea I know it’s not a lot. We are a somewhat small non profit. Honestly I would rather max out both servers ram wise and switch to all SSDs since the price is literally the same for Nearline SAS.
I’m assuming the thin provisioning is the culprit but that’s only an assumption. I recall reading that it would slow down in time vs thick provisioning being a bit speedier.
-
Technically right now the host is 16 Tb raw storage and 8 Tb usable (maybe a notch less) since it’s raid 10 but the XenServer HVM doesn’t see the extra 4 Tb yet. I dunno why 🤨. See I’ve done capacity expansions before on other additional single drives no problem. But on the raid I’m lost. Since it was “added” capacity from bios, I assume the hyper visor would just “see” it.
-
@Pete-S that I can do.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Since it was “added” capacity from bios, I assume the hyper visor would just “see” it.
Once the RAID is expanded, you would generally need to add or grow a logical volume to preset to the hypervisor. This isn't a RAID problem, it's an LVM problem. That you have RAID is not a factor, since that has already grown. That's no longer part of the equation.
To verify, go into your RAID management tools and see what size RAID array it believes that it has. Then see what size LV it is offering to the hypervisor. Sounds like they don't match.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
I’m assuming the thin provisioning is the culprit but that’s only an assumption. I recall reading that it would slow down in time vs thick provisioning being a bit speedier.
By only a TINY bit, and only during times of thin expansion. If you are seeing a performance issue, then that's not it.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Things are working but we are definitely seeing plenty of times where the speed is not where it use to be.
Disk slow as they become fragmented (not a problem for modern SSD, but spinning rust still has this), as contention increases (more workloads slows what is available for everyone), and as the disks fill up. Once you are past 85% capacity, storage gets slower.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
I'm just deeply assuming the thin provisioning is the culprit but that's without proof, only assumptions.
Can't be, so that can be ruled out right now. Thin provisioning has so little overhead, hence we you "never" turn it off except in really specific circumstances and those are pretty obvious (large scale, crazy high performance databases, for example.)
There is no possibility that thin provisioning is a factor here.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 so my original installation of Xen Server was to 4 disks, raid 10. When we added the additional 4, dell did a "live" update to the raid configuration so that it added the capacity. That part looks fine but then from Xen Server I'm under the "assumption" it will see the Raid has changed automagically and expand but alas it didn't
So your saying I need to add a new SR? Is that different than expanding the existing one?
You should not expect this to show up as Dell (LSI) RAID controllers contain a LVM layer. Expanding the RAID only makes the RAID card's LVM ready to be configured. Until that is done, it should be impossible for the hypervisor to see the additional storage. The RAID card doesn't know how the additional capacity is meant to be used, so it can't do anything else with it until you tell it what to do.
This is the caveat with the power of LVM on the hardware RAID.... it gives you awesome benefits like making a single RAID array that you can present as an many disks as you want to the OS. But the caveat is that you have to configure it every time, as there is no possible default setting.
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Yes, XenServer doesn't care what you did to the physical block device, as it's "block device" is set to the limits of the original OBR10.
With a Dell/LSI card, it doesn't see the OBR10, it sees the LV on top of that presented by the card.
-
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
He definitely has an LVM on the hardware RAID. He may or may not have Linux LVM2 in software as well. Two standard places for it to exist. So we know he has one, he might have two. Whatever ones are there, have to be configured.
-
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
-
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
Technically those patches killed performance for everyone, not just XenServer.
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
Technically those patches killed performance for everyone, not just XenServer.
Fair point. If he's limited to 7.1 for whatever reason it'll be worse, they managed to claw back some of the performance in later releases, but the last patches on 7.1 were deadly. Seems to be worse depending on the specific CPUs from what I was testing on our gear a couple of months back. Our R720s weren't quite rendered useless but it's been a huge factor for new gear for our XenDesktop VDI hosts. The newer procs in the R730s at another site didn't seem to be hit as hard, but it was still significant.
-
@scottalanmiller so what should I do?
-
@scottalanmiller I also recall you telling me years ago, that there was a limit to how many hard drives (spinning rust) I could add to a raid 10 depending on if I was looking for a boost in either read or write. If I recall, read was the only thing that would speed up.
-
@notverypunny See I want to get on the latest and greatest release. I just want to avoid having to nuke everything.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller I also recall you telling me years ago, that there was a limit to how many hard drives (spinning rust) I could add to a raid 10 depending on if I was looking for a boost in either read or write. If I recall, read was the only thing that would speed up.
That's RAID 1, not RAID 10.
-
@notverypunny I mean after looking around on the net I see what Scott is saying the better performance might actually be better with thick provisioning, but it's negligible when people run analytics on it vs thin. Like only a couple of %.
But I think I see what Scott is also saying that maybe one of the HDD's if possibly throwing errors or failing. I've already replaced 2 drives in 2 years.
