Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Things are working but we are definitely seeing plenty of times where the speed is not where it use to be.
Disk slow as they become fragmented (not a problem for modern SSD, but spinning rust still has this), as contention increases (more workloads slows what is available for everyone), and as the disks fill up. Once you are past 85% capacity, storage gets slower.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
I'm just deeply assuming the thin provisioning is the culprit but that's without proof, only assumptions.
Can't be, so that can be ruled out right now. Thin provisioning has so little overhead, hence we you "never" turn it off except in really specific circumstances and those are pretty obvious (large scale, crazy high performance databases, for example.)
There is no possibility that thin provisioning is a factor here.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 so my original installation of Xen Server was to 4 disks, raid 10. When we added the additional 4, dell did a "live" update to the raid configuration so that it added the capacity. That part looks fine but then from Xen Server I'm under the "assumption" it will see the Raid has changed automagically and expand but alas it didn't
So your saying I need to add a new SR? Is that different than expanding the existing one?
You should not expect this to show up as Dell (LSI) RAID controllers contain a LVM layer. Expanding the RAID only makes the RAID card's LVM ready to be configured. Until that is done, it should be impossible for the hypervisor to see the additional storage. The RAID card doesn't know how the additional capacity is meant to be used, so it can't do anything else with it until you tell it what to do.
This is the caveat with the power of LVM on the hardware RAID.... it gives you awesome benefits like making a single RAID array that you can present as an many disks as you want to the OS. But the caveat is that you have to configure it every time, as there is no possible default setting.
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Yes, XenServer doesn't care what you did to the physical block device, as it's "block device" is set to the limits of the original OBR10.
With a Dell/LSI card, it doesn't see the OBR10, it sees the LV on top of that presented by the card.
-
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
He definitely has an LVM on the hardware RAID. He may or may not have Linux LVM2 in software as well. Two standard places for it to exist. So we know he has one, he might have two. Whatever ones are there, have to be configured.
-
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
-
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
Technically those patches killed performance for everyone, not just XenServer.
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
Technically those patches killed performance for everyone, not just XenServer.
Fair point. If he's limited to 7.1 for whatever reason it'll be worse, they managed to claw back some of the performance in later releases, but the last patches on 7.1 were deadly. Seems to be worse depending on the specific CPUs from what I was testing on our gear a couple of months back. Our R720s weren't quite rendered useless but it's been a huge factor for new gear for our XenDesktop VDI hosts. The newer procs in the R730s at another site didn't seem to be hit as hard, but it was still significant.
-
@scottalanmiller so what should I do?
-
@scottalanmiller I also recall you telling me years ago, that there was a limit to how many hard drives (spinning rust) I could add to a raid 10 depending on if I was looking for a boost in either read or write. If I recall, read was the only thing that would speed up.
-
@notverypunny See I want to get on the latest and greatest release. I just want to avoid having to nuke everything.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller I also recall you telling me years ago, that there was a limit to how many hard drives (spinning rust) I could add to a raid 10 depending on if I was looking for a boost in either read or write. If I recall, read was the only thing that would speed up.
That's RAID 1, not RAID 10.
-
@notverypunny I mean after looking around on the net I see what Scott is saying the better performance might actually be better with thick provisioning, but it's negligible when people run analytics on it vs thin. Like only a couple of %.
But I think I see what Scott is also saying that maybe one of the HDD's if possibly throwing errors or failing. I've already replaced 2 drives in 2 years.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller so what should I do?
Start by looking at the RAID controller and see if the space has been set up yet.
-
@scottalanmiller on one of our servers it's properly done. One the other not so much. I'm going to get the dell engineer I originally worked with to fix the one that isn't working right. So when we use Dell's tool to look at the raid it shows the proper capacity expansion on one server. The other server doesn't show the increase.
When I go into Xen Center, it doesn't "dynamically" see the adjust space. So I see what you're saying I was hoping it would automagically be there, not the case. So how do I get it to add it?
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller on one of our servers it's properly done. One the other not so much. I'm going to get the dell engineer I originally worked with to fix the one that isn't working right. So when we use Dell's tool to look at the raid it shows the proper capacity expansion on one server. The other server doesn't show the increase.
When I go into Xen Center, it doesn't "dynamically" see the adjust space. So I see what you're saying I was hoping it would automagically be there, not the case. So how do I get it to add it?
Use XAPI interface to create a new SR or go through the same process using XO and a gui.
-
Create a new SR? Wouldn't that break something? I'm confident when adding a single or more hard drive to the SR. I'm not very confident when we talking about the RAID
it doesn't seem to make sense. -
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Create a new SR? Wouldn't that break something? I'm confident when adding a single or more hard drive to the SR. I'm not very confident when we talking about the RAID
it doesn't seem to make sense.You need to create a new SR or expand the existing one. It's a logical volume on top of the block device.
Since expanding a production LV is likely more risky, it's easier to just create a new storage repository on your block device.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Create a new SR? Wouldn't that break something? I'm confident when adding a single or more hard drive to the SR. I'm not very confident when we talking about the RAID
it doesn't seem to make sense.It's all making perfect sense. Thinking of RAID as a special case is confusing you.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller on one of our servers it's properly done. One the other not so much. I'm going to get the dell engineer I originally worked with to fix the one that isn't working right. So when we use Dell's tool to look at the raid it shows the proper capacity expansion on one server. The other server doesn't show the increase.
Oh, that would cause an issue if it didn't really expand.
