Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance
-
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
He definitely has an LVM on the hardware RAID. He may or may not have Linux LVM2 in software as well. Two standard places for it to exist. So we know he has one, he might have two. Whatever ones are there, have to be configured.
-
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
-
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
Technically those patches killed performance for everyone, not just XenServer.
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
The SPECTRE / MELTDOWN patches absolutely killed XS performance for some of our workloads. Ensure that your BIOS and other FW is up to date and consider trying XCP-ng.
Technically those patches killed performance for everyone, not just XenServer.
Fair point. If he's limited to 7.1 for whatever reason it'll be worse, they managed to claw back some of the performance in later releases, but the last patches on 7.1 were deadly. Seems to be worse depending on the specific CPUs from what I was testing on our gear a couple of months back. Our R720s weren't quite rendered useless but it's been a huge factor for new gear for our XenDesktop VDI hosts. The newer procs in the R730s at another site didn't seem to be hit as hard, but it was still significant.
-
@scottalanmiller so what should I do?
-
@scottalanmiller I also recall you telling me years ago, that there was a limit to how many hard drives (spinning rust) I could add to a raid 10 depending on if I was looking for a boost in either read or write. If I recall, read was the only thing that would speed up.
-
@notverypunny See I want to get on the latest and greatest release. I just want to avoid having to nuke everything.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller I also recall you telling me years ago, that there was a limit to how many hard drives (spinning rust) I could add to a raid 10 depending on if I was looking for a boost in either read or write. If I recall, read was the only thing that would speed up.
That's RAID 1, not RAID 10.
-
@notverypunny I mean after looking around on the net I see what Scott is saying the better performance might actually be better with thick provisioning, but it's negligible when people run analytics on it vs thin. Like only a couple of %.
But I think I see what Scott is also saying that maybe one of the HDD's if possibly throwing errors or failing. I've already replaced 2 drives in 2 years.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller so what should I do?
Start by looking at the RAID controller and see if the space has been set up yet.
-
@scottalanmiller on one of our servers it's properly done. One the other not so much. I'm going to get the dell engineer I originally worked with to fix the one that isn't working right. So when we use Dell's tool to look at the raid it shows the proper capacity expansion on one server. The other server doesn't show the increase.
When I go into Xen Center, it doesn't "dynamically" see the adjust space. So I see what you're saying I was hoping it would automagically be there, not the case. So how do I get it to add it?
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller on one of our servers it's properly done. One the other not so much. I'm going to get the dell engineer I originally worked with to fix the one that isn't working right. So when we use Dell's tool to look at the raid it shows the proper capacity expansion on one server. The other server doesn't show the increase.
When I go into Xen Center, it doesn't "dynamically" see the adjust space. So I see what you're saying I was hoping it would automagically be there, not the case. So how do I get it to add it?
Use XAPI interface to create a new SR or go through the same process using XO and a gui.
-
Create a new SR? Wouldn't that break something? I'm confident when adding a single or more hard drive to the SR. I'm not very confident when we talking about the RAID
it doesn't seem to make sense. -
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Create a new SR? Wouldn't that break something? I'm confident when adding a single or more hard drive to the SR. I'm not very confident when we talking about the RAID
it doesn't seem to make sense.You need to create a new SR or expand the existing one. It's a logical volume on top of the block device.
Since expanding a production LV is likely more risky, it's easier to just create a new storage repository on your block device.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
Create a new SR? Wouldn't that break something? I'm confident when adding a single or more hard drive to the SR. I'm not very confident when we talking about the RAID
it doesn't seem to make sense.It's all making perfect sense. Thinking of RAID as a special case is confusing you.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@scottalanmiller on one of our servers it's properly done. One the other not so much. I'm going to get the dell engineer I originally worked with to fix the one that isn't working right. So when we use Dell's tool to look at the raid it shows the proper capacity expansion on one server. The other server doesn't show the increase.
Oh, that would cause an issue if it didn't really expand.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
When I go into Xen Center, it doesn't "dynamically" see the adjust space. So I see what you're saying I was hoping it would automagically be there, not the case. So how do I get it to add it?
First you have to add it in the LVM on the RAID controller after the RAID is expanded. THEN you can look at growing what XC sees.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@notverypunny See I want to get on the latest and greatest release. I just want to avoid having to nuke everything.
I feel your pain, but in the interests of being able to sleep at night you might be better off moving your stuff and starting from scratch.
In your position I would have created a new virtual disk on the raid controller and added it as a new SR, mainly because I've never considered the option of expanding an existing volume, seems risky without any real benefit that I can see right off the bat.
-
@notverypunny said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@notverypunny See I want to get on the latest and greatest release. I just want to avoid having to nuke everything.
I feel your pain, but in the interests of being able to sleep at night you might be better off moving your stuff and starting from scratch.
In your position I would have created a new virtual disk on the raid controller and added it as a new SR, mainly because I've never considered the option of expanding an existing volume, seems risky without any real benefit that I can see right off the bat.
This assumes you can just add more mountpoints to whatever VMs he's using. In some situations you simply want to grow the disk you're using, not add a whole new disk to the VM.
-
@scottalanmiller said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
When I go into Xen Center, it doesn't "dynamically" see the adjust space. So I see what you're saying I was hoping it would automagically be there, not the case. So how do I get it to add it?
First you have to add it in the LVM on the RAID controller after the RAID is expanded. THEN you can look at growing what XC sees.
RAID controller - grow RAID 10 from 4 to 6 disks (or whatever new number you have)
RAID controller - Use LVM to grow the virtual disk the RAID controller presents to Xen Server
Xen Server - Use LVM to grow the SR to include the new space it sees now.
Xen Server - do whatever you need for the VMs to use the new space.
