Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance
-
Running an older version 7.1 and will be looking to move to the latest/greatest build this summer.
-
Dual Socket Host
-
128 GB of RAM
-
16 TB of raw storage, using a Raid 10 /w Dell Per H730 (I don't know to make the Xen Server see the added storage, I added 8 TB of storage to my RAID 10 /w help from Dell Technical Support. The storage is "there" but the hypervisor doesn't see the added storage.
-
Disks are all spinning rust nearline sas
-
VM's are thin provisioned
-
Windows and Linux VM's
Ram and CPU aren't overcommitted. Things are working but we are definitely seeing plenty of times where the speed is not where it use to be.
-
-
For adding the storage to your XS installation you need to create an SR on the added storage and mount that SR to your XS installation.
As for the performance issues have you asked Dell or someone like @scottalanmiller to get Dpack running and see where the performance might be?
-
@DustinB3403 I started back using Dell Live Optics, I need to finish adding it to each VM server so we can determine the performance characteristics. I'm just deeply assuming the thin provisioning is the culprit but that's without proof, only assumptions.
-
@DustinB3403 so my original installation of Xen Server was to 4 disks, raid 10. When we added the additional 4, dell did a "live" update to the raid configuration so that it added the capacity. That part looks fine but then from Xen Server I'm under the "assumption" it will see the Raid has changed automagically and expand but alas it didn't
So your saying I need to add a new SR? Is that different than expanding the existing one?
-
Just an idea but if you can add another SR to the storage array it might actually be a good idea to do that as a test. Thin or thick provisioning is per SR so you can make the new one thick provisioned, move a VM there and see if it improves the performance.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 so my original installation of Xen Server was to 4 disks, raid 10. When we added the additional 4, dell did a "live" update to the raid configuration so that it added the capacity. That part looks fine but then from Xen Server I'm under the "assumption" it will see the Raid has changed automagically and expand but alas it didn't
So your saying I need to add a new SR? Is that different than expanding the existing one?
Yes, XenServer doesn't care what you did to the physical block device, as it's "block device" is set to the limits of the original OBR10.
Add a second SR as a thick provisioned SR using the remaining space and see how it performs.
-
@krisleslie said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 so my original installation of Xen Server was to 4 disks, raid 10. When we added the additional 4, dell did a "live" update to the raid configuration so that it added the capacity. That part looks fine but then from Xen Server I'm under the "assumption" it will see the Raid has changed automagically and expand but alas it didn't
So your saying I need to add a new SR? Is that different than expanding the existing one?
RAID is an underlying mechanism. Very underlying. So you have several layers on top that need to be configured in order for additional capacity to be available to things running on the server, like Xen.
-
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
-
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
-
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
Xen server supports LVM but he may not have set it up for his storage repository
-
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
Xen server supports LVM but he may not have set it up for his storage repository
It's the default to not use LVM, right?
-
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@Dashrender said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@black3dynamite said in Having sluggish performance on my Xen Server VM's, looking for suggestions to boost performance:
@DustinB3403 Been a while since I've used XenServer, wouldn't he need to power down the VMs, unplug the block device, and then use LVM to expand the storage?
I have to look up the exact process but that sounds about right but because it's not worth it and he's trying to troubleshoot another issue I would literally just create a separate sr and call it thick provision and leave it at that
I don't know what I'm saying - but I'm asking anyway - does he even have LVM?
Xen server supports LVM but he may not have set it up for his storage repository
It's the default to not use LVM, right?
The default is LVM... They should probably update that.
-
@Dashrender idk. Never really paid that close attention to it.
-
@DustinB3403 Yea because I've seen all the advances made on 8 release, so I anxiously want to move to that. But I know until I can get another server here its a bit tough.
-
Im thinking of getting a 3rd server that can just have enough storage to move my vm's over, then nuke the old host one by one or upgrade them. I'm not sure if 7.1 is upgraded to the XCP-NG releases.
-
12 TB of storage and 128GB or RAM isn't much, I have 24TB raw in my lab alone, the RAM I don't need and don't have.
You could purchase a cheap server and get that much capacity for cheap.
-
So you have dual socket CPU with 128 GB of RAM. Which CPUs do you have?
In XenCenter you can select the host and then it's under the General tab and the CPUs section.And then 16TB disks in 8x2TB config with 7200 rpm SAS drives. So 8TB usable space and your using close to 50% now?
Do you know how much cache your H730 RAID controller has?
VM's are thin provisioned and it's a mix of Windows and Linux VM's. How many VMs do you have running and how much of the RAM is provisioned now?
To get some info on performance, in XenCenter select the host and go to the Performance tab. Right click on the graph and select Actions>New graph... Make a graph with IOwait on your SR. Post a screen shot of it together with CPU performance when you are having sluggish performance.
-
@DustinB3403 yea I know it’s not a lot. We are a somewhat small non profit. Honestly I would rather max out both servers ram wise and switch to all SSDs since the price is literally the same for Nearline SAS.
I’m assuming the thin provisioning is the culprit but that’s only an assumption. I recall reading that it would slow down in time vs thick provisioning being a bit speedier.
-
Technically right now the host is 16 Tb raw storage and 8 Tb usable (maybe a notch less) since it’s raid 10 but the XenServer HVM doesn’t see the extra 4 Tb yet. I dunno why 🤨. See I’ve done capacity expansions before on other additional single drives no problem. But on the raid I’m lost. Since it was “added” capacity from bios, I assume the hyper visor would just “see” it.
