Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@black3dynamite Scott has been begging me for a while to switch over to KVM
I have enough servers at home to give it a try. What are you using for host management? Last I saw, Kimichi(sp?) was hot.You can also use Cockpit.
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
it barely gets better on any host. I have roughly 10-20 vm's on each host. 3 at work are in production, 1 server at my home is being setup. Out of the 4 servers, 2 are brand new. The others aren't super old just 2.5 years old. I have my own lab and 2 servers also. Only 1 of my home servers is in use, but its running a Raid 0 for 2 10K RPM drives. All my servers have crappy speed when it pertains to windows file transfer. I do use linux also but I'm not as proficient with it so I focus right now only on Windows. My new servers are R540's one with full SSD's the other with 4 SAS 10K RPM hdd's. Only the full ssd host is in production. XCP-NG is installed on a BOSS ssd.
Just for reference, my home server is a Dell R220 and a R710. I've installed XCP-NG on everything you can imagine, still same issue. Bottleneck of Windows and I did confirm on my R710 it does use a broadcom nic.
I have a 10G and a 12G Dell in production at work, and an older HP server at home, all with XCP-ng. None of them have storage speed issues. Mix of thin and thick, too. I definitely prefer thin.
I hate to say this, because it seems obvious, but are you transferring millions of tiny files? Because that is always slow AF.
-
I would poke around https://xcp-ng.org/forum/category/2/storage a little and maybe there are other suggestions for tweaking your storage performance. I know it comes up a lot over there.
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@black3dynamite Scott has been begging me for a while to switch over to KVM
I have enough servers at home to give it a try. What are you using for host management? Last I saw, Kimichi(sp?) was hot.For host management, it will be Cockpit. Just make sure
cockpit-machinesplugin is installed.
sudo dnf install cockpit cockpit-storaged cockpit-machines -
@bnrstnr well hate to tell you but you hit the nail on the head. It's literally 1000's of small files between 100-500k.
-
@black3dynamite thats why you the man, I sure will.
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@bnrstnr well hate to tell you but you hit the nail on the head. It's literally 1000's of small files between 100-500k.
Try zipping them into a single, or multiple large files, then transfer.
I have found that transferring via xcopy is much faster for small files than just drag and drop in windows.
-
@Dashrender said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
I have found that transferring via xcopy is much faster for small files than just drag and drop in windows.
I use Robocopy, I believe it's more powerful, but I think it's a little more complex too. Either will be way better at handling all of those small files than copying them in explorer.
-
@bnrstnr said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@Dashrender said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
I have found that transferring via xcopy is much faster for small files than just drag and drop in windows.
I use Robocopy, I believe it's more powerful, but I think it's a little more complex too. Either will be way better at handling all of those small files than copying them in explorer.
yeah, xcopy when it's a simple copy.. Robo when you need more power.
-
@Dashrender I actually started the job off with Robocopy. :0) 500 GB got transferred after about 12 hours or so. Thats SLOW.
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@Dashrender I actually started the job off with Robocopy. :0) 500 GB got transferred after about 12 hours or so. Thats SLOW.
well, small files.
Now, make 500 1 GB files and see how fast it goes.
-
@Dashrender said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@Dashrender I actually started the job off with Robocopy. :0) 500 GB got transferred after about 12 hours or so. Thats SLOW.
well, small files.
Now, make 500 1 GB files and see how fast it goes.
Or 1 500GB file. Even better.
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
@Dashrender I actually started the job off with Robocopy. :0) 500 GB got transferred after about 12 hours or so. Thats SLOW.
Did you add the switch to use Multiple CPU threads?
-
The data is from one of my Ubiquiti UniFi Video (NVR) server. So the files are not really big. So are you proposing like maybe zipping the files into one file then transferring?
-
@bnrstnr That I did not, I'll look back over the command tomorrow. I have one of my servers home and will be testing the BIOS settings change.
-
@krisleslie said in Using XCP-NG latest release, Windows 2019 Server VM file copies are pegged at 10 MB/s:
The data is from one of my Ubiquiti UniFi Video (NVR) server. So the files are not really big. So are you proposing like maybe zipping the files into one file then transferring?
If they are pre-compressed, then just tar them, don't zip them.