Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD
-
As the title suggests, I'm at a client's where for some reason, transferring large files to their file server has slowed to a crawl. What typically happens is the transfer begins as normal with throughput around 350MBps to 400MBps (yes, their host is connected to Ubiquiti ES switch 10Gb port and primary video dev workstation is connected to second 10Gb port) but then after 1 or 2 seconds, the transfer speed drops to 0 for a couple minutes and when transfer resumes, it hovers between 500Kbps to 2MBps. The real kicker is that, when their IT guy transfers the same file from the workstation to the Hyper-V host instead of the file server VM, transfer just speeds along at 500MBps to 600MBps.
Host is running Windows Server 2012 R2 with Hyper-V role (their resident IT guy isn't comfortable without the GUI). The host and file server VM are both fully patched. Jumbo frames is enabled throughout including switch. The VM is running Windows Server 2012 R2 and its sole purpose is as a file server. Workstation is running Windows 10 Pro (fully updated). The file server VM has a 40TB data disk with >7TB of available space. The host has >15TB available space.
As a second test, tried to transfer file from the Hyper-V host that is hosting the file server VM to the file server VM and the transfer is the same garbage performance as from workstation to file server V.
When I run ping -t tests from the workstation to the host, replies are consistently <1ms or 1ms and 0% loss. Same ping -t tests to the file server VM result in <1ms to as high as 11ms with 0% loss.
It seems to me that the issue is with the VM itself but can't find anything wrong. Any suggestions as to where else I can look?
-
Is the same thing occurring if you attempt to pull a file from the VM to your system?
-
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Is the same thing occurring if you attempt to pull a file from the VM to your system?
Yes, same issue exists whether transferring to or from the file server VM.
-
Also check that VMQ is disabled on your NICS.
Check out the Disable-NetAdapterVMQ powershell commands.
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
I've made the recommendation to try turning off jumbo frames when I first took the client on and they did a few tests with no changes in performance so rather than change every PC and server and switch, they decided to keep it on (they had only tested turning it off to see if there was a difference).
-
I have limited experience with Hyper-v
Where is the storage? Local? iscsi?
Is the host and guest on the same network? Is the router getting overwhelmed? Does the router even get traffic when this happens?
what does the switch say? You should be able to log in and check its realtime log. -
I should also mention there are no iSCSI issues here (not enabled).
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
I've made the recommendation to try turning off jumbo frames when I first took the client on and they did a few tests with no changes in performance so rather than change every PC and server and switch, they decided to keep it on (they had only tested turning it off to see if there was a difference).
The only reasonable answer here was if there was no change, to have it disabled so the switch and systems operate in the most common fashion.
I'd recommend going and disabling it.
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
I have limited experience with Hyper-v
Where is the storage? Local? iscsi?
Is the host and guest on the same network? Is the router getting overwhelmed? Does the router even get traffic when this happens?
what does the switch say? You should be able to log in and check its realtime log.You just beat me to it. Correct, everything is local storage.
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
I've made the recommendation to try turning off jumbo frames when I first took the client on and they did a few tests with no changes in performance so rather than change every PC and server and switch, they decided to keep it on (they had only tested turning it off to see if there was a difference).
The only reasonable answer here was if there was no change, to have it disabled so the switch and systems operate in the most common fashion.
I'd recommend going and disabling it.
If jumbo frames was the issue, I'd expect the file transfer to the host to also be affected but it's not. That's performing really, really well. It's only to the VM where the issue is seen.
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
I've made the recommendation to try turning off jumbo frames when I first took the client on and they did a few tests with no changes in performance so rather than change every PC and server and switch, they decided to keep it on (they had only tested turning it off to see if there was a difference).
The only reasonable answer here was if there was no change, to have it disabled so the switch and systems operate in the most common fashion.
I'd recommend going and disabling it.
If jumbo frames was the issue, I'd expect the file transfer to the host to also be affected but it's not. That's performing really, really well. It's only to the VM where the issue is seen.
Has this issue been tested with JF disable, or was JF disabled just to see what would happen.
Generally unless you have a good reason, just leave the default settings alone. Is the rule of thumb to go by.
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
When I run ping -t tests from the workstation to the host, replies are consistently <1ms or 1ms and 0% loss. Same ping -t tests to the file server VM result in <1ms to as high as 11ms with 0% loss.
11ms is high for a local ping.
From the workstation
try a tracert to host
try tracert to vmFrom host
do a tracert to vm
do a tracert to workstationHow is the vnic on the guest setup?
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
When I run ping -t tests from the workstation to the host, replies are consistently <1ms or 1ms and 0% loss. Same ping -t tests to the file server VM result in <1ms to as high as 11ms with 0% loss.
11ms is high for a local ping.
From the workstation
try a tracert to host
try tracert to vmFrom host
do a tracert to vm
do a tracert to workstationHow is the vnic on the guest setup?
All results look good.
Workstation to VM Guest = 1 1ms <1ms <1ms FILE-SERV [192.168.100.83] Workstation to Host = 1 1ms <1ms <1ms vHOST [192.168.100.232] VM Guest to Workstation = 1 1ms <1ms <1ms VID-STATION [192.168.100.71] Host to Workstation = 1 1ms <1ms <1ms VID-STATION [192.168.100.71]The vNic is a simple external network. Both workstation and host physical NICs are Intel X520-DA1 (drivers are current).
-
@momurda said in [Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD]
11ms is high for a local ping.
Very high. Everything here is less then 1ms.
-
Hmm, well that looks fine.
All the machines on same network.Perhaps the storage cannot keep up with 10G (or 5G). What is the local storage? Spinning raid 10? Check the 2012 file server event viewer for storage timeout/retries.
edit: whoops i forgot xfer to host is fast -
@minion said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@momurda said in [Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD]
11ms is high for a local ping.
Very high. Everything here is less then 1ms.
Agreed, which is why I'm leaning towards there being something up with the VM, especially since transfers to and from the host are perfect. At a loss for where to look next as I've eliminated the usual suspects.
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Hmm, well that looks fine.
All the machines on same network.Perhaps the storage cannot keep up with 10G (or 5G). What is the local storage? Spinning raid 10? Check the 2012 file server event viewer for storage timeout/retries.
The storage can definitely handle the 10G. As I mentioned earlier, I'm easily able to reach transfer speeds of 600MBps to and from the host. It is configured as RAID10 nearline 10TB Seagate drives.
-
Are the drivers up to date inside of the VM? I know you said that they are for the nic but it was a bit vague and could've been only for the hypervisor and not the VM its self.
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Are the drivers up to date inside of the VM? I know you said that they are for the nic but it was a bit vague and could've been only for the hypervisor and not the VM its self.
All PCs and servers and VMs are fully up to date. If there is one thing they do well at this place is patch/update often and early.
