Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD
-
@momurda said in [Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD]
11ms is high for a local ping.
Very high. Everything here is less then 1ms.
-
Hmm, well that looks fine.
All the machines on same network.Perhaps the storage cannot keep up with 10G (or 5G). What is the local storage? Spinning raid 10? Check the 2012 file server event viewer for storage timeout/retries.
edit: whoops i forgot xfer to host is fast -
@minion said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@momurda said in [Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD]
11ms is high for a local ping.
Very high. Everything here is less then 1ms.
Agreed, which is why I'm leaning towards there being something up with the VM, especially since transfers to and from the host are perfect. At a loss for where to look next as I've eliminated the usual suspects.
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Hmm, well that looks fine.
All the machines on same network.Perhaps the storage cannot keep up with 10G (or 5G). What is the local storage? Spinning raid 10? Check the 2012 file server event viewer for storage timeout/retries.
The storage can definitely handle the 10G. As I mentioned earlier, I'm easily able to reach transfer speeds of 600MBps to and from the host. It is configured as RAID10 nearline 10TB Seagate drives.
-
Are the drivers up to date inside of the VM? I know you said that they are for the nic but it was a bit vague and could've been only for the hypervisor and not the VM its self.
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Are the drivers up to date inside of the VM? I know you said that they are for the nic but it was a bit vague and could've been only for the hypervisor and not the VM its self.
All PCs and servers and VMs are fully up to date. If there is one thing they do well at this place is patch/update often and early.
-
I'd go back to looking at the NICs. In Hyper-V the management console can have one physical NIC, and the VMs another physical NIC. Ensure that VMQ is disabled as @dafyre mentioned, and check the configuration of the VM NIC. You may want to try swapping the NIC assignments and then try your tests again.
If teaming is enabled for the VM NICs, I would check this config, and possibly dissolve the team to test.
-
Checking the vmqueue first.
Then try benchmarking disks on host and vm with any free tool just to understand if this is a disk issue in the vm or a networking one -
How about the physical port on the switch? change where the VM host is plugged in, test, then change the PC and test.
-
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also check that VMQ is disabled on your NICS.
Check out the Disable-NetAdapterVMQ powershell commands.
So much this. Check all pieces. The above command can disable it on the host, but the guest also has a setting that should be disabled.
-
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
-
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
-
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
-
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
Yepp. Firmware is up to date (did an update about 2 months ago, I think). The systems have been operational for a year or better.
It's all Cisco gear on the networking side. I've read the same thing that you have that says 10gig is fine to enable VMQ, but on Intel & Broadcomm NICs, it is not.
-
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
Yepp. Firmware is up to date (did an update about 2 months ago, I think). The systems have been operational for a year or better.
It's all Cisco gear on the networking side. I've read the same thing that you have that says 10gig is fine to enable VMQ, but on Intel & Broadcomm NICs, it is not.
Those are the NICs I'm talking about, Intel & Broadcom. Maybe I just have un-effected models in all of the servers I've dealt with.
-
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
Yepp. Firmware is up to date (did an update about 2 months ago, I think). The systems have been operational for a year or better.
It's all Cisco gear on the networking side. I've read the same thing that you have that says 10gig is fine to enable VMQ, but on Intel & Broadcomm NICs, it is not.
Those are the NICs I'm talking about, Intel & Broadcom. Maybe I just have un-effected models in all of the servers I've dealt with.
I hope so, lol. But if you start having problems with Intel or Broadcomm NICs, that' the first place I'd start.
-
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
Yepp. Firmware is up to date (did an update about 2 months ago, I think). The systems have been operational for a year or better.
It's all Cisco gear on the networking side. I've read the same thing that you have that says 10gig is fine to enable VMQ, but on Intel & Broadcomm NICs, it is not.
Those are the NICs I'm talking about, Intel & Broadcom. Maybe I just have un-effected models in all of the servers I've dealt with.
I hope so, lol. But if you start having problems with Intel or Broadcomm NICs, that' the first place I'd start.
I have. And it was always with old-firmware'd 1Gb broadcoms. Never 10Gb.
