SAN data transfer - very slow
-
I have a SAN attached via fiber to a Windows server in a colo, 250 miles of 10G fiber down the road (we own the fiber) I have another colo that has a SAN attached via fiber to a Windows server. I'm trying to transfer a 2.5TB VM from one SAN to the other and it is only getting around 50MB/s on average of transfer.
What am I missing? This transfer should be zooming up into the hundreds at least as there isn't a single person in the company besides me that is on the network using anything on either SAN (paid holiday). So it's not a load issue...
-
Is the SAN and server directly connected to the fibre? Or is it 1gbe to the devices and on the wire 10Gbe?
-
@DustinB3403 said in SAN data transfer - very slow:
Is the SAN and server directly connected to the fibre? Or is it 1gbe to the devices and on the wire 10Gbe?
The SAN is hooked to the server via FC, then 1Gbe from the device to the switch, the path between the colos is 10Gbe. At least that's my understanding as I've never been to the other colo and I've only been to the one here once when I started.
-
From the description I would think the issue is due to the connection between the storage and the WAN
-
@DustinB3403 said in SAN data transfer - very slow:
From the description I would think the issue is due to the connection between the storage and the WAN
I'm thinking it is just that I have an iSCSI connection that is trying to go hundreds of miles. I know that's asking for trouble, but I'm not sure how to do a reasonably timed data transfer otherwise... short of having the data physically shipped to me.
-
Are you able to have the connection between sites directly connected, so you can use wire speed?
What kind of delta is used here?
-
@DustinB3403 said in SAN data transfer - very slow:
Are you able to have the connection between sites directly connected, so you can use wire speed?
What kind of delta is used here?
Sorry, I don't think I understand the two questions very clearly... Can you clarify a little for me please?
-
Can you have the 10Gbe directly attached to the storage between the colos?
-
@DustinB3403 said in SAN data transfer - very slow:
Can you have the 10Gbe directly attached to the storage between the colos?
You mean don't have the SAN hooked up to the Windows server? How would I manage what is on the LUN if I did that?
-
@BBigford said in SAN data transfer - very slow:
@DustinB3403 said in SAN data transfer - very slow:
Is the SAN and server directly connected to the fibre? Or is it 1gbe to the devices and on the wire 10Gbe?
The SAN is hooked to the server via FC, then 1Gbe from the device to the switch, the path between the colos is 10Gbe. At least that's my understanding as I've never been to the other colo and I've only been to the one here once when I started.
So the bottleneck is 1Gb/s and 50MB/s is about half of your theoretical max of around 100-120 MB/s so really, that is not too bad when you take into account the TCP Windowing overhead and latency and such. Could it go faster? Maybe. Reasonably? Not much.
-
@DustinB3403 said in SAN data transfer - very slow:
Can you have the 10Gbe directly attached to the storage between the colos?
Not likely, it's SAN. You can't have SAN talk to SAN in 99% of cases. You'd need some seriously special logic at all layers to allow for that. And it is FC so you would need to figure out how to get FC over IP which is not a standard protocol. So layer after layer of problems there.
-
@BBigford said in SAN data transfer - very slow:
@DustinB3403 said in SAN data transfer - very slow:
Can you have the 10Gbe directly attached to the storage between the colos?
You mean don't have the SAN hooked up to the Windows server? How would I manage what is on the LUN if I did that?
He's thinking, I think, of making a raw RAID 1 over the WAN. It will not work well, it would likely end in tears
-
@BBigford I agree with @scottalanmiller, that's close to what I would expect to see over a 1Gb link.
-
@scottalanmiller said in SAN data transfer - very slow:
@BBigford said in SAN data transfer - very slow:
@DustinB3403 said in SAN data transfer - very slow:
Is the SAN and server directly connected to the fibre? Or is it 1gbe to the devices and on the wire 10Gbe?
The SAN is hooked to the server via FC, then 1Gbe from the device to the switch, the path between the colos is 10Gbe. At least that's my understanding as I've never been to the other colo and I've only been to the one here once when I started.
So the bottleneck is 1Gb/s and 50MB/s is about half of your theoretical max of around 100-120 MB/s so really, that is not too bad when you take into account the TCP Windowing overhead and latency and such. Could it go faster? Maybe. Reasonably? Not much.
Can you do a rough break down on the math of how you got the theoretical, please? So what are my options, short of waiting +48 hours, or having the data physically shipped to me?
-
@BBigford said in SAN data transfer - very slow:
@scottalanmiller said in SAN data transfer - very slow:
@BBigford said in SAN data transfer - very slow:
@DustinB3403 said in SAN data transfer - very slow:
Is the SAN and server directly connected to the fibre? Or is it 1gbe to the devices and on the wire 10Gbe?
The SAN is hooked to the server via FC, then 1Gbe from the device to the switch, the path between the colos is 10Gbe. At least that's my understanding as I've never been to the other colo and I've only been to the one here once when I started.
So the bottleneck is 1Gb/s and 50MB/s is about half of your theoretical max of around 100-120 MB/s so really, that is not too bad when you take into account the TCP Windowing overhead and latency and such. Could it go faster? Maybe. Reasonably? Not much.
Can you do a rough break down on the math of how you got the theoretical, please? So what are my options, short of waiting +48 hours, or having the data physically shipped to me?
I can't break down the math for you off the top of my head. I've got lots of 1Gb links around here tho, so I can give you the output of iperf (test the max speed a network can go.)
Single transfer thread:
iperf -c 192.168.0.45 ------------------------------------------------------------ Client connecting to 192.168.0.45, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.0.36 port 52014 connected with 192.168.0.45 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 105 MBytes 88.4 Mbits/secFour threads in parallel:
iperf -P 4 -c 192.168.0.45 ------------------------------------------------------------ Client connecting to 192.168.0.45, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.0.36 port 52086 connected with 192.168.0.45 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 105 MBytes 88.1 Mbits/secNow if you had 10 Gb network all through you should really be concerned. With a 1Gb link, I think you get the choice of waiting 48+ hours or having someone ship you a drive or ship it over. (Truck loaded with drives is still way more bandwidth, ping just takes days!)
-
@BBigford said in SAN data transfer - very slow:
So the bottleneck is 1Gb/s and 50MB/s is about half of your theoretical max of around 100-120 MB/s so really, that is not too bad when you take into account the TCP Windowing overhead and latency and such. Could it go faster? Maybe. Reasonably? Not much.
Can you do a rough break down on the math of how you got the theoretical, please? So what are my options, short of waiting +48 hours, or having the data physically shipped to me?
Well 1GigE is your network bottleneck from the description, everything faster than that doesn't have a direct connection so aren't much of a factor (other than allowing the TCP Windows to mismatch causing more overhead.) So we start with 1,000 Mb/s.
1,000 / 8 = 125 MB/s (bits to bytes.)
So that is the theoretical max of the link with zero overhead. Rule of thumb is to assume that the actual max is 80% of Ethernet. That is 100MB/s.
So 100MB/s is pretty much the theoretical fastest that you could get over pure GigE without any bridging, switching or other slowing factors.
No consider that you are putting storage protocols onto that which means that you have some bottlenecks, at least from time to time, from the storage system so you likely will not be able to feed that perfectly. Then each conversion from Ethernet to whatever the WAN link is will add overhead in the form of the translation (different window sizes and other efficiency issues.) And then you have to add in the latency of the link - latency causes a throughput reduction when we are talking about a reliable protocol like TCP. This would not be the case for a streaming UDP connection so much, but you can't send storage data in an unreliable, best effort way so that doesn't matter. You are forced to wait for receipt responses.
Put this all together, and cutting your connection in half is normal.
