How would you build a File server with 170TB of Usable Storage?
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
I'm not familiar with ZFS.
ZFS is specifically the filesystem made by SUN for exactly this purpose. Before there was Storinator, before there were BackBlaze Pods, there was the SUN Thumper - an enterprise class, top loading, customer engineering storage server made to hold an insane number of drives and keep them safe and operational in a single chassis. All those others are attempts to replicate it without the enterprise components. The difference is, Thumper was designed to be a server, the Pods were not, but rather nodes.
The ZFS / Thumper team is who worked with me to design the first SAM-SD as well.
ZFS was super important to making Thumper work as it added RAID 7 (aka RAID 5-3) as an option, and all kinds of cache control, and a file system that could scale as big as Thumper could go.
-
@black3dynamite said in How would you build a File server with 170TB of Usable Storage?:
LVM Thin Pool and XFS for a file server?
Yes, XFS is super reliable and pretty fast.
-
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpg -
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgWell sure, if you want to do it the easy way.
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
I would use a real server platform, no matter what the product need is, I use production gear. That's a starting point, even when you need it cheap, you still have to support it.
Not sure why I didn't think of this before. Why not use storage shelves?
If I'm using storage shelves (drive enclosures), then a RAID card + 2 or 3 Shelves to get me up to enough drives to hold my data at RAID10 would be a more viable option. That way I could pretty much whatever I want as the base server and go from there.
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
For 170TB, if it were me, I'd have a RAIN cluster. You didn't go into all of your theoretical details, but assuming pretty common needs, Gluster might be a good choice across a handful of chassis.
I made it theoretical so you could use your imagination a little.
From the get-go, I expected somebody to come back with something like Gluster a little bit quicker. Yes, I realize that adds complexity and networking needs and such. It does solve the problem though. Building 3 nodes with enough storage and using Gluster to make the entire 170GB available for clients to use.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
If I'm using storage shelves (drive enclosures), then a RAID card + 2 or 3 Shelves to get me up to enough drives to hold my data at RAID10 would be a more viable option. That way I could pretty much whatever I want as the base server and go from there.
Because they are a pain at this size. You can get more than enough internal storage for 170TB, no need for shelves. But yes, shelve work.
No need for a RAID card, software RAID will be better for this use case.
-
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgha ha ha.
That's actually a pretty neat setup. I configured it on their site for kicks & grins using 18 x 4TB NVME drives... Yours for only $60k... Gets you up to 36TB in RAID10...
-
If you need more storage density you could also go with another chassis using the NF1 format SSDs from Samsung.
That's a whopping 0.54 PB in 1U.@dafyre I checked our supplier and fully stocked with 36 15TB Samsung PM983 NVMe drives it would be priced at around $115K with 768GB RAM and two Xeon CPUs and a couple of 25Gb NICs.
https://www.supermicro.com/a_images/products/views/1029P-NMR36L_angle.jpg
-
I need to dig in and see what Chasis our new Nimble Storage units are (They're the new ones form HPE). I know those run about $60k each. I need to see how much storage we get in each.
-
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgThis is the best answer for the given question. Anything else is going into assumption land which was specifically not allowed.
-
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
Your own configuration does not provide 170TB usable. You calculated using unformatted capacity, and didn't take into account that even unformatted, a 14TB drive is only a decimal representation used for marketing, which takes another chunk out of the usable capacity
-
@Vlinderbeest said in How would you build a File server with 170TB of Usable Storage?:
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
Your own configuration does not provide 170TB usable. You calculated using unformatted capacity, and didn't take into account that even unformatted, a 14TB drive is only a decimal representation used for marketing, which takes another chunk out of the usable capacity
Plus RAID and FS overhead. With 14TB drives using mirroring, you'd need at least 26 drives, and 28 to be safe.
-
Just for the amusement factor, I just checked out one of the 90 bay SuperMicro chassis. With 14TB drives, that's 500TB usable in a single 4u chassis. All for around $60k
-
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
Tee hee. Now that's pretty funny. Even HPE uses SuperMicro!
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
Tee hee. Now that's pretty funny. Even HPE uses SuperMicro!
Don't know about the chasis itself, but the mobo's are, for sure.
I'll get a screen cap if I'm able.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
Tee hee. Now that's pretty funny. Even HPE uses SuperMicro!
Don't know about the chasis itself, but the mobo's are, for sure.
I'll get a screen cap if I'm able.
Oh okay HP has all kinds of people make mobos.
-
Bringing something up from the "other" thread. Would RAID rebuild times play a factor into choosing a method of supplying this kind of storage amount?
-
@hobbit666 said in How would you build a File server with 170TB of Usable Storage?:
Bringing something up from the "other" thread. Would RAID rebuild times play a factor into choosing a method of supplying this kind of storage amount?
My first thought is YES! DEFINITELY! However, it actually may depend on how you set it up. Systems like Ceph and Gluster don't necessarily need RAID since they can work with different disks in the same node.
