How would you build a File server with 170TB of Usable Storage?
-
@travisdh1 said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
This I would absolutely avoid. The Storinator has no place in a production SMB environment. It's exclusively designed for use as a RAIN node as part of an N+X cluster. It's disposable / ephemeral only. Any attempt to use it in another way, which would include any configuration with RAID, is misusing it by intent and will lead to support issues. It is not hot swap, it is not designed to be reliable. It's not meant to be serviced. All top considerations for a storage node. It's actually, other than capacity, the exact opposite of what you would seek in a storage device.
@scottalanmiller -- How would you build it? That is specifically my question or discussion point... How would you go about building a storage platform for 170TB usable data?
Assuming a single chassis, pick a linux distribution, and setup software raid, LVM, and XFS. My other two preferences would be ZFS on BSD, or hardware RAID.
Now if we start talking performance, backup or failover, it quickly becomes more difficult.
I'm not familiar with ZFS. I will stay far FAR FAAR FAAAAAAARRRRRRRR away from BTRFS. I've seen too many problems with it at my day job, lol.
I'd likely go with Linux + XFS for the file system.
-
LVM Thin Pool and XFS for a file server?
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@travisdh1 said in How would you build a File server with 170TB of Usable Storage?:
Now if we start talking performance, backup or failover, it quickly becomes more difficult.
I'm specifically avoiding those topics for the sake of this post. I'm mainly curious as to what the hardware would look like... but more detailed than "buy a server and put hard drives in it" lol.
Dumb topic them.
-
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Dumb topic them.
:thumbs_up:
-
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
It is relevant because I have no idea what was put into the plan. For all I know, they only have 35TB of data that will only grow to 46TB over the next 5 years. If that's the case, it could change every aspect of the best solution.
If you are saying to just take the 170TB that they just want that amount and tjays that... Then I don't care to put any thought into this thread and will just say buy a server, some DAS, and move on.
-
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
It is relevant because I have no idea what was put into the plan. For all I know, they only have 35TB of data that will only grow to 46TB over the next 5 years. If that's the case, it could change every aspect of the solution.
If you are saying to just take the 170TB that they just want that amount and tjays that... Then I don't care to put any thought into this thread and will just say buy a server, some DAS, and move on.
That is the point of the thread.. what hardware to buy to do that..
The point of the thread is not to analyze some made up needs. The original thread does that.
This thread is simply to say, I have this valid need for 170TB of storage. How do I actually build it with what hardware.
-
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
It is relevant because I have no idea what was put into the plan. For all I know, they only have 35TB of data that will only grow to 46TB over the next 5 years. If that's the case, it could change every aspect of the solution.
If you are saying to just take the 170TB that they just want that amount and tjays that... Then I don't care to put any thought into this thread and will just say buy a server, some DAS, and move on.
That is the point of the thread.. what hardware to buy to do that..
The point of the thread is not to analyze some made up needs. The original thread does that.
This thread is simply to say, I have this valid need for 170TB of storage. How do I actually build it with what hardware.
I see.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
No Setup / Configuration
Not sure what you mean. Storinator doesn't ship any production ready configuration last I knew. ALL of the work has to be done by you. I don't think that they have any RAIN offerings or even support. Not that they would, they are a chassis vendor, not a storage vendor. But it would be a lot of work to do correctly.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
This I would absolutely avoid. The Storinator has no place in a production SMB environment. It's exclusively designed for use as a RAIN node as part of an N+X cluster. It's disposable / ephemeral only. Any attempt to use it in another way, which would include any configuration with RAID, is misusing it by intent and will lead to support issues. It is not hot swap, it is not designed to be reliable. It's not meant to be serviced. All top considerations for a storage node. It's actually, other than capacity, the exact opposite of what you would seek in a storage device.
@scottalanmiller -- How would you build it? That is specifically my question or discussion point... How would you go about building a storage platform for 170TB usable data?
I would use a real server platform, no matter what the product need is, I use production gear. That's a starting point, even when you need it cheap, you still have to support it.
For 170TB, if it were me, I'd have a RAIN cluster. You didn't go into all of your theoretical details, but assuming pretty common needs, Gluster might be a good choice across a handful of chassis.
But you can do a single chassis, if I was doing that I would use RAID 10. Vendors like SuperMicro and Huawei make servers for this purpose (Storinator is a grid node, not a server on its own, it's a component of a larger system only) that act as servers with server class components and things like hot swap on the drives.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
I'm not familiar with ZFS.
ZFS is specifically the filesystem made by SUN for exactly this purpose. Before there was Storinator, before there were BackBlaze Pods, there was the SUN Thumper - an enterprise class, top loading, customer engineering storage server made to hold an insane number of drives and keep them safe and operational in a single chassis. All those others are attempts to replicate it without the enterprise components. The difference is, Thumper was designed to be a server, the Pods were not, but rather nodes.
The ZFS / Thumper team is who worked with me to design the first SAM-SD as well.
ZFS was super important to making Thumper work as it added RAID 7 (aka RAID 5-3) as an option, and all kinds of cache control, and a file system that could scale as big as Thumper could go.
-
@black3dynamite said in How would you build a File server with 170TB of Usable Storage?:
LVM Thin Pool and XFS for a file server?
Yes, XFS is super reliable and pretty fast.
-
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpg -
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgWell sure, if you want to do it the easy way.
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
I would use a real server platform, no matter what the product need is, I use production gear. That's a starting point, even when you need it cheap, you still have to support it.
Not sure why I didn't think of this before. Why not use storage shelves?
If I'm using storage shelves (drive enclosures), then a RAID card + 2 or 3 Shelves to get me up to enough drives to hold my data at RAID10 would be a more viable option. That way I could pretty much whatever I want as the base server and go from there.
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
For 170TB, if it were me, I'd have a RAIN cluster. You didn't go into all of your theoretical details, but assuming pretty common needs, Gluster might be a good choice across a handful of chassis.
I made it theoretical so you could use your imagination a little.
From the get-go, I expected somebody to come back with something like Gluster a little bit quicker. Yes, I realize that adds complexity and networking needs and such. It does solve the problem though. Building 3 nodes with enough storage and using Gluster to make the entire 170GB available for clients to use.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
If I'm using storage shelves (drive enclosures), then a RAID card + 2 or 3 Shelves to get me up to enough drives to hold my data at RAID10 would be a more viable option. That way I could pretty much whatever I want as the base server and go from there.
Because they are a pain at this size. You can get more than enough internal storage for 170TB, no need for shelves. But yes, shelve work.
No need for a RAID card, software RAID will be better for this use case.
-
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgha ha ha.
That's actually a pretty neat setup. I configured it on their site for kicks & grins using 18 x 4TB NVME drives... Yours for only $60k... Gets you up to 36TB in RAID10...
-
If you need more storage density you could also go with another chassis using the NF1 format SSDs from Samsung.
That's a whopping 0.54 PB in 1U.@dafyre I checked our supplier and fully stocked with 36 15TB Samsung PM983 NVMe drives it would be priced at around $115K with 768GB RAM and two Xeon CPUs and a couple of 25Gb NICs.
https://www.supermicro.com/a_images/products/views/1029P-NMR36L_angle.jpg
-
I need to dig in and see what Chasis our new Nimble Storage units are (They're the new ones form HPE). I know those run about $60k each. I need to see how much storage we get in each.
-
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgThis is the best answer for the given question. Anything else is going into assumption land which was specifically not allowed.
