45TB-60TB / 45-60 mil file volume recommendations?
-
@dashrender Yes - Raid 10 ADM was over cautious / overbuilt. Went from Raid 60 -> Raid 10 ADM after reading some of @scottalanmiller stuff & understanding the tradeoffs of rebuilds for 60 vs 10. After 3 years of not losing a single drive, I'm comfortable moving to simple Raid 10 with a hot swap.
My reason for considering NAS or something like Exablox was because as I understand it some linux filesystems (I'm not familiar with which ones) are safer than NTFS for large volumes with tons of files. This latest hiccup of file corruption is what alarmed me. I bend towards not preferring this as it makes my life more difficult / less flexible have a 40GB SQL Database etc on Windows Server. Only spend about 10-15% of my time on hardware stuff so much more interested in things that just work.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
Hope I'm not out of line to create two topics. I have DL360 G8 / 3 D2600s / 36 3TB HP MB3000FBUCN HDs / Raid 10ADM / Windows Server 2012 that hosts about 30 mil files used by 10-12 people daily. As jobs are processed, these files are moved off the server into cold storage.
I need to increase our storage to 45-60TB. My plan was to just migrate to Raid 10 instead of Raid 10 ADM (I haven't lost a 3TB drive since moving off of raid 60 to Raid 10ADM in 3 years). I'm a little alarmed after a recent file corruption issue described here - https://mangolassi.it/topic/23278/file-corruption-on-copy-issue
I've been a hardcore HP fanboy because I've had very very few problems with their stuff even buying used. Their continued aggressiveness at putting drivers behind paywalls + proprietary HD cost has me bent out of shape pretty hard. I'm looking at two possible solutions.
1 - Switch from Raid 10ADM -> Raid 10 / 30TB -> 45TB. Upgrade DL360 Gen8 to Windows Server 2019 (possible new Windows versions make NTFS volume of this size safer?)
2 - Switch to NAS / Exablox - StorageCraft type solution (I know this used to be shilled pretty hard - is it still top notch set & forget type solution? Will speed be comparable to 36 spindles on Raid 10?)
*I currently have a backup solution using VEEAM + DL380 w 12 drives + tapes for offline storage.
Option 3. New server hardware and no external SAS boxes. 8 x 16TB drives in RAID 10.
I'm not up to speed on HPs server options but 8-12 drives in a 2U server shouldn't be a problem.
-
@pete-s said in 45TB-60TB / 45-60 mil file volume recommendations?:
Option 3. New server hardware and no external SAS boxes. 8 x 16TB drives in RAID 10.
I'm not up to speed on HPs server options but 8-12 drives in a 2U server shouldn't be a problem.
He might need spindles for IOPs reasons.
But then, I'd look at SSD at RAID 5 (Scott has articles that explain the math on this and why it's safe) and you're crush your IOPs needs and likely be able to stay in a single chassis.
-
@pete-s said in 45TB-60TB / 45-60 mil file volume recommendations?:
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
Hope I'm not out of line to create two topics. I have DL360 G8 / 3 D2600s / 36 3TB HP MB3000FBUCN HDs / Raid 10ADM / Windows Server 2012 that hosts about 30 mil files used by 10-12 people daily. As jobs are processed, these files are moved off the server into cold storage.
I need to increase our storage to 45-60TB. My plan was to just migrate to Raid 10 instead of Raid 10 ADM (I haven't lost a 3TB drive since moving off of raid 60 to Raid 10ADM in 3 years). I'm a little alarmed after a recent file corruption issue described here - https://mangolassi.it/topic/23278/file-corruption-on-copy-issue
I've been a hardcore HP fanboy because I've had very very few problems with their stuff even buying used. Their continued aggressiveness at putting drivers behind paywalls + proprietary HD cost has me bent out of shape pretty hard. I'm looking at two possible solutions.
1 - Switch from Raid 10ADM -> Raid 10 / 30TB -> 45TB. Upgrade DL360 Gen8 to Windows Server 2019 (possible new Windows versions make NTFS volume of this size safer?)
2 - Switch to NAS / Exablox - StorageCraft type solution (I know this used to be shilled pretty hard - is it still top notch set & forget type solution? Will speed be comparable to 36 spindles on Raid 10?)
*I currently have a backup solution using VEEAM + DL380 w 12 drives + tapes for offline storage.
Option 3. New server hardware and no external SAS boxes. 8 x 16TB drives in RAID 10.
I'm not up to speed on HPs server options but 8-12 drives in a 2U server shouldn't be a problem.
I'm not sure if there really is a problem with large NTFS volumes and if that would apply to 2019 as well. It just sounds unlikely.
Well, you could split it into two volumes and possibly even make two RAID10 arrays.
-
@dashrender said in 45TB-60TB / 45-60 mil file volume recommendations?:
@pete-s said in 45TB-60TB / 45-60 mil file volume recommendations?:
Option 3. New server hardware and no external SAS boxes. 8 x 16TB drives in RAID 10.
I'm not up to speed on HPs server options but 8-12 drives in a 2U server shouldn't be a problem.
He might need spindles for IOPs reasons.
But then, I'd look at SSD at RAID 5 (Scott has articles that explain the math on this and why it's safe) and you're crush your IOPs needs and likely be able to stay in a single chassis.
SSD cache is an option here too.
People tend to forget that high capacity drives are faster than lower capacity drives, due to their much higher data density.
-
@pete-s said in 45TB-60TB / 45-60 mil file volume recommendations?:
People tend to forget that high capacity drives are faster than lower capacity drives, due to their much higher data density.
Sure, but I'm guessing that only goes so far - I wonder where the trade off is.
-
@dashrender said in 45TB-60TB / 45-60 mil file volume recommendations?:
@pete-s said in 45TB-60TB / 45-60 mil file volume recommendations?:
People tend to forget that high capacity drives are faster than lower capacity drives, due to their much higher data density.
Sure, but I'm guessing that only goes so far - I wonder where the trade off is.
It's not a LOT faster, if that is what you are asking. It's faster but not like as fast as a different type of drive.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
My plan was to just migrate to Raid 10 instead of Raid 10 ADM
Yeah, RAID 10 has a reliability of less than one data loss failure every 80,000 years (and it might be a LOT less than, that's where we gave up the studies.) RAID 10 ADM does essentially nothing to mitigate further issues. It's a waste.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
Raid 10ADM / Windows Server 2012
This is a huge mismatch. All kinds of money being thrown at the "long tail" of RAID 10 ADM. That RAID level exists for things like military equipment in the field and nuclear power stations. No ordinary company should ever let it cross their minds to use (except in insanely rare high read, low write performance scenarios where cost and capacity aren't big issues.)
Then Windows (the least reliable major server OS) and 2012 (an ancient, not super reliable version of Windows) are being run.
So on one hand, you have a system built at the hardware level like it's one of the most critical workloads on earth. And then a totally casual small business "reliability doesn't matter much" OS is installed on it and not kept even remotely updated. The two just don't go together.
Basically, for the hardware it is reckless overspending on reliability that can't really matter to that degree. On the other hand, it's not taking the OS seriously.
In this case, the hardware has been so overbuilt that the software is 99% of the potential risks. Typically it's more like 90/10 not 1/99.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
I'm a little alarmed after a recent file corruption issue described here - https://mangolassi.it/topic/23278/file-corruption-on-copy-issue
There was no file corruption there, you proved that and stated so. There is a network issue, not a file issue. That you saved a file correctly of something that got changed as it went through the network is not the same as file corruption.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
1 - Switch from Raid 10ADM -> Raid 10 / 30TB -> 45TB. Upgrade DL360 Gen8 to Windows Server 2019 (possible new Windows versions make NTFS volume of this size safer?)
This is a perfectly fine solution. Does NTFS get safer when you do this? Sure, I guess so. But by 2012 NTFS was already insanely reliable. I'm not saying that it hasn't continued to improve, only that the improvements are pretty minor at this point. You should never be running old, unmaintained software in production no matter what. So your move to 2019 (or anything modern, no need to be on Windows) is good for so many reasons. But getting NTFS to a safe state isn't one of them. Remember you had zero NTFS issues before, I feel like you are emotionally associating something unrelated with NTFS, but can't figure out why you are projecting in this way.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
2 - Switch to NAS / Exablox - StorageCraft type solution (I know this used to be shilled pretty hard - is it still top notch set & forget type solution? Will speed be comparable to 36 spindles on Raid 10?)
This is fine. But it's odd that you are skipping the obvious, standard approach which is to use your own hardware and just install Linux on it. That's cheaper, faster, and more reliable than getting the same thing done for you on a NAS.
But there is a general rule of thumb... if you don't need Windows, you shouldn't be using it. It's only a rule of thumb, but it's an important one. Windows is costly (to purchase AND to support AND to update), and it is risky (not real risky, but it's definitely not even in the same game with anything else you would reasonably consider.)
Running a Linux here (Ubuntu, Fedora, Suse, etc.) would mean a system that you can keep up to date forever, for free. And without the insane patching and update problems that Windows has had for many years now. And with filesystems even more reliable than NTFS (XFS is the reliability kind, ZFS the durability king.) Not that NTFS isn't awesome, but someone has to be the best.
-
@scottalanmiller said in 45TB-60TB / 45-60 mil file volume recommendations?:
use your own hardware and just install Linux on it
I get this - but as stated I spend 10-15% of my time on Hardware. Based on my personal interactions with linux on personal offtime nothing is officially supported and there are 21 different bash scripts needed to run to enable standard stuff while everyone claims it is consumer friendly.
This machine also runs SQL Server w 40GB database so moving to linux for file store would require different hardware for the Windows SQL Server (SQL for Linux I have no interest in doing)
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
Based on my personal interactions with linux on personal offtime nothing is officially supported
Um, WHAT? Nothing, and I mean, NOTHING is more supported. It's not just the most supported operating system, by far. It's the most support software ever made.
Windows, on the other hand, ACTUALLY doesn't have support options.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
there are 21 different bash scripts needed to run to enable standard stuff while everyone claims it is consumer friendly.
Well, that's just not true. It IS consumer friendly and you don't need to do anything close to what you need to do on Windows to do the same task.
You CAN run a PowerShell task for every little thing on Windows (and, BTW, that's the enterprise way to use Windows), but you don't have to.
Same with Linux.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
This machine also runs SQL Server
Perfect, since Linux is officially supported and broadly considered the best way (and definitely the most cost effective way) to run MS SQL Server.
You do realize MS has released their own Linux distro, right?
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
SQL for Linux I have no interest in doing
So this is purely an emotional thing, not an IT one? Why would you have no interest? It's ridiculously simple. Literally easier than installing on Windows (we run both.)
Your current system has proven to be a problem with maintenance. So while describing a situation where Windows has proven to be unable to be maintained and is no longer safe AND just experienced data corruption.... you are saying you don't want to entertain another option even though it would address every concern you listed, and every concern you mentioned but didn't point out with zero downsides? Why?
Also, you mentioned the Linux based NAS as an option. The ONLY difference is if you install Linux and have an enterprise setup, or someone else does it for you and you have an SMB setup. But when you install yourself you can run MS SQL Server in its most enterprise fashion. But with the NAS, MS SQL Server is not an option. So how did you rule out installing Linux yourself based on your desire for Windows at any cost for SQL Server, but didn't rule out the NAS when that would make it impossible rather than easy? The logic doesn't align. Whatever logic allowed you to consider the NAS, guarantees that Linux you install yourself (which is insanely easy, you can't possibly claim that that is harder than Windows) must be an option.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
but as stated I spend 10-15% of my time on Hardware.
Im' not sure what this means. You mean you do rack and stacks?
Linux is easier to maintain, easier to support. Requires less effort and it is ridiculously easier to get someone to assist you when you need help. The less time you have to fool around and fiddle, the more important Linux becomes compared to a more "time intensive" product like Windows. I'm not knocking Windows, someone has to be easier, someone has to have better support, someone has to take less time... and Linux wins by a landslide here in all three cases for the use cases you are mentioning: file sharing and MS SQL Server.
And updates take, what, 1% as much time? Maybe less, for real, maybe .1% the time of maintaining updates on Windows. If your time matters, then why is Windows on the radar for these use cases?
-
@scottalanmiller May spin up Ubuntu VM & try this again, last time I did I ran into issues & got lost down rabbit hole of trying to figure out what the problem was.
-
@jim9500 said in 45TB-60TB / 45-60 mil file volume recommendations?:
May spin up Ubuntu VM & try this again, last time I did I ran into issues & got lost down rabbit hole of trying to figure out what the problem was.
Great. For starters, be SURE to get Ubuntu 21.04. Do NOT let them convince you to run three versions old, which is some bizarre thing that Ubuntu does. It's crazy. It's nice that they have this "run old stuff with support" option. But for the best support, best performance, best reliability and least admin effort, stick with current, always.
