ZFS is Perfectly Safe on Hardware RAID
-
So one final, important point about how this came to be.
The reason that such a horribly misleading and agenda-filled marketing document was able to get such a run and get so much repetition and support might seem hard to understand until we investigate the Jurassic Park effect of the NAS OS world.
This document comes from FreeNAS, a NAS OS vendor. They make a good product, but it is still a NAS OS, fundamentally a bad idea category of product. The customers of FreeNAS are, basically by definition, not experienced storage or systems engineers; if they were they would not be expected to use a product of this nature as systems like FreeBSD, Solaris, Ubuntu and Suse do all of the same things but do them better. The use of FreeNAS increases risk and overhead. The sole selling point of FreeNAS over FreeBSD specifically is that it is "faster to get up and running for someone that doesn't know FreeBSD." It's that simple.
So the community around FreeNAS is, essentially by definition, one of non-storage experts and non-systems experts who all bounce ideas off of each other but none of them are experts in those areas. This results in horrific information sharing as there is little to no oversight and misinformation that would be quickly corrected in a community for storage and systems people rapidly become myths and lore in a community for whom storage is a "magic black box." FreeNAS and ZFS have become more like a cargo cult in nature than as systems for IT professionals.
Cargo cults are those cults in the south Pacific islands where a religion has grown up around military or cargo ship vessels dumping or losing cargo and since anything over the immediate horizon is "heaven" and outside of their scope, the source of the boxes of food or clothing is worshiped as a deity, even though it is just humans on ships losing boxes.
This is a risk in any community, of course, but most communities, like ML or SpiceWorks for example, draw professionals from all levels and encourage peer review. There is an expectation for voices of reason to step in. But in a community that specifically eliminates all experts from practical participation, and especially one that is vendor focused and so carries a very specific promotional agenda in addition to support, there is a totally different effect and there is no reason to expect good data to arise from it.
-
So is ZFS designed to work on JBOD and is it safer than running it on top of a hardware raid platform? What kind of reliability is expected from each?
-
@Grey said in ZFS is Perfectly Safe on Hardware RAID:
So is ZFS designed to work on JBOD and is it safer than running it on top of a hardware raid platform?
ZFS as a filesystem isn't designed to run one place or another. It's just a filesystem. Works like any other.
ZFS as a RAID platform is meant to replace hardware RAID with software RAID. Just like any other software RAID.
So if you want to use ZFS as your RAID system, you don't get hardware RAID. But if you want to use hardware RAID then you use hardware RAID. This decision is independent to if ZFS will or will not be used. ZFS is three different products mashed into one, which is how the "marketing" makes things confusing.
-
@Grey said in ZFS is Perfectly Safe on Hardware RAID:
What kind of reliability is expected from each?
Totally depends. All RAID controllers are not equal. And all setups of ZFS are not equal. Here are some differences:
- ZFS Software RAID supports RAID 7, nothing else does. So that is a unique feature of using it for RAID. But that only matters if you want RAID 7. Very few people do. So normally this doesn't matter.
- If you are using non-business class hardware RAID, you have a write hole protection advantage with ZFS. But if you are doing this, you didn't care about reliability anyway, so this advantage doesn't matter. No business class hardware RAID is subject to the write hole, so this advantage is purely academic for ZFS.
- Any business class hardware RAID has either battery backed or non-volatile cache options that ZFS cannot do as this requires hardware. So hardware RAID has safety advantages unless ZFS is run without cache, then hardware RAID has cache advantages.
- ZFS has advanced checksumming that is rare or impossible to find in hardware RAID which is useful if the data is becoming corrupt on disk, not something we are normally concerned about but which is a nice feature.
- All business class hardware RAID does blind swap, a pretty massive protection feature in the SMB market.
Remember that there is nothing stopping your hardware RAID from implementing ZFS itself as its RAID option. So it is literally impossible for software ZFS RAID to offer something that hardware RAID "can't" do. All we ever compare are specific implementations. So ZFS software RAID set up as X compared to Adapter controller Z setup as Y, for example.
A cool academic exercise would be to build a ZFS based hardware RAID card. That would be pretty cool. Then you could add non-volatile RAM to ZFS itself.
-
That list makes hardware RAID sound safer than ZFS, which is probably not quite true. But is the case is that the average implementation of hardware RAID is quite a bit safer than the average implementation of ZFS software RAID. Hardware RAID "handles everything for you" protecting you from most bad decisions. ZFS leaves all the nitty gritty details up to you which makes it super, duper easy to mess something up and leave yourself vulnerable. This is exacerbated by the Cult of ZFS problem and loads of misinformation swirling about its use. So the average person using ZFS is not even remotely prepared for what is needed to use it safely.
Some problems that we see people have when using ZFS without fully understanding storage:
- Believing that ZFS doesn't use RAID (this is extremely common.)
- Believing that RAIDZ is magic, rather than a brand name, and that normal RAID concerns do not apply. So we often see people implement RAID 5 in reckless, insane situations using "it's RAIDZ" as an excuse as if RAIDZ isn't just RAID 5 - literally just a brand name for RAID 5.
- Treating common features common to all RAID systems as "unique" and believing that ZFS has feature after feature of protection that makes the need to protect against storage failure unnecessary.
- Not understanding hot swap and blind swap differences and creating systems that they do not know how to address should a drive fail.
- Believing that ZFS being magic is not at risk from power loss and failing to protect caches from power issues - something that they are not normally used to dealing with as hardware RAID does this for you.
- Not understanding the CPU and memory needs of ZFS, especially with features like dedupe and RAIDZ3.
- Ignoring common RAID knowledge and thinking that using ZFS means not using mirroring technologies.
-
This post is deleted! -
@saniplastic said in ZFS is Perfectly Safe on Hardware RAID:
hello to all
in my freenas server, each disk is on Raid0 Hardware Raid P410i and separate raid0 disk present to zfs and use freenas raid and disable S.M.A.R.T services on freenas and let RAID card handles the SMART monitoring, handles the alerting.
I khow i can use hardware raid and present single logical unit to freenas. but our server is under load. should i use hardware raid or continue with exist config ?
what is your idea about this?Quoting here so that you will see this if you come back to this thread, but going to split out the question to its own thread as it is a unique question.
-
Recently I am noticing a change in you when it comes to ZFS.
Also Poutine is overrated.
-
@Emad-R said in ZFS is Perfectly Safe on Hardware RAID:
Also Poutine is overrated.
OMG, that's it, I can't take anyone seriously on this forum! Sacré bleu!
:face_with_stuck-out_tongue_winking_eye:
-
@Emad-R said in ZFS is Perfectly Safe on Hardware RAID:
Also Poutine is overrated.
For sure, it's never been something that I liked.
-
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
Even more of a reason to use ZFS in a VM if ZFS is necessary, or HBA in hardware if having to go down that road.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
I've spoken on why this is a BS sales pitch many times. Yes, it's a slightly different team, but still one "selling" something. And remember, these are developers, not operations people, so what the developers of something feel is really worth listening to, but then we have to put on our IT hats and evaluate it carefully.... and in doing so it is really obvious that ZFS on its own is fine, but ZFS on hardware RAID is also fine and that their points aren't a reason to always choose one or the other. It is with their concerns 100% evaluated that we come to the conclusions.
If their concerns were correct, then their theory is that all production systems in the world must not be good enough to use. It's an absurd point and makes their point invalid.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
Even more of a reason to use ZFS in a VM if ZFS is necessary, or HBA in hardware if having to go down that road.
Because the maintainers aren't understanding operations and we can't trust them? Hubris is one of the most dangerous things in production.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own
This is where we have to be careful. You see, if this is true, then they either agree with my point, or else believe that ALL other filesystems, ALL operating systems that don't support ZFS, and ALL hardware RAID is not safe enough to use in production.
But generally what we see is the ZFS team stating something reasonable... like that ZFS is slightly safer than other options... and then it being misinterpreted that because it is unmeasurably safer, that that must be the sole determining factor in all decisions, which is not even close to being true. And to be safer, it has to ignore certain operational realities that often make ZFS far less safe in the real world.
For example the "which is generally safer" reason, hardware RAID often trumps ZFS. ZFS might have a technical advantage, but hardware RAID often has a human one that is vastly more significant.
But, I never really see the ZFS people saying that. It's just how people read into it. But their own statements normally don't suggest that. They simply point out that ZFS on hardware RAID is maybe a tiny bit safer than say NTFS or EXT4 on it. And I think most of us would agree with that. What most people would also agree with is that all of those combinations are so safe that under all normal conditions, it doesn't matter at all.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
-
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
BTW, we use linux software RAID (md) and on the LSI cards we have in several machines you just don't define any arrays and the drives show up as expected and are handled by the OS directly with full access to the drives. OS automatically loads the mpt2sas driver which is included in the kernel.
So it's a RAID card but the RAID logic lies dormant and the OS handles the drives directly.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
How else would you connect 24 hard drives, or connedlct a DAS box, in that manner, without an HBA?
Why would you pay good money for a raid card just to not use it?
Ideally you would purchase a server vendor supported card.
