ZFS is Perfectly Safe on Hardware RAID
-
@Grey said in ZFS is Perfectly Safe on Hardware RAID:
What kind of reliability is expected from each?
Totally depends. All RAID controllers are not equal. And all setups of ZFS are not equal. Here are some differences:
- ZFS Software RAID supports RAID 7, nothing else does. So that is a unique feature of using it for RAID. But that only matters if you want RAID 7. Very few people do. So normally this doesn't matter.
- If you are using non-business class hardware RAID, you have a write hole protection advantage with ZFS. But if you are doing this, you didn't care about reliability anyway, so this advantage doesn't matter. No business class hardware RAID is subject to the write hole, so this advantage is purely academic for ZFS.
- Any business class hardware RAID has either battery backed or non-volatile cache options that ZFS cannot do as this requires hardware. So hardware RAID has safety advantages unless ZFS is run without cache, then hardware RAID has cache advantages.
- ZFS has advanced checksumming that is rare or impossible to find in hardware RAID which is useful if the data is becoming corrupt on disk, not something we are normally concerned about but which is a nice feature.
- All business class hardware RAID does blind swap, a pretty massive protection feature in the SMB market.
Remember that there is nothing stopping your hardware RAID from implementing ZFS itself as its RAID option. So it is literally impossible for software ZFS RAID to offer something that hardware RAID "can't" do. All we ever compare are specific implementations. So ZFS software RAID set up as X compared to Adapter controller Z setup as Y, for example.
A cool academic exercise would be to build a ZFS based hardware RAID card. That would be pretty cool. Then you could add non-volatile RAM to ZFS itself.
-
That list makes hardware RAID sound safer than ZFS, which is probably not quite true. But is the case is that the average implementation of hardware RAID is quite a bit safer than the average implementation of ZFS software RAID. Hardware RAID "handles everything for you" protecting you from most bad decisions. ZFS leaves all the nitty gritty details up to you which makes it super, duper easy to mess something up and leave yourself vulnerable. This is exacerbated by the Cult of ZFS problem and loads of misinformation swirling about its use. So the average person using ZFS is not even remotely prepared for what is needed to use it safely.
Some problems that we see people have when using ZFS without fully understanding storage:
- Believing that ZFS doesn't use RAID (this is extremely common.)
- Believing that RAIDZ is magic, rather than a brand name, and that normal RAID concerns do not apply. So we often see people implement RAID 5 in reckless, insane situations using "it's RAIDZ" as an excuse as if RAIDZ isn't just RAID 5 - literally just a brand name for RAID 5.
- Treating common features common to all RAID systems as "unique" and believing that ZFS has feature after feature of protection that makes the need to protect against storage failure unnecessary.
- Not understanding hot swap and blind swap differences and creating systems that they do not know how to address should a drive fail.
- Believing that ZFS being magic is not at risk from power loss and failing to protect caches from power issues - something that they are not normally used to dealing with as hardware RAID does this for you.
- Not understanding the CPU and memory needs of ZFS, especially with features like dedupe and RAIDZ3.
- Ignoring common RAID knowledge and thinking that using ZFS means not using mirroring technologies.
-
This post is deleted! -
@saniplastic said in ZFS is Perfectly Safe on Hardware RAID:
hello to all
in my freenas server, each disk is on Raid0 Hardware Raid P410i and separate raid0 disk present to zfs and use freenas raid and disable S.M.A.R.T services on freenas and let RAID card handles the SMART monitoring, handles the alerting.
I khow i can use hardware raid and present single logical unit to freenas. but our server is under load. should i use hardware raid or continue with exist config ?
what is your idea about this?Quoting here so that you will see this if you come back to this thread, but going to split out the question to its own thread as it is a unique question.
-
Recently I am noticing a change in you when it comes to ZFS.
Also Poutine is overrated.
-
@Emad-R said in ZFS is Perfectly Safe on Hardware RAID:
Also Poutine is overrated.
OMG, that's it, I can't take anyone seriously on this forum! Sacré bleu!
:face_with_stuck-out_tongue_winking_eye:
-
@Emad-R said in ZFS is Perfectly Safe on Hardware RAID:
Also Poutine is overrated.
For sure, it's never been something that I liked.
-
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
Even more of a reason to use ZFS in a VM if ZFS is necessary, or HBA in hardware if having to go down that road.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
I've spoken on why this is a BS sales pitch many times. Yes, it's a slightly different team, but still one "selling" something. And remember, these are developers, not operations people, so what the developers of something feel is really worth listening to, but then we have to put on our IT hats and evaluate it carefully.... and in doing so it is really obvious that ZFS on its own is fine, but ZFS on hardware RAID is also fine and that their points aren't a reason to always choose one or the other. It is with their concerns 100% evaluated that we come to the conclusions.
If their concerns were correct, then their theory is that all production systems in the world must not be good enough to use. It's an absurd point and makes their point invalid.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
Even more of a reason to use ZFS in a VM if ZFS is necessary, or HBA in hardware if having to go down that road.
Because the maintainers aren't understanding operations and we can't trust them? Hubris is one of the most dangerous things in production.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own
This is where we have to be careful. You see, if this is true, then they either agree with my point, or else believe that ALL other filesystems, ALL operating systems that don't support ZFS, and ALL hardware RAID is not safe enough to use in production.
But generally what we see is the ZFS team stating something reasonable... like that ZFS is slightly safer than other options... and then it being misinterpreted that because it is unmeasurably safer, that that must be the sole determining factor in all decisions, which is not even close to being true. And to be safer, it has to ignore certain operational realities that often make ZFS far less safe in the real world.
For example the "which is generally safer" reason, hardware RAID often trumps ZFS. ZFS might have a technical advantage, but hardware RAID often has a human one that is vastly more significant.
But, I never really see the ZFS people saying that. It's just how people read into it. But their own statements normally don't suggest that. They simply point out that ZFS on hardware RAID is maybe a tiny bit safer than say NTFS or EXT4 on it. And I think most of us would agree with that. What most people would also agree with is that all of those combinations are so safe that under all normal conditions, it doesn't matter at all.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
-
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
BTW, we use linux software RAID (md) and on the LSI cards we have in several machines you just don't define any arrays and the drives show up as expected and are handled by the OS directly with full access to the drives. OS automatically loads the mpt2sas driver which is included in the kernel.
So it's a RAID card but the RAID logic lies dormant and the OS handles the drives directly.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
How else would you connect 24 hard drives, or connedlct a DAS box, in that manner, without an HBA?
Why would you pay good money for a raid card just to not use it?
Ideally you would purchase a server vendor supported card.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
How else would you connect 24 hard drives, or connedlct a DAS box, in that manner, without an HBA?
Why would you pay good money for a raid card just to not use it?
Ideally you would purchase a server vendor supported card.
Some servers comes with integrated RAID. At least on newer machines most of them are able to passthrough disk directly to the OS, just as I mentioned above on our LSI cards. So the RAID card works exactly like a dumb HBA - if you want.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
The problem is, many RAID cards won't turn off the RAID logic and encapsulation.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
In that scenario, where you've already chosen to use the software RAID and aren't in a position of having hardware that makes it problematic, yes.
