ZFS is Perfectly Safe on Hardware RAID
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
I've spoken on why this is a BS sales pitch many times. Yes, it's a slightly different team, but still one "selling" something. And remember, these are developers, not operations people, so what the developers of something feel is really worth listening to, but then we have to put on our IT hats and evaluate it carefully.... and in doing so it is really obvious that ZFS on its own is fine, but ZFS on hardware RAID is also fine and that their points aren't a reason to always choose one or the other. It is with their concerns 100% evaluated that we come to the conclusions.
If their concerns were correct, then their theory is that all production systems in the world must not be good enough to use. It's an absurd point and makes their point invalid.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
Hmm, the actual designers and implementers of ZFS disagree with you. Here's their take on hardware RAID controllers and ZFS without a FreeNAS/iX sales pitch: "Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own." They then go on to explain why this is so with a reasonably brief amount of technical detail.
Even more of a reason to use ZFS in a VM if ZFS is necessary, or HBA in hardware if having to go down that road.
Because the maintainers aren't understanding operations and we can't trust them? Hubris is one of the most dangerous things in production.
-
@pjrobar said in ZFS is Perfectly Safe on Hardware RAID:
While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own
This is where we have to be careful. You see, if this is true, then they either agree with my point, or else believe that ALL other filesystems, ALL operating systems that don't support ZFS, and ALL hardware RAID is not safe enough to use in production.
But generally what we see is the ZFS team stating something reasonable... like that ZFS is slightly safer than other options... and then it being misinterpreted that because it is unmeasurably safer, that that must be the sole determining factor in all decisions, which is not even close to being true. And to be safer, it has to ignore certain operational realities that often make ZFS far less safe in the real world.
For example the "which is generally safer" reason, hardware RAID often trumps ZFS. ZFS might have a technical advantage, but hardware RAID often has a human one that is vastly more significant.
But, I never really see the ZFS people saying that. It's just how people read into it. But their own statements normally don't suggest that. They simply point out that ZFS on hardware RAID is maybe a tiny bit safer than say NTFS or EXT4 on it. And I think most of us would agree with that. What most people would also agree with is that all of those combinations are so safe that under all normal conditions, it doesn't matter at all.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
-
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
BTW, we use linux software RAID (md) and on the LSI cards we have in several machines you just don't define any arrays and the drives show up as expected and are handled by the OS directly with full access to the drives. OS automatically loads the mpt2sas driver which is included in the kernel.
So it's a RAID card but the RAID logic lies dormant and the OS handles the drives directly.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
How else would you connect 24 hard drives, or connedlct a DAS box, in that manner, without an HBA?
Why would you pay good money for a raid card just to not use it?
Ideally you would purchase a server vendor supported card.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
How else would you connect 24 hard drives, or connedlct a DAS box, in that manner, without an HBA?
Why would you pay good money for a raid card just to not use it?
Ideally you would purchase a server vendor supported card.
Some servers comes with integrated RAID. At least on newer machines most of them are able to passthrough disk directly to the OS, just as I mentioned above on our LSI cards. So the RAID card works exactly like a dumb HBA - if you want.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
The problem is, many RAID cards won't turn off the RAID logic and encapsulation.
-
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
In that scenario, where you've already chosen to use the software RAID and aren't in a position of having hardware that makes it problematic, yes.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
How else would you connect 24 hard drives, or connedlct a DAS box, in that manner, without an HBA?
Why would you pay good money for a raid card just to not use it?
Ideally you would purchase a server vendor supported card.
Some servers comes with integrated RAID. At least on newer machines most of them are able to passthrough disk directly to the OS, just as I mentioned above on our LSI cards. So the RAID card works exactly like a dumb HBA - if you want.
Sure, but I'm guessing that's not @Obsolesce point. If you don't have an integrated solution - HBA would/should be less expensive than buying a RAID card, plus it's one less piece of complication to get in the way.
additionally - most systems don't have the ability to take 24 drives with an integrated solution.
-
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
The problem is, many RAID cards won't turn off the RAID logic and encapsulation.
Hence the need for RAID 0, which could shield the actual drive from ZFS, I'm guessing.
-
@Dashrender said in ZFS is Perfectly Safe on Hardware RAID:
HBA would/should be less expensive than buying a RAID card, plus it's one less piece of complication to get in the way.
HBAs aren't always less expensive. And the complexity is identical no matter what.... you shift the complexity from hardware to OS or vice versa, but that's it. Same pieces, you are just choosing which place they go in.
But managing hardware RAID is standard and every bench tech knows how to do it, and every Windows admin, and everyone that uses FreeNAS.
ZFS is highly complicated and if you aren't a UNIX admin with ZFS experience, which essentially includes anyone that would consider something like FreeNAS, then ZFS is quite complicated from a human usability perspective. It exposes way more risk to the admin.
THis is why we see humans failing at ZFS RAID functions causing data loss all the time.
-
@Dashrender said in ZFS is Perfectly Safe on Hardware RAID:
additionally - most systems don't have the ability to take 24 drives with an integrated solution.
I'm not aware of ANY that don't, 24 is a tiny number. Most integrated solutions are in the hundreds or thousands.
-
@Dashrender said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
The problem is, many RAID cards won't turn off the RAID logic and encapsulation.
Hence the need for RAID 0, which could shield the actual drive from ZFS, I'm guessing.
No, needing RAID 0 is the thing I'm talking about. The RAID is still there, hence "can't turn it off."
-
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Dashrender said in ZFS is Perfectly Safe on Hardware RAID:
additionally - most systems don't have the ability to take 24 drives with an integrated solution.
I'm not aware of ANY that don't, 24 is a tiny number. Most integrated solutions are in the hundreds or thousands.
People get confused by the number of SAS ports on a card and think that equals the number of drives the card can support.
It does not.
There is something called a SAS expander that allows many more drives on the same port. Some servers can be configured with integrated SAS expanders on the SAS backplane. Then you can run say 24 SAS drives on a RAID/HBA card with 4 SAS ports.
SAS expanders are also used in JBOD enclosures. So you don't need a 48 port RAID controller and the corresponding amount of cables to be able to run 48 bays.
-
P420i which is an old, integrated card, does 60 for example. On the low end, but higher than most people can provide to it since no chassis holds that much, and no standard two chassis do, either.
-
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Dashrender said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Obsolesce said in ZFS is Perfectly Safe on Hardware RAID:
Hardware RAID controllers should not be used with ZFS
This part is obviously just BS. They even have this details list of reasons that if you study it, none of them would result in this conclusion. So this is where the "someone writing the docs through in a little unwarranted opinion" without anything substantiating it. In older docs, this was wrapped with a qualifying statement that includes "if you want ZFS to handle some of these functions." It looks like someone who didn't understand the source document copied what everyone things is the implication, but isn't at all, and missed their own technical details.
I'm not following the context here... I didn't say what you quoted me saying.
In a scenario where one wants to use ZFS as a LVM and software RAID, that is why I said to use HBA on hardware, rather than a RAID card: You would NOT want to use a hardware RAID card and use either passthrough disks or set each disk as a RAID 0. You should use an HBA instead, for ZFS' LVM and software RAID use.
What's the difference here? A RAID card is just like a pure HBA but with RAID logic.
The problem is, many RAID cards won't turn off the RAID logic and encapsulation.
Hence the need for RAID 0, which could shield the actual drive from ZFS, I'm guessing.
No, needing RAID 0 is the thing I'm talking about. The RAID is still there, hence "can't turn it off."
That is what I was saying - the RAID is still there.. it's a RAID 0.. where HBA wouldn't have RAID at all. The disk is just presented as a disk.
-
@Pete-S said in ZFS is Perfectly Safe on Hardware RAID:
@scottalanmiller said in ZFS is Perfectly Safe on Hardware RAID:
@Dashrender said in ZFS is Perfectly Safe on Hardware RAID:
additionally - most systems don't have the ability to take 24 drives with an integrated solution.
I'm not aware of ANY that don't, 24 is a tiny number. Most integrated solutions are in the hundreds or thousands.
People get confused by the number of SAS ports on a card and think that equals the number of drives the card can support.
It does not.
There is something called a SAS expander that allows many more drives on the same port. Some servers can be configured with integrated SAS expanders on the SAS backplane. Then you can run say 24 SAS drives on a RAID/HBA card with 4 SAS ports.
SAS expanders are also used in JBOD enclosures. So you don't need a 48 port RAID controller and the corresponding amount of cables to be able to run 48 bays.
Awww.. thanks for that.
