Revisiting ZFS and FreeNAS in 2019
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
"XXX is special" - Don't know why you think someone would say that. So let's just skip over that.
I know, right? But if you go on places like the Spiceworks community or the FreeNAS forums, people would state that stuff constantly about caching, snapshots, etc. Basic stuff that we had had for a long time. The biggest thing seemed to be anything that was held in the LVM layer was unknown to many people and when ZFS offered it, they felt like they were discovering something new. So there were countless weird conversations of people acting like they had just discovered the snapshot, lol.
Here is an example from as late as 2016: "The ZFS snapshots and ZFS send/receive features make this a very enterprise and mature file system. I find it hard not to choose ZFS when given an option."
Basically he thought that because it could do snaps, that that alone made it mature and he was picking it based on that. The send/receive is unique, and neat, but ultimately not a big deal. Handy, but that's about it. If he knew that other filesystems were being snapped many years earlier (making them more "mature") would he then have ruled out ZFS based on the same logic?
-
Here is another one, this time back to 2011: "I am testing ZFS is FreeNAS. ...My main reason for wanting ZFS is the awesome snapshot capablity."
Already at that point you can tell from the responses that having things like snaps listed as key differentiation features were a big thing that had already been discussed a bit. It's an odd thing, but people were really pushing it.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
If you have a HDD with bad cache ram on it, ZFS (and, to a lesser extent, btrfs) is the only thing that's going to warn you that something is wrong with that device (and it's not like this is uncommon - I've discovered two drives in the last year that were writing corrupt data). This happens.
This is increasingly uncommon today as this is mostly a risk with Winchester (spinning) drives. As the world moves to more reliable SSD, this has dropped off. It doesn't go away, but most corruption is believed to come from the network not the storage media, which nothing protects against (yet). But for the media level issues, newer firmware and tech, better components, and the biggest thing - moving from magnetic media to solid state all reduce this risk a bit. It remains, of course, but most companies go years or decades without experiencing it and probably the average person will never see it in a life time. It happens, but so often to things of no consequence (like empty areas of a disk) that it doesn't matter. When it does strike, typically it is a file so trivial to replace that it's not considered consequential. A real risk and certain scenarios definitely require watching for it carefully, but by and large between how rarely it impacts people and that most shops have moved to safer tech it's all but vanished as a talking point. In the 2000s, it was a bit concern.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
The 'expansion' problem comes about because if you add another spindle to that ZRAID2, suddenly block 1 would be on spindle 1 at sector 10, spindle 3 at sector 500 and spindle 6 at sector 1000 (or whatever). So the location of the data would be wrong, and everything falls apart.
ZFS doesn't have that problem actually. You can grow a ZFS RAIDZ pool: https://www.tech-recipes.com/rx/1416/zfs-grow-or-add-more-disk-space-to-pool-or-filesystem/
It is the zpool add command. You can do it live, while the system is running. Most RAID implementations support this in some form.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I'm not going to get into SSDs, because I've had terrible results with a couple that wildly skew the stats - but they've burned me so badly....
I would definitely go back and reevaluate the foundation of this. SSDs are, in general, much safer than Winchesters. There are exceptions, but they are very rare and getting rarer every day. Modern SSDs are insanely safe, and have been for a decade. Winchesters are safe, but not really any safer than they were fifteen years ago. SSDs have lept forward, while Winchesters basically have been stagnant. Chances are you had something else happen that caused the SSDs to seem to be the thing that failed or were just super unlucky with your SSDs. But the math and real world usage puts SSDs leaps and bounds ahead of HDDs, to the point that for high end systems people are often skipping RAID (not that I recommend that) because the reliability will often meet or exceed HDDs in a RAID setup.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
... Hardware RAID is useless - Yep, these days it's a terrible idea to use hardware RAID, unless you're prepared to keep identical hardware cards with identical firmware versions in stock, right next to every machine with hardware RAID. If you haven't been bitten by a hardware RAID card being unable to import a RAID when the previous card died, you're a lucky lucky person.
Back to the basics. This is not true at all. Hardware RAID remains almost totally ubiquitous in commodity hardware, which has mostly overtaken everything else today, because Windows still has no enterprise alternative (and that drags Hyper-V along with it), and VMware has no alternative at all. Basically all enterprise servers ship with hardware RAID included, so doing without it generally means disabling or removing it, and it brings real benefits.
We've covered this a lot in the community. Few people promote software RAID as much as I do, but the benefits of hardware RAID are real and in no way is hardware RAID dead, dying, or useless. It remains the only choice for a large percentage of the world (those that need Windows, Hyper-V, VMware, etc.), and the standard choice for many more, and the only common way to get blind swap, which most shops need.
Hardware RAID might not be as fast as software RAID, but it is so fast that no one cares, both are ridiculously fast to the point that it is rarely relevant. It's like "which is faster, Ferrari or Porsche" and the real answer is "who cares, you can't go that fast between here and the grocery store." Hardware RAID generally offers blind swap, cache offload, CPU offload, and OS agnostic portability which are all real benefits. Speed always sounds important, but ZFS is already not a super fast file system, so in cases where we are talking hardware RAID vs. RAIDZ it's probably a wash anyway.
I've done hundreds or thousands of hardware card swaps (I've done the world's largest published study on RAID reliability with 160,000 array years, and 80,000 controller years) and the number of failed imports was zero. Of course this assumes enterprise hardware, proper firmware management, and so forth. But that's stuff any IT shop should be able to do with ease. Imports can fail, but are so rare in a properly maintained environment to be statistically able to be ignored.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
You're so hung up about HOW BLOCKS ARE STORED ON THE DISK that you've ignored everything else I've said.
So just to recap where we are at this point, because this line was intended to make us forget....
The point was that several of us had warned that there are many myths around ZFS and FreeNAS and that essentially all use of them is based on misinformation and misunderstanding. You picked up the challenge and decided to take the list of concerns and attempt to refute many or most of them to show why you felt ZFS (and FreeNAS) were good choices.
In doing so, you essentially repeating the myths either as they were or with slight variations, and based most of your "why we were wrong" on the misunderstanding of what ZFS fundamentally was and how it fundamentally worked. Being caught up on how the blocks were stored on disk is what we were addressing because you felt that how they were stored on the disk created all kinds of unique, special situations that could not exist anywhere else which would, obviously, make justifying ZFS easy.
Digging into ZFS and understanding how it works therefore was the opposite of ignoring what you said, and was what listening to it was - because essentially every point you made was based on ZFS storing blocks on disk in a special way that was not RAID, not parity, not standard, not available to other systems for the most part. This belief also led to the underlying misunderstanding of why hardware RAID works better than you had realized, that other systems can do useful scrubbing, and so forth.
Now that we are where we are and have established that the basis for your belief that ZFS had special use cases for you, and similarly we've looking into other myths like the hardware RAID and scrubbing, do you understand that we did this because we were paying attention to your holistic decision around ZFS because by showing that ZFS wasn't the myth that you thought that it was, that it shows that every one of the "standard myths and concerns" is valid?
I would suggest starting over. As the basis for the discussion no longer exists, you should look at your decision to use ZFS again. Your understanding of ZFS was off in ways that led you not only to feel ZFS was better than it really is, but also to believe that RAID was not as good as it really is. So you were led astray in multiple ways. You had stated that RAID couldn't do many things that it can and does do, that hardware RAID couldn't have standard features that it generally does have. With that new information, that most of the things you liked about ZFS are either equally available without ZFS or don't exist anywhere, I would recommend starting the whole decision process fresh. The underlying foundations of your entire approach to local storage have shifted, not just one small piece. That piece was the basis for seemingly everything that you were thinking about how storage worked.
So please, don't see this and think anything you've said has been ignored. Everything you've said has been paid very close attention to. All of it. Anything that you feel I've not addressed, I've done so because the amount of proof and course correction that you needed felt overwhelming any more than was absolutely necessary. But I've carefully read everything and ignored nothing, and believe if you read the thread and source materials again, you will see that all of your reasons for feeling FreeNAS and ZFS made sense here were thoroughly addressed.
-
@scottalanmiller Dude, dumping a huge amount of stuff in 15 different posts is TOTALLY UNCOOL and is really unfriendly. Please don't do that. I now have to quote multiple things, scroll backwards and forwards, and generally waste even more of my time.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
As the world moves to more reliable SSD, this has dropped off. It doesn't go away, but most corruption is believed to come from the network not the storage media, which nothing protects against (yet).
ZFS does. And, my experience is that I've had 2 SSDs silently fail and return corrupt data (of out 30 or so) and 2 spinning disks fail (out of several hundred). That's why I said it's a statistical anomaly.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
and probably the average person will never see it in a life time.
They will probably never NOTICE it, unless they're running btrfs or ZFS, which has inherent checksum validation.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
ZFS doesn't have that problem actually. You can grow a ZFS RAIDZ pool
No, you can't. In fact, the announcement of the POTENTIAL of the ability to do drew excitement from all the storage nerds. What you linked to is appending another zdev to a zpool. You can't expand a raidz.
https://www.reddit.com/r/homelab/comments/83wo88/any_news_on_zfs_raidz_expansion/
This is what frustrates me here - I know this stuff IN DEPTH (yes, I was wrong about parity vs copies - I dug up some of my old course notes and it said copies there - THAT was the source of my error), and you're trying to claim that you know this better than me, when you obviously don't. It's massively frustrating.
Re 'WTF are you doing with Hardware RAID, it's dead':
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:This is not true at all. Hardware RAID remains almost totally ubiquitous in commodity hardware
Funnily enough, almost all 'hardware RAID' cards are actually software RAID with a wrapper around them. And if they're not, they're going to be slower than your CPU anyway, so, back to my original point - why ADD slowness and INCREASE failure types? Pretty much the only Hardware RAID cards these days are the PERC-esque cards. I'm not going to go into EXPLICIT details, but if your RAID card doesn't have a heatsink on it, it's almost certainly a software raid implementation.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
Few people promote software RAID as much as I do, but the benefits of hardware RAID are real and in no way is hardware RAID dead, dying, or useless.
Hardware RAID is slower, and more finnicky, and provides less visibility of individual platters than software RAID. For example, can a standard hardware RAID card provide access to SMART data of each drive? (No).
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
I've done hundreds or thousands of hardware card swaps and the number of failed imports was zero.
As I said originally - the only way that is true is if you had identical cards with identical firmware versions on standby. That's perfectly fine for an EMC sized company, but it's not fine for anyone with only 200 or 300 spindles. I've had multiple P410i's refuse to import a RAID that was generated with a different version of firmware. This is not something uncommon, this is something that happens ALL THE TIME.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
. You picked up the challenge and decided to take the list of concerns and attempt to refute many or most of them to show why you felt ZFS (and FreeNAS) were good choices.
FreeNAS is just a wrapper for ZFS, with all the tools everyone needs built in.
ZFS is, unfortunately for those that are trying to make a living in the HARDWARE RAID space, a significant nail in their coffin. I brought up a whole bunch of things where your statements were wrong, or misleading, or in some cases totally irrelevant.
In retrospect, from your comments, it seems that that you make a living from hardware RAID, so it's somewhat unsurprising that you're trying to spread a pile of FUD on ZFS. Comments like 'people say it's magic' are just casting dispersion on it, purely to disparage it without that meaning anything.
And ZFS is so portable that I can literally pull the drives from a FreeNAS box, plug them into an Ubuntu machine, run 'zfs import' and all my data is there. Can you do that when you move your HDDs from a HP to a Dell to an IBM?
There. See how you can reply in ONE comment, rather than 30? It makes it much more constructive.
-
@scottalanmiller Well, in this case because I'm trying to get across a REALLY COMPLEX thing, that you're having difficulty with, please respond IN ONE MESSAGE. That'll be easier to keep track of, OK? Otherwise there's no coherency.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Funnily enough, almost all 'hardware RAID' cards are actually software RAID with a wrapper around them. And if they're not, they're going to be slower than your CPU anyway, so, back to my original point - why ADD slowness and INCREASE failure types? Pretty much the only Hardware RAID cards these days are the PERC-esque cards. I'm not going to go into EXPLICIT details, but if your RAID card doesn't have a heatsink on it, it's almost certainly a software raid implementation.
That's called FakeRAID and those are not considered hardware RAID cards at all. Because there is no RAID in the hardware.
https://mangolassi.it/topic/6068/what-is-fakeraid
Yes, they are common. But they are considered non-business class and again, speed isn't important. They are risky in most cases because of the attempt to mislead people and no need to actually do what they say. It's fundamentally bad. Hardware RAID is fine, software RAID is fine, but faking people out to trick them into buying fake hardware, not fine.
It's a topic we cover pretty thoroughly around here. We have lots of threads digging into individual cards as well, as there are some that are just REALLY awful hardware RAID, and some do an amazing job of hiding what is going on under the hood.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
This is what frustrates me here - I know this stuff IN DEPTH (yes, I was wrong about parity vs copies - I dug up some of my old course notes and it said copies there - THAT was the source of my error), and you're trying to claim that you know this better than me, when you obviously don't. It's massively frustrating.
I assume your making a funny.
You do realize this sounds like an eighth grader trying to claim to be an expert on something, and everyone realizes that they have no clue what they are talking about and aren't even putting together a coherent argument, then claims that their class notes are wrong! Trust me, domain experts don't go to class notes to "remember" how underpinnings of basic concepts work. They actually learn and understand them, and rarely from a class.
Something you could get wrong that wouldn't completely undermine understanding the problem domain might be like forgetting exactly which RAID level is triple parity. But not knowing that ZFS has RAID, that RAID is mirroring and parity, that ZFS is mirroring and parity, that you believed all kinds of myths based on the belief that ZFS wasn't RAID... there's no plausible way to claim you are an expert here. It's like claiming you are a race care driver and failing a basic conversation where you weren't sure if modern cars used steering wheels, and being confused and thinking that was made the Tesla special was the lack of steering wheel, rather than the switch to electric power. Then claiming that you had to "check your notes about cars" and "oh yeah, they still use steering wheels" all while saying it is "so frustrating when people don't realize your the expert on cars" when clearly, you aren't old enough to drive yet.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
As I said originally - the only way that is true is if you had identical cards with identical firmware versions on standby. That's perfectly fine for an EMC sized company, but it's not fine for anyone with only 200 or 300 spindles. I've had multiple P410i's refuse to import a RAID that was generated with a different version of firmware. This is not something uncommon, this is something that happens ALL THE TIME.
I do a lot of work in the SMB space and even companies with just one server have no issue with this. This isn't a problem in the real world. Unless you are running ancient hardware that is out of support and the supply chains no longer carry the old hardware. Seriously, even several generations old gear, getting a matching or even just supported hardware RAID card is totally trivial from every enterprise vendor.
You seem to be running into problems that are very far outside of the norm. No one with HPE, Dell, Lenovo, IBM, or similar is running into these issues. No one with third party enterprise cards is, either. All it takes is a server support contract, or just a good vendor like PCM and you are covered.
-
OK, so you're obviously just trolling now. I've asked you nicely 4 times to stop spamming, and you're going out of your way to do it.
So fine, you win. I will not talk about ZFS here. You can say whatever you want, and I won't correct you. Enjoy.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Hardware RAID is slower, and more finnicky, and provides less visibility of individual platters than software RAID. For example, can a standard hardware RAID card provide access to SMART data of each drive? (No).
Again, stating that it is slower, when we've said that slower doesn't matter as they are "so close" in speed, makes it sound like you are trying to make us forget what the real issues are.
And yes, you most certainly can see the SMART data on enterprise controllers. Here is an example:
http://john.cuppi.net/view-smartctl-data-of-hard-disks-for-dell-perc-and-lsi-megaraid/
No business class hardware RAID doesn't provide this data. If you've seen the problems you are reporting, it means either you've been using FakeRAID or have non-business class controllers. In which case, yes, you'd expect all kinds of problems like you mention. Same would be true for using some kid's experimental software RAID. Anything bad and broken is bad and broken. But just because you've not been using proper hardware RAID doesn't mean it's bad or doesn't do what needs to be done.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
OK, so you're obviously just trolling now. I've asked you nicely 4 times to stop spamming, and you're going out of your way to do it.
And I explained before you asked why a wall of text is unprofessional and impolite and you proved it to be true. So just because you demand something ridiculous doesn't imply that I have to act like a jerk and do it.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
So fine, you win. I will not talk about ZFS here. You can say whatever you want, and I won't correct you. Enjoy.
Again, you're kidding, right? Do you remember the discussion that we just had where you weren't familiar with any tenant of how RAID works? You've been belligerent and braggadocios about how you are the subject matter expert, but have demonstrated a lack of basic knowledge and familiarity that would suggest that you'd fail an A+ exam. This stuff is ridiculously basic and you are acting like not knowing the fundamentals doesn't matter because somehow you can be an expert without even a passing knowledge of what RAID is? This makes no sense.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
FreeNAS is just a wrapper for ZFS, with all the tools everyone needs built in.
This is a bizarre statement. FreeNAS is a complete software appliance that includes a full OS, loads of utilities, lots of bloat, unneeded parts that can (and have) led to dataloss, and only really brings value to people who don't understand ZFS and the OS on their own. Knowing enough ZFS to be safe with it would make FreeNAS' interface be "in the way".
FreeNAS is way, way more than a wrapper on ZFS. And a wrapper on ZFS would be a bad idea - as has been covered.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
ZFS is, unfortunately for those that are trying to make a living in the HARDWARE RAID space, a significant nail in their coffin. I brought up a whole bunch of things where your statements were wrong, or misleading, or in some cases totally irrelevant.
Um, ZFS has been around since 2005 and has had no effect on the hardware space. It has no significant affect. I feel like you just read about ZFS and have never seen how servers are really used and purchased. Tell VMware that they no longer have a working system because they use hardware RAID and don't support ZFS. They will laugh at you pretty heavily.
You claimed a bunch of my statements were wrong, but you've provided no support for any of those claims. You use brow beating, but don't actually state why your wild claims are true. And all of the "true bits" in your claims seem to be things pulled out of my own articles - like stating that sofware RAID is faster, something I have championed more than anyone for 19 years now. But, I also point out that being faster doesn't matter 99%+ of the time, so bringing it up isn't refuting anything, it just highlights that what is important is being missed (or at least ignored.)
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
In retrospect, from your comments, it seems that that you make a living from hardware RAID, so it's somewhat unsurprising that you're trying to spread a pile of FUD on ZFS. Comments like 'people say it's magic' are just casting dispersion on it, purely to disparage it without that meaning anything.
Oh yeah, THAT'S the take away here. And this in a thread where you personally claimed and continue to claim even after we proved you didn't understand even what is was that ZFS had magic properties that defined storage laws and did things that were impossible and that that was why it was so much better than RAID. You proved the point, in this thread, that you claim I made up.
You also showed how, even when presented with a list of the myths, you were so ready to defend those myths and ready to attack anyone that didn't support your pet project that you didn't even bother to learn about the myths and figure out what a plausible claim would look like and just demonstrated how people using FreeNAS and ZFS do so out of emotion and don't understand what they are doing.
We ended the first part of the discussion with the discovery that you literally don't know what ZFS is. I think you are trying to claim that even though you were completely unclear as to what it fundamentally is that you are still the expert on it.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I'm trying to get across a REALLY COMPLEX thing
It might seem that way to you. But yelling that something is complex doesn't make it so. RAID is actually really simple. I would encourage you to check out this video that helps to explain it: https://mangolassi.it/topic/19511/
If you are new to storage these concepts might seem overwhelming. Storage is tough for everyone at first. But remaining calm, assuming that people who use this all the time (and teach it) might have some good insight for you, and starting from the basics and building up will make it actually clear that the concepts are very simple. RAID is easy when you learn how it works. You've experience the frustration of trying to skip understanding it and just memorizing words and facts and numbers; it might help you pass a class test quickly but if you get anything wrong, you won't know and you can skew off very quickly.
If you take the time and learn what RAID is and how it works, it makes it impossible to have the kinds of challenges you are seeing. If you knew what RAID was and how the math works, there is no way to have someone fool you into thinking that ZFS isn't RAID if you know how ZFS works, for example. And thinking that copies could replace parity and still scale like parity - obviously if you think about it that can't be how it works. Or saying ZRAID8, when the highest level is RAIDZ3.
So start with the basics, once you know what RAID is, it'll make these things easier and you'll see that it really isn't complex. It's just a few basic concepts to master like what parity is, how multiple parity interact, how mirroring and striping work, how you can nest, etc. It all just builds.
