Revisiting ZFS and FreeNAS in 2019
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
... Hardware RAID is useless - Yep, these days it's a terrible idea to use hardware RAID, unless you're prepared to keep identical hardware cards with identical firmware versions in stock, right next to every machine with hardware RAID. If you haven't been bitten by a hardware RAID card being unable to import a RAID when the previous card died, you're a lucky lucky person.
Back to the basics. This is not true at all. Hardware RAID remains almost totally ubiquitous in commodity hardware, which has mostly overtaken everything else today, because Windows still has no enterprise alternative (and that drags Hyper-V along with it), and VMware has no alternative at all. Basically all enterprise servers ship with hardware RAID included, so doing without it generally means disabling or removing it, and it brings real benefits.
We've covered this a lot in the community. Few people promote software RAID as much as I do, but the benefits of hardware RAID are real and in no way is hardware RAID dead, dying, or useless. It remains the only choice for a large percentage of the world (those that need Windows, Hyper-V, VMware, etc.), and the standard choice for many more, and the only common way to get blind swap, which most shops need.
Hardware RAID might not be as fast as software RAID, but it is so fast that no one cares, both are ridiculously fast to the point that it is rarely relevant. It's like "which is faster, Ferrari or Porsche" and the real answer is "who cares, you can't go that fast between here and the grocery store." Hardware RAID generally offers blind swap, cache offload, CPU offload, and OS agnostic portability which are all real benefits. Speed always sounds important, but ZFS is already not a super fast file system, so in cases where we are talking hardware RAID vs. RAIDZ it's probably a wash anyway.
I've done hundreds or thousands of hardware card swaps (I've done the world's largest published study on RAID reliability with 160,000 array years, and 80,000 controller years) and the number of failed imports was zero. Of course this assumes enterprise hardware, proper firmware management, and so forth. But that's stuff any IT shop should be able to do with ease. Imports can fail, but are so rare in a properly maintained environment to be statistically able to be ignored.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
You're so hung up about HOW BLOCKS ARE STORED ON THE DISK that you've ignored everything else I've said.
So just to recap where we are at this point, because this line was intended to make us forget....
The point was that several of us had warned that there are many myths around ZFS and FreeNAS and that essentially all use of them is based on misinformation and misunderstanding. You picked up the challenge and decided to take the list of concerns and attempt to refute many or most of them to show why you felt ZFS (and FreeNAS) were good choices.
In doing so, you essentially repeating the myths either as they were or with slight variations, and based most of your "why we were wrong" on the misunderstanding of what ZFS fundamentally was and how it fundamentally worked. Being caught up on how the blocks were stored on disk is what we were addressing because you felt that how they were stored on the disk created all kinds of unique, special situations that could not exist anywhere else which would, obviously, make justifying ZFS easy.
Digging into ZFS and understanding how it works therefore was the opposite of ignoring what you said, and was what listening to it was - because essentially every point you made was based on ZFS storing blocks on disk in a special way that was not RAID, not parity, not standard, not available to other systems for the most part. This belief also led to the underlying misunderstanding of why hardware RAID works better than you had realized, that other systems can do useful scrubbing, and so forth.
Now that we are where we are and have established that the basis for your belief that ZFS had special use cases for you, and similarly we've looking into other myths like the hardware RAID and scrubbing, do you understand that we did this because we were paying attention to your holistic decision around ZFS because by showing that ZFS wasn't the myth that you thought that it was, that it shows that every one of the "standard myths and concerns" is valid?
I would suggest starting over. As the basis for the discussion no longer exists, you should look at your decision to use ZFS again. Your understanding of ZFS was off in ways that led you not only to feel ZFS was better than it really is, but also to believe that RAID was not as good as it really is. So you were led astray in multiple ways. You had stated that RAID couldn't do many things that it can and does do, that hardware RAID couldn't have standard features that it generally does have. With that new information, that most of the things you liked about ZFS are either equally available without ZFS or don't exist anywhere, I would recommend starting the whole decision process fresh. The underlying foundations of your entire approach to local storage have shifted, not just one small piece. That piece was the basis for seemingly everything that you were thinking about how storage worked.
So please, don't see this and think anything you've said has been ignored. Everything you've said has been paid very close attention to. All of it. Anything that you feel I've not addressed, I've done so because the amount of proof and course correction that you needed felt overwhelming any more than was absolutely necessary. But I've carefully read everything and ignored nothing, and believe if you read the thread and source materials again, you will see that all of your reasons for feeling FreeNAS and ZFS made sense here were thoroughly addressed.
-
@scottalanmiller Dude, dumping a huge amount of stuff in 15 different posts is TOTALLY UNCOOL and is really unfriendly. Please don't do that. I now have to quote multiple things, scroll backwards and forwards, and generally waste even more of my time.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
As the world moves to more reliable SSD, this has dropped off. It doesn't go away, but most corruption is believed to come from the network not the storage media, which nothing protects against (yet).
ZFS does. And, my experience is that I've had 2 SSDs silently fail and return corrupt data (of out 30 or so) and 2 spinning disks fail (out of several hundred). That's why I said it's a statistical anomaly.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
and probably the average person will never see it in a life time.
They will probably never NOTICE it, unless they're running btrfs or ZFS, which has inherent checksum validation.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
ZFS doesn't have that problem actually. You can grow a ZFS RAIDZ pool
No, you can't. In fact, the announcement of the POTENTIAL of the ability to do drew excitement from all the storage nerds. What you linked to is appending another zdev to a zpool. You can't expand a raidz.
https://www.reddit.com/r/homelab/comments/83wo88/any_news_on_zfs_raidz_expansion/
This is what frustrates me here - I know this stuff IN DEPTH (yes, I was wrong about parity vs copies - I dug up some of my old course notes and it said copies there - THAT was the source of my error), and you're trying to claim that you know this better than me, when you obviously don't. It's massively frustrating.
Re 'WTF are you doing with Hardware RAID, it's dead':
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:This is not true at all. Hardware RAID remains almost totally ubiquitous in commodity hardware
Funnily enough, almost all 'hardware RAID' cards are actually software RAID with a wrapper around them. And if they're not, they're going to be slower than your CPU anyway, so, back to my original point - why ADD slowness and INCREASE failure types? Pretty much the only Hardware RAID cards these days are the PERC-esque cards. I'm not going to go into EXPLICIT details, but if your RAID card doesn't have a heatsink on it, it's almost certainly a software raid implementation.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
Few people promote software RAID as much as I do, but the benefits of hardware RAID are real and in no way is hardware RAID dead, dying, or useless.
Hardware RAID is slower, and more finnicky, and provides less visibility of individual platters than software RAID. For example, can a standard hardware RAID card provide access to SMART data of each drive? (No).
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
I've done hundreds or thousands of hardware card swaps and the number of failed imports was zero.
As I said originally - the only way that is true is if you had identical cards with identical firmware versions on standby. That's perfectly fine for an EMC sized company, but it's not fine for anyone with only 200 or 300 spindles. I've had multiple P410i's refuse to import a RAID that was generated with a different version of firmware. This is not something uncommon, this is something that happens ALL THE TIME.
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
. You picked up the challenge and decided to take the list of concerns and attempt to refute many or most of them to show why you felt ZFS (and FreeNAS) were good choices.
FreeNAS is just a wrapper for ZFS, with all the tools everyone needs built in.
ZFS is, unfortunately for those that are trying to make a living in the HARDWARE RAID space, a significant nail in their coffin. I brought up a whole bunch of things where your statements were wrong, or misleading, or in some cases totally irrelevant.
In retrospect, from your comments, it seems that that you make a living from hardware RAID, so it's somewhat unsurprising that you're trying to spread a pile of FUD on ZFS. Comments like 'people say it's magic' are just casting dispersion on it, purely to disparage it without that meaning anything.
And ZFS is so portable that I can literally pull the drives from a FreeNAS box, plug them into an Ubuntu machine, run 'zfs import' and all my data is there. Can you do that when you move your HDDs from a HP to a Dell to an IBM?
There. See how you can reply in ONE comment, rather than 30? It makes it much more constructive.
-
@scottalanmiller Well, in this case because I'm trying to get across a REALLY COMPLEX thing, that you're having difficulty with, please respond IN ONE MESSAGE. That'll be easier to keep track of, OK? Otherwise there's no coherency.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Funnily enough, almost all 'hardware RAID' cards are actually software RAID with a wrapper around them. And if they're not, they're going to be slower than your CPU anyway, so, back to my original point - why ADD slowness and INCREASE failure types? Pretty much the only Hardware RAID cards these days are the PERC-esque cards. I'm not going to go into EXPLICIT details, but if your RAID card doesn't have a heatsink on it, it's almost certainly a software raid implementation.
That's called FakeRAID and those are not considered hardware RAID cards at all. Because there is no RAID in the hardware.
https://mangolassi.it/topic/6068/what-is-fakeraid
Yes, they are common. But they are considered non-business class and again, speed isn't important. They are risky in most cases because of the attempt to mislead people and no need to actually do what they say. It's fundamentally bad. Hardware RAID is fine, software RAID is fine, but faking people out to trick them into buying fake hardware, not fine.
It's a topic we cover pretty thoroughly around here. We have lots of threads digging into individual cards as well, as there are some that are just REALLY awful hardware RAID, and some do an amazing job of hiding what is going on under the hood.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
This is what frustrates me here - I know this stuff IN DEPTH (yes, I was wrong about parity vs copies - I dug up some of my old course notes and it said copies there - THAT was the source of my error), and you're trying to claim that you know this better than me, when you obviously don't. It's massively frustrating.
I assume your making a funny.
You do realize this sounds like an eighth grader trying to claim to be an expert on something, and everyone realizes that they have no clue what they are talking about and aren't even putting together a coherent argument, then claims that their class notes are wrong! Trust me, domain experts don't go to class notes to "remember" how underpinnings of basic concepts work. They actually learn and understand them, and rarely from a class.
Something you could get wrong that wouldn't completely undermine understanding the problem domain might be like forgetting exactly which RAID level is triple parity. But not knowing that ZFS has RAID, that RAID is mirroring and parity, that ZFS is mirroring and parity, that you believed all kinds of myths based on the belief that ZFS wasn't RAID... there's no plausible way to claim you are an expert here. It's like claiming you are a race care driver and failing a basic conversation where you weren't sure if modern cars used steering wheels, and being confused and thinking that was made the Tesla special was the lack of steering wheel, rather than the switch to electric power. Then claiming that you had to "check your notes about cars" and "oh yeah, they still use steering wheels" all while saying it is "so frustrating when people don't realize your the expert on cars" when clearly, you aren't old enough to drive yet.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
As I said originally - the only way that is true is if you had identical cards with identical firmware versions on standby. That's perfectly fine for an EMC sized company, but it's not fine for anyone with only 200 or 300 spindles. I've had multiple P410i's refuse to import a RAID that was generated with a different version of firmware. This is not something uncommon, this is something that happens ALL THE TIME.
I do a lot of work in the SMB space and even companies with just one server have no issue with this. This isn't a problem in the real world. Unless you are running ancient hardware that is out of support and the supply chains no longer carry the old hardware. Seriously, even several generations old gear, getting a matching or even just supported hardware RAID card is totally trivial from every enterprise vendor.
You seem to be running into problems that are very far outside of the norm. No one with HPE, Dell, Lenovo, IBM, or similar is running into these issues. No one with third party enterprise cards is, either. All it takes is a server support contract, or just a good vendor like PCM and you are covered.
-
OK, so you're obviously just trolling now. I've asked you nicely 4 times to stop spamming, and you're going out of your way to do it.
So fine, you win. I will not talk about ZFS here. You can say whatever you want, and I won't correct you. Enjoy.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Hardware RAID is slower, and more finnicky, and provides less visibility of individual platters than software RAID. For example, can a standard hardware RAID card provide access to SMART data of each drive? (No).
Again, stating that it is slower, when we've said that slower doesn't matter as they are "so close" in speed, makes it sound like you are trying to make us forget what the real issues are.
And yes, you most certainly can see the SMART data on enterprise controllers. Here is an example:
http://john.cuppi.net/view-smartctl-data-of-hard-disks-for-dell-perc-and-lsi-megaraid/
No business class hardware RAID doesn't provide this data. If you've seen the problems you are reporting, it means either you've been using FakeRAID or have non-business class controllers. In which case, yes, you'd expect all kinds of problems like you mention. Same would be true for using some kid's experimental software RAID. Anything bad and broken is bad and broken. But just because you've not been using proper hardware RAID doesn't mean it's bad or doesn't do what needs to be done.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
OK, so you're obviously just trolling now. I've asked you nicely 4 times to stop spamming, and you're going out of your way to do it.
And I explained before you asked why a wall of text is unprofessional and impolite and you proved it to be true. So just because you demand something ridiculous doesn't imply that I have to act like a jerk and do it.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
So fine, you win. I will not talk about ZFS here. You can say whatever you want, and I won't correct you. Enjoy.
Again, you're kidding, right? Do you remember the discussion that we just had where you weren't familiar with any tenant of how RAID works? You've been belligerent and braggadocios about how you are the subject matter expert, but have demonstrated a lack of basic knowledge and familiarity that would suggest that you'd fail an A+ exam. This stuff is ridiculously basic and you are acting like not knowing the fundamentals doesn't matter because somehow you can be an expert without even a passing knowledge of what RAID is? This makes no sense.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
FreeNAS is just a wrapper for ZFS, with all the tools everyone needs built in.
This is a bizarre statement. FreeNAS is a complete software appliance that includes a full OS, loads of utilities, lots of bloat, unneeded parts that can (and have) led to dataloss, and only really brings value to people who don't understand ZFS and the OS on their own. Knowing enough ZFS to be safe with it would make FreeNAS' interface be "in the way".
FreeNAS is way, way more than a wrapper on ZFS. And a wrapper on ZFS would be a bad idea - as has been covered.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
ZFS is, unfortunately for those that are trying to make a living in the HARDWARE RAID space, a significant nail in their coffin. I brought up a whole bunch of things where your statements were wrong, or misleading, or in some cases totally irrelevant.
Um, ZFS has been around since 2005 and has had no effect on the hardware space. It has no significant affect. I feel like you just read about ZFS and have never seen how servers are really used and purchased. Tell VMware that they no longer have a working system because they use hardware RAID and don't support ZFS. They will laugh at you pretty heavily.
You claimed a bunch of my statements were wrong, but you've provided no support for any of those claims. You use brow beating, but don't actually state why your wild claims are true. And all of the "true bits" in your claims seem to be things pulled out of my own articles - like stating that sofware RAID is faster, something I have championed more than anyone for 19 years now. But, I also point out that being faster doesn't matter 99%+ of the time, so bringing it up isn't refuting anything, it just highlights that what is important is being missed (or at least ignored.)
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
In retrospect, from your comments, it seems that that you make a living from hardware RAID, so it's somewhat unsurprising that you're trying to spread a pile of FUD on ZFS. Comments like 'people say it's magic' are just casting dispersion on it, purely to disparage it without that meaning anything.
Oh yeah, THAT'S the take away here. And this in a thread where you personally claimed and continue to claim even after we proved you didn't understand even what is was that ZFS had magic properties that defined storage laws and did things that were impossible and that that was why it was so much better than RAID. You proved the point, in this thread, that you claim I made up.
You also showed how, even when presented with a list of the myths, you were so ready to defend those myths and ready to attack anyone that didn't support your pet project that you didn't even bother to learn about the myths and figure out what a plausible claim would look like and just demonstrated how people using FreeNAS and ZFS do so out of emotion and don't understand what they are doing.
We ended the first part of the discussion with the discovery that you literally don't know what ZFS is. I think you are trying to claim that even though you were completely unclear as to what it fundamentally is that you are still the expert on it.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I'm trying to get across a REALLY COMPLEX thing
It might seem that way to you. But yelling that something is complex doesn't make it so. RAID is actually really simple. I would encourage you to check out this video that helps to explain it: https://mangolassi.it/topic/19511/
If you are new to storage these concepts might seem overwhelming. Storage is tough for everyone at first. But remaining calm, assuming that people who use this all the time (and teach it) might have some good insight for you, and starting from the basics and building up will make it actually clear that the concepts are very simple. RAID is easy when you learn how it works. You've experience the frustration of trying to skip understanding it and just memorizing words and facts and numbers; it might help you pass a class test quickly but if you get anything wrong, you won't know and you can skew off very quickly.
If you take the time and learn what RAID is and how it works, it makes it impossible to have the kinds of challenges you are seeing. If you knew what RAID was and how the math works, there is no way to have someone fool you into thinking that ZFS isn't RAID if you know how ZFS works, for example. And thinking that copies could replace parity and still scale like parity - obviously if you think about it that can't be how it works. Or saying ZRAID8, when the highest level is RAIDZ3.
So start with the basics, once you know what RAID is, it'll make these things easier and you'll see that it really isn't complex. It's just a few basic concepts to master like what parity is, how multiple parity interact, how mirroring and striping work, how you can nest, etc. It all just builds.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
And ZFS is so portable that I can literally pull the drives from a FreeNAS box, plug them into an Ubuntu machine, run 'zfs import' and all my data is there. Can you do that when you move your HDDs from a HP to a Dell to an IBM?
If I'm using any enterprise software RAID, or any matching hardware RAID system... absolutely, yes I can. But obviously wouldn't want to even with ZFS.
But let's not forget what a completely dumb idea this would be and that this is not an operational process. I realize you are desperately trying to figure out something that makes ZFS special. This definitely isn't it.
Pulling a bunch of drives from one system and moving them to another isn't really how businesses work. It sounds neat in a lab and in a class room a professor whose never really worked in IT might envision people would do this all of the time. But in the real world, this is just crazy stuff. Neat tech, sure, but everyone has it and no one uses it. The only difference is that in the ZFS / FreeNAS world people talk about this stuff as a "feature", and in the rest of the world we know that actually doing this is foolish and isn't something to talk about.
-
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:
But good news, Google now ranks us the very top hit for ZRAID8 Literally, since this is the first place it has ever been mentioned.
Hahahhaha that just made my month!
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
This is where the zraid stuff scales. You can have a ZRAID8 for example (requiring 9 or more disks), where there are 8 copies of the data. Literal copies. The data is replicated onto those disks. There are no parity calculations, as there are with RAID5 and 6.
So, as long as you can get the idea of parity out of your mind, suddenly the zraid stuff makes sense.This is where it was really off of the rails. All of the details in this explanation of a system that doesn't exist. Tons of "and because of this... this other weird stuff". Raving about this "literal copies" stuff, and some eight way system, the new ZRAID name, something that requires nine disks... so many false features. This wasn't casually getting mirroring and parity backwards, an entire new product world was built in his head around this fantasy he had concocted. There must be six made up things just in this one statement that all are required for the other to exist.
- ZRAID (made up term)
- ZRAID8 (ignore the ZRAID bit, there is no RAID with the number 8 in it.)
- 8 copies of data. No RAID level does this except RAID 1, and that's just a mirror.
- 9 drives requires. Why nine? Who knows, but no RAID level has ever required that.
- Copies (the wrong term for mirror but whatever) instead of parity.
- Special scaling based on this made up technology.
Six discrete fantasy products, features, or requirements in one statement. There is, in fact, not a single factual thing in this description, not even buried in the subtext.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
ZFS works on copies. So, when you write 1, 2, 3 and 4 to a zpool, you get something like this:
Disk 1: x 1 x 3 x
Disk 2: 1 x 2 x 3
Disk 3: x 1 2 x 4
Disk 4: 4 x 2 x 3
Disk 5: x x x x 4
Copies. Of. The. Data.And this, so much detail about made up products. Where did this all come from? It's like a whole novel about a fake storage technology had been written somewhere.
You know, I once was in a college class and one of the kids didn't know the difference between fiction and non-fiction and in a serious article on ethics they used the "Laws of Robots" from iRobot. This is like that, somewhere there must be some source for so much details, but what it refers to, we will never know.
-
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
As a bit of a rebuttal, you're kinda getting confused and claming FreeNAS is a bunch of things it isn't. In fact, in that 'common myths' page is ... I dunno, random made up stuff?
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
So, as a Solaris administrator from way back, let's go through a couple of the misapprehensions about ZFS in that document you linked!
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I am your guest ZFS expert! Ask me anything
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Truly, honestly, that's not how ZFS works. I cross my heart.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Dude, I've been doing this for almost 30 years
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I got my Windows cert, and CCIE, and a bunch of other things before moving into DevOps. So, please - trust me when I say I know what I'm talking about.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Honestly, this is where you are 100% wrong, and you refuse to listen to me.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
But telling me I'm wrong isn't going to get you anywhere, because I know what I'm talking about here. This is my field of expertise.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
Now, if you can take a breath, admit that you've learned something new about ZFS
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I've been using ZFS for 15 years now, and I'm extremely confident in my knowledge.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I'm a sysadmin. I've done courses.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
I now have to quote multiple things, scroll backwards and forwards, and generally waste even more of my time.
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
This is what frustrates me here - I know this stuff IN DEPTH
@xrobau said in Revisiting ZFS and FreeNAS in 2019:
you're trying to claim that you know this better than me, when you obviously don't. It's massively frustrating.
This is the greatest collection of "I'm smart and I'm an expert" statements I've ever seen. The more someone has to tell you how brilliant they are....
