The Software RAID Inflection Point
-
In June, 2001 something amazing happened in the IT world: Intel released the Tualatin based Pentium IIIS 1.0 GHz processor. This was one of the first few Intel processors (IA32 architecture) to have crossed the 1 GHz clock barrier and the first of any significance. It was also special in that it had dual processor support and a double sized cache compared to its Coppermine based forerunners or it's non-"S" Tualatin successor (that followed just one month behind.) The PIIIS system boards were insanely popular in their era and formed the backbone of high performance commodity servers, such as Proliant and PowerEdge, in 2001 and for the next few years culminating in the Pentium IIIS 1.4GHz dual processor systems that were so important that they resulted in kicking off the now famous HP Proliant "G" naming convention. The Pentium III boxes were "G1".
What does any of this have to do with RAID? Well, we need to step back and look at where RAID was up until May, 2001. From the 1990s and up to May, 2001 hardware RAID was the standard for the IA32 server world which mainly included systems like Novell Netware, Windows NT 4, Windows 2000 and some Linux. Software RAID did exist for some of these systems (not Netware) but servers were always struggling for CPU and memory resources and expending these precious resources on RAID functions was costly and would cause applications to compete with RAID for access and the systems would often choke on the conflict. Hardware RAID solved this by adding dedicated CPU and RAM just for these functions.
RAID in the late 1990s and early 2000s was also very highly based around RAID 5 and to a lesser degree, RAID 6, parity striping because disks were tiny and extremely expensive for capacity and squeezing maximum capacity out of the available disks was of utmost priority and risks like URE were so trivial due to the small capacity sizes that parity RAID was very reliable, all things considered. The factors were completely different than they would be by 2009. In 2001, it was still common to see 2.1GB, 4.3GB and 9GB hard drives in enterprise servers!
Because parity RAID was the order of the day, and many drives were typically used on each server, RAID had more CPU overhead on average in 2000 than it did in 2010! So the impact of RAID on system resources was very significant.
And that is the background. But in June, 2001 suddenly the people who had been buying very low powered IA32 systems had access to the Tualatin Pentium IIIS processors with greatly improved clock speeds, efficient dual processor support and double sized on chip caches that presented an astounding leap in system performance literally over night. With all this new power and no corresponding change in software demands systems that traditionally were starved for CPU and RAM suddenly had more than they knew how to use, especially as additional threads were available and most applications of the time were single threaded.
The system CPUs, even in the Pentium III era, were dramatically more powerful than the small CPUs, which were often entry level PowerPC or MIPS chips, on the hardware RAID controllers and the available system memory was often much larger than the hardware RAM caches and investing in extra system memory was often far more effective and generally advantages so with the availability of free capacity on the main system RAID functions could, on average be moved from the hardware RAID cards to the central system and gain performance, even while giving up the additional CPU and RAM of the hardware RAID cards. This was not true on overloaded systems, those starved for resources and was more relevant for parity RAID systems with RAID 6 benefiting the most and non-parity systems like RAID 1 and 0 benefiting the least.
But June, 2001 was the famous inflection point - before that date the average IA32 system was faster when using hardware RAID. And after June, 2001 new systems purchased would on average be faster with software RAID. With each passing year the advantages have leaned more and more towards software RAID with the abundance of underutilized core CPUs and idle threads and spare RAM exploding with the only advantage towards hardware RAID being the drop in parity RAID usage as mirrored RAID took over as the standard as disk sizes increased dramatically while capacity costs dropped.
Today is has been more than fifteen years since the notion that hardware RAID would be faster has been retired. The belief lingers on due primarily to the odd "Class of 1998" effect. But this has long been a myth repeated improperly by those that did not take the time to understand the original source material. Hardware RAID continues to have benefits, but performance has not been one of them for the majority of the time that we've had RAID and is not expected to ever rise again.
-
OK this mostly makes sense, But what prevented the RAID controllers getting better processors that would keep them at pace or better than the main system CPUs?
On a side note, have you done any, or do you know of any test results that prove the OP?
-
@Dashrender said in The Software RAID Inflection Point:
OK this mostly makes sense, But what prevented the RAID controllers getting better processors that would keep them at pace or better than the main system CPUs?
Cost and practicality. RAID doesn't benefit from multi-threading so the manner in which normal CPUs have gotten faster is not very beneficial to RAID but the cost of a fast processor is decently high (say, at least $400) while few are willing to spend more than around $900 on a good hardware RAID card. It's just a matter of practicality. Using "spare" CPU resources for RAID costs very little and is extreme effective. Getting equivalent resources onto the RAID card is totally doable, but unnecessarily expensive and there is little to no value to it. And it is very inflexible - if you go with RAID 10 and use very little CPU, a hardware RAID card just sits idle, it doesn't return that unused power to the main pool. And the RAM you buy is the RAM you get. With software RAID, you can tune as needed or let it expand. If you have 16GB not being used, you can use it. You don't have to hardware partition it.
There is just no reason to be building insanely powerful RAID cards, it's a niche need that doesn't exist any longer.
-
@Dashrender said in The Software RAID Inflection Point:
On a side note, have you done any, or do you know of any test results that prove the OP?
Lots of testing was done on this 15+ years ago. It's been so long since it was standard knowledge that no one has cared in forever. I'm sure that people have results archived somewhere but by today it is a little like asking if we can prove that airplanes are actually faster than walking. There is no reason to think that hardware RAID would be able to keep up, there is no factor making it competitive. Even in the late 1990s it was only faster when the main CPUs were already taxed.
-
The thing that is really showing the performance differences today is large SSD arrays. It only takes about four to six enterprise SSDs to overrun a typical hardware RAID card but many, many times that to tax the core CPU. It would be awesome to do some tests of that with unlimited hardware to see when you actually max out different CPUs or threads of CPUs.
-
@scottalanmiller said in The Software RAID Inflection Point:
The thing that is really showing the performance differences today is large SSD arrays. It only takes about four to six enterprise SSDs to overrun a typical hardware RAID card but many, many times that to tax the core CPU. It would be awesome to do some tests of that with unlimited hardware to see when you actually max out different CPUs or threads of CPUs.
What options are available from a Software RAID standpoint that would be as "easy" to use as HW RAID given the point you've just made re SSDs?
-
@FATeknollogee said in The Software RAID Inflection Point:
@scottalanmiller said in The Software RAID Inflection Point:
The thing that is really showing the performance differences today is large SSD arrays. It only takes about four to six enterprise SSDs to overrun a typical hardware RAID card but many, many times that to tax the core CPU. It would be awesome to do some tests of that with unlimited hardware to see when you actually max out different CPUs or threads of CPUs.
What options are available from a Software RAID standpoint that would be as "easy" to use as HW RAID given the point you've just made re SSDs?
None are super easy. And it depends on the situation. Hardware RAID is abstracted so as long as you are on the same physical servers you can use the same hardware RAID for any OS or HV. But software RAID has to come on the OS or HV (it could be separate but in the real world it never is) so you are limited to the one or two choices that are available with your OS or HV and several have opted to just do without any or without any that are truly viable (Windows, Hyper-V, VMware, Netware.)
So "easy" is relative. None are really hard, but none are totally trivial like hardware RAID normally is. And it is unique to your platform.
-
So some normal choices:
- Linux: MD RAID, MD RAID inside LVM2, ZFS and soon BtrFS
- FreeBSD: ZFS or BSD LVM RAID
- Dragonfly: HammerFS
- Solaris: ZFS or Solstace
- Xen and KVM: Same as the Dom0 Linux instance or kernel version they utilize
- XenServer: MD RAID
- AIX: LVM
Microsoft is attempting to beef up their formally anemic software RAID offering after two whole decades of it being effectively unusable and are rebranding it as Storage Spaces and in time we might trust and recommend that system. But as of yet it does not have a track record to warrant consideration outside of a lab and the rebranding of historically bad software to hide former shame is never a good starting point for new software. This impacts both Windows and Hyper-V equally.
VMware has simply opted out of the software RAID business and does not offer it. They do not target customers at the size where RAID makes a lot of sense, for those that need it hardware RAID works just fine.
-
@scottalanmiller With Storage Spaces being "new" & VMware having no SW RAID, I'm surprised that no one has tried/attempted to make MD RAID 'easier/user friendly/simpler' (insert your fav term) to use & implement.
With the abundance of bigger, faster, cheaper SSDs & CPUs...this seems like a good time for MD RAID to be popular?
-
@FATeknollogee said in The Software RAID Inflection Point:
@scottalanmiller With Storage Spaces being "new" & VMware having no SW RAID, I'm surprised that no one has tried/attempted to make MD RAID 'easier/user friendly/simpler' (insert your fav term) to use & implement.
With the abundance of bigger, faster, cheaper SSDs & CPUs...this seems like a good time for MD RAID to be popular?
Blind swap has a ton of value....I don't see it growing much legs, especially in SMB until it can provide that.
-
@FATeknollogee said in The Software RAID Inflection Point:
@scottalanmiller With Storage Spaces being "new" & VMware having no SW RAID, I'm surprised that no one has tried/attempted to make MD RAID 'easier/user friendly/simpler' (insert your fav term) to use & implement.
MD really couldn't be much easier to use than it already is. Even having it send you email notifications isn't that difficult if you can read a man page.
I've not heard of anyone trying to port MD to a different system yet, and it really doesn't make sense except for one case as almost every player in the market has a software RAID system that's already as stable and fast, and that one player refuses to use one of the open source software RAID systems in favor of their own questionable code.
With the abundance of bigger, faster, cheaper SSDs & CPUs...this seems like a good time for MD RAID to be popular?
Can't get much more popular than it already is for those of us in the know, honestly.
-
@travisdh1 said in The Software RAID Inflection Point:
Can't get much more popular than it already is for those of us in the know, honestly.
Why is ML still recommending HW RAID?
Have you solved the blind swap issue or is that something you aren't really concerned with?
-
@travisdh1 said in The Software RAID Inflection Point:
MD really couldn't be much easier to use than it already is. Even having it send you email notifications isn't that difficult if you can read a man page.
The issue with MD is how do you get yourself the ability to do "blind swap" ?
Hot swap isn't that big of a deal... as long as you hot swap the right drive, lol.
-
@FATeknollogee said in The Software RAID Inflection Point:
@travisdh1 said in The Software RAID Inflection Point:
Can't get much more popular than it already is for those of us in the know, honestly.
Why is ML still recommending HW RAID?
Have you solved the blind swap issue or is that something you aren't really concerned with?
Blind swap isn't an issue with hardware raid, and in general, I think, for SMBs, at least, HW raid win's because of that.
-
@FATeknollogee said in The Software RAID Inflection Point:
@travisdh1 said in The Software RAID Inflection Point:
Can't get much more popular than it already is for those of us in the know, honestly.
Why is ML still recommending HW RAID?
Have you solved the blind swap issue or is that something you aren't really concerned with?
Because the software RAID would rarely, if ever, be aware of the type of hardware it's it, i.e. a HP server, a Dell server, etc - it probably has no way of mapping a drive to a specific port on the chassis and lighting the lights, etc - the things needed to make Blind Swap work.
-
@dafyre said in The Software RAID Inflection Point:
Blind swap isn't an issue with hardware raid, and in general, I think, for SMBs, at least, HW raid win's because of that.
Agreed, but if MD is "better", why recommend a solution that is not as good!
-
@FATeknollogee said in The Software RAID Inflection Point:
@dafyre said in The Software RAID Inflection Point:
Blind swap isn't an issue with hardware raid, and in general, I think, for SMBs, at least, HW raid win's because of that.
Agreed, but if MD is "better", why recommend a solution that is not as good!
Better is a matter of opinion - what's better a faster car or a more fuel efficient car?
That's the issue we have here - The main CPU has enough left over processor, etc to handle software RAID, but you give up blind swap. Both are very valuable.
-
@scottalanmiller recently made a post about how hardware RAID wasn't dead yet.. I don't recall if it was in here or it's own thread.
here it is.
https://mangolassi.it/topic/12043/why-the-smb-still-needs-hardware-raid/6
-
@Dashrender said in The Software RAID Inflection Point:
Because the software RAID would rarely, if ever, be aware of the type of hardware it's it, i.e. a HP server, a Dell server, etc - it probably has no way of mapping a drive to a specific port on the chassis and lighting the lights, etc - the things needed to make Blind Swap work.
In today's world with all the smart people we have, no one can write software that can map drives to a chassis?
-
@FATeknollogee said in The Software RAID Inflection Point:
@Dashrender said in The Software RAID Inflection Point:
Because the software RAID would rarely, if ever, be aware of the type of hardware it's it, i.e. a HP server, a Dell server, etc - it probably has no way of mapping a drive to a specific port on the chassis and lighting the lights, etc - the things needed to make Blind Swap work.
In today's world with all the smart people we have, no one can write software that can map drives to a chassis?
If a hardware vendor wanted to, they easily could, but it would mean seriously hampering the number of options available on a general purpose server platform. NAS vendors do this all the time.
