New IT update 60TB / 60 mil files / 20 people - HP Equipment
-
Hey guys, thought I'd get your input. Our org has grown and I'm needing to double our internal production server capacity. We are currently running an HP DL360 G8 / Windows Server 2012 Standard / P822 / 3 D2600 + 36 HP 3TB SAS drives (MB3000FBUCN) configured in raid 10ADM (3 drives per set).
I'm getting horrible benchmarks on the random read MB/s access with CrystalDiskMark RND4K Q1T1 (3.73 vs 2780 sequential). This server stores about 30 million files ranging from 50kb - 10MB and the full storage capacity rotates probably 2 - 3 times a year as we process these jobs then move them to cold storage.
I'm thinking
A.) - 2 D3600s + 6TB drives (drop to raid 10) / maybe look into HP smart caching and add D3700 with some SSDs for faster performance
or
B.) - 2 D3700 2.4TB SFF HP drives (higher speed / more spindles = faster)- Is SmartCaching off of SSDs really going to be significantly faster than just having 48 10K spindles?
- Is SmartCaching by HP in any way going to make the array more likely to fail? I partially picked Raid 10 for robustness / offloading complexity from controller - haven't had to replace a drive in 2 years now - if a SSD goes out in the caching is HP just going to pass thru to main disk / ignore caching?
- May be open to other storage suggestions - we just started with HP, I've had great luck using all used parts except for HDs for 8 years now, spend most of my time head down in development stuff, so vastly prefer set & forget hardware.
-
@jim9500 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Is SmartCaching off of SSDs really going to be significantly faster than just having 48 10K spindles?
So much faster.
-
@jim9500 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Is SmartCaching by HP in any way going to make the array more likely to fail? I partially picked Raid 10 for robustness / offloading complexity from controller - haven't had to replace a drive in 2 years now - if a SSD goes out in the caching is HP just going to pass thru to main disk / ignore caching?
SmartCaching is more to fail, but it's a cache, not a tier, so if the cache fails you don't lose data unless you have write caching going on without any protection. but as a read cache, it presents no data loss risk.
-
36TB on SSD is probably the way to go. While not small, that's not ridiculously large of a data set. The insanely fast performance and you can move from triple mirror RAID 10 to RAID 5 or 6 and you'll need so many fewer disks, while getting so much more performance. The cost difference might be small. Maybe even cheaper.
-
@jim9500 The speed of access for SSDs compared to spindles is so much faster, there is really no comparison. SSDs actually became cheaper years ago because you can use RAID 5/6 with them without the huge risk of a rebuild failure on spinning drives. At todays prices SSD just makes so much more sense.
If you have the 25x 2.5" chassis on that HPDL360 G8, you could easily fit your needed storage in it using the 1.9TB SSDs (something like Intel DC series).
-
I'd love to see the pricing here.
I could have sworn the last time I priced storage (a year go) RAID 10 for 4 TB vs SSD, SSD RAID 5 was still more expensive using vendor branded drives.
-
All we need is the price of some drives from a vendor to do a bunch of comparisons.
-
@dashrender said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
I could have sworn the last time I priced storage (a year go) RAID 10 for 4 TB vs SSD, SSD RAID 5 was still more expensive using vendor branded drives.
You left out the key part that matters.... how many drives.
If you only need four drives in RAID 10, the the RAID 10 is going to be WAY cheaper. If you need a thousand, the SSD is going to be WAY cheaper. Most things fall in the middle.
-
Some numbers from CDW...
HPE Midline (NL-SAS) 4TB: $1005 https://www.cdw.com/product/hpe-4tb-sas-12g-midline-7200rpm-lff-3.5-lp-digitally-signed-hdd/4972857?pfm=srh
HPE SSD 1.92TB: $1137 https://www.cdw.com/product/hpe-read-intensive-solid-state-drive-1.92-tb-sas-12gb-s/5531989?pfm=srh
-
@scottalanmiller said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Some numbers from CDW...
HPE Midline (NL-SAS) 4TB: $1005 https://www.cdw.com/product/hpe-4tb-sas-12g-midline-7200rpm-lff-3.5-lp-digitally-signed-hdd/4972857?pfm=srh
HPE SSD 1.92TB: $1137 https://www.cdw.com/product/hpe-read-intensive-solid-state-drive-1.92-tb-sas-12gb-s/5531989?pfm=srh
-
@scottalanmiller said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
@dashrender said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
I could have sworn the last time I priced storage (a year go) RAID 10 for 4 TB vs SSD, SSD RAID 5 was still more expensive using vendor branded drives.
You left out the key part that matters.... how many drives.
If you only need four drives in RAID 10, the the RAID 10 is going to be WAY cheaper. If you need a thousand, the SSD is going to be WAY cheaper. Most things fall in the middle.
Yeah that’s my situation... I need 4 to of space... two SSD raid 10
Or 10+ drives to get a min iops on drives
-
So tiny array.... 8TB...
RAID 10 with NL-SAS: $4020
RAID 5 with SSD: $5685Close enough that I'd do the SSD almost every time due to the massive performance gap. Those NL-SAS are very slow. Those are not 10K drives, that's for sure.
-
@dustinb3403 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
@scottalanmiller said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Some numbers from CDW...
HPE Midline (NL-SAS) 4TB: $1005 https://www.cdw.com/product/hpe-4tb-sas-12g-midline-7200rpm-lff-3.5-lp-digitally-signed-hdd/4972857?pfm=srh
HPE SSD 1.92TB: $1137 https://www.cdw.com/product/hpe-read-intensive-solid-state-drive-1.92-tb-sas-12gb-s/5531989?pfm=srh
Grey market, probably off of the back of a truck.
-
@jim9500 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Hey guys, thought I'd get your input. Our org has grown and I'm needing to double our internal production server capacity. We are currently running an HP DL360 G8 / Windows Server 2012 Standard / P822 / 3 D2600 + 36 HP 3TB SAS drives (MB3000FBUCN) configured in raid 10ADM (3 drives per set).
I'm getting horrible benchmarks on the random read MB/s access with CrystalDiskMark RND4K Q1T1 (3.73 vs 2780 sequential). This server stores about 30 million files ranging from 50kb - 10MB and the full storage capacity rotates probably 2 - 3 times a year as we process these jobs then move them to cold storage.
Forget 2.5" HDDs. SSDs has killed that market with huge performance improvements for the same or lower price. And much higher reliability.
Where magnetic media reigns supreme is large storage arrays using 10+TB 3.5" HDDs. Best price per TB for this technology.
SSDs have moved on to the NVMe interface for performance reasons but there are still legacy applications for SAS and SATA SSDs.
Don't know how much storage you need at what speed, but you'd get the best out of both worlds with a mix of 3.5" HDDs and SSDs (preferable NVMe).
If you're using an external SAS enclosure, put the SSDs in the server and the HDDs in the enclosure. That way you'll minimize the performance drop. SSDs will otherwise saturate the SAS links.
If you want to go all SSDs, you might want to go for SATA if you want lots of them and want to put them on a raid card. Current pricing is about $170 per TB for Samsung's value enterprise SATA and also their NVMe drives. There are few SAS SSDs available and they cost more. If you buy from HP (or Dell) expect to pay 2 to 3 times as much.
As a comparison 3.5" enterprise HDDs are below $30 per TB.
As for drive sizes, SSDs will usually get cheaper per TB as you go bigger up to about 8TB or so. Fewer drives are better than more drives so go as big as you need.
For 3.5" drives the point of diminishing return is 16TB. There are larger drives but they are not as cost effective and can be hard to find.
-
Much thanks for the replies guys!
@scottalanmiller Interesting - so the SSD tech got rid of the parity calc corruption issue on raid 5 / raid 6 reubuilds?
@scottalanmiller said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Some numbers from CDW...
HPE Midline (NL-SAS) 4TB: $1005 https://www.cdw.com/product/hpe-4tb-sas-12g-midline-7200rpm-lff-3.5-lp-digitally-signed-hdd/4972857?pfm=srh
HPE SSD 1.92TB: $1137 https://www.cdw.com/product/hpe-read-intensive-solid-state-drive-1.92-tb-sas-12gb-s/5531989?pfm=srh
Trying to figure out what the difference between the $1005 & the $405 versions are. https://www.cdw.com/product/hpe-midline-hard-drive-4-tb-sas-12gb-s/6424407?pfm=srh - see this on their 8TB too. Would CDW or serversupply.com be selling grey market drives?
-
@jim9500 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Much thanks for the replies guys!
@scottalanmiller Interesting - so the SSD tech got rid of the parity calc corruption issue on raid 5 / raid 6 reubuilds?
@scottalanmiller said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
Some numbers from CDW...
HPE Midline (NL-SAS) 4TB: $1005 https://www.cdw.com/product/hpe-4tb-sas-12g-midline-7200rpm-lff-3.5-lp-digitally-signed-hdd/4972857?pfm=srh
HPE SSD 1.92TB: $1137 https://www.cdw.com/product/hpe-read-intensive-solid-state-drive-1.92-tb-sas-12gb-s/5531989?pfm=srh
Trying to figure out what the difference between the $1005 & the $405 versions are. https://www.cdw.com/product/hpe-midline-hard-drive-4-tb-sas-12gb-s/6424407?pfm=srh - see this on their 8TB too. Would CDW or serversupply.com be selling grey market drives?
GOod question. I have no idea.
But easily, they might be. Or used.
-
@jim9500 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
@scottalanmiller Interesting - so the SSD tech got rid of the parity calc corruption issue on raid 5 / raid 6 reubuilds?
Nope, it's the same issue. People forgot about it because when the SSDs started to appear on servers, they were small drives like 100GB. Nowadays they are the same size as magnetic media.
The factor was the probability of unrecoverable bit errors during the array rebuild process. The only redeeming factor for enterprise SSDs is that they are better in this regard. JEDEC standard for enterprise SSDs is 10^16 or better, while it's commonly 10^15 for enterprise HDDs.
So it's still a question about probability of failure during rebuild. And the same rules applies but using different numbers.
-
@jim9500 said in New IT update 60TB / 60 mil files / 20 people - HP Equipment:
D3700
I'd add one thing that sometimes gets overlooked with all-flash storage. A lot of software or hardware-based storage solutions offer inline dedupe and compression that helps save even more storage with SSD.
