Posts made by biggen

The SFF-8654 to dual SFF-8643 is a bit of a unicorn isn't it? Heck, the SFF-8654 isn't even listed in the SAS wiki.

@PhlipElder Excellent! Bookmarking your blog as well.

On a side note, I really really like Mangolassi.it Actual realife sys admins who you can bounce stuff off and ask questions. Glad this site is doing well. Its always my first search for something technical that I know someone in here will have dealt with at some point in their career.

So this Icy Dock enclosure would connect to both of those SlimSAS port with what exactly? Four of these?

Edit: No that wouldn't work. Like you said, need a Y-cable. Something like this?

Ha I just found an Anandtech article about that exact board: https://www.anandtech.com/show/15835/asrock-rack-offers-rome-matx-motherboard-with-only-6-memory-channels

@PhlipElder

The ROMED6U-2L2T is mATX? Whats the advantage there over a full size ATX board?

Yeah I have no problem whiteboxing stuff for me (or close family), but when you do it for others, they expect tech support for life. I don't really want to go down that road

But a PoC build may be more "in line" with his budge needs. Thanks for that @PhlipElder !

I appreciate all the help guys. Yeah I'm compiling a price list but it ain't cheap. Server alone would be about $7k and that's on the low end with smaller NVMe drives (1.6TB). Then still have to purchase the switch and then have to purchase the 10Gbe NICs for the workstations themselves.

Its a large investment that I bet never sees the light of day. It will turn into "I have $2k, what can you build with that?"

Chatting with Dell, they don't offer any of their Epyc servers with any 40Gbe offerings. They only go up to dual 25Gbe. They offer HDR100 Infiniband and Fibre channel, but these are pretty foreign to me and I don't even know if they can be used.

@taurex Thanks for that information. More to go over for me it seems!

@Pete-S I figure going Dell or HPE is the way to go for him. He needs to have a support contract behind something like this and it doesn't need to be me.

I hadn't considered uplinks of 40Gbe+. Makes sense.

Skip the hypervisor, huh? I figured it would add a performance penalty but makes backups that are so much easier. I don't even know how to perform bare metal backups on servers. Backing up the video files being worked on would be easy via a traditional Synology NAS (or custom built solution) but backing up the OS in the event that a update renders it broken would take some thought.

I assume Samba could keep up with 8GB/sec (assumes ~8 users all transferring at the same time) so long as the underlying storage is performant enough so Samba isn't waiting?

@travisdh1 Yeah it would be a Debian VM providing the SMB share (via Proxmox or xcp-ng) so MD RAID isn't an issue. Proxmox can use ZFS Raid 1 whilst xcp-ng can do standard MD RAID.

Edit: Dell even has that BOSS add-in system that allows for a RAID 1 bootable volume just for the OS. The NVMe drives could be VM storage only if I go that route.

I don't see any VROC mentioning in the system builder for any of the configurations I've done for the Intel systems. I'm guessing that is because Dell wants you to buy a PERC instead.

@Pete-S I'll have to look again then at Intel offering. I figured AMD had Intel blown out of the water as far as cost-per-core offerings go nowadays.

@PhlipElder What is the camera VMS solution you are using? Milestone? Axis?

@PhlipElder said in NVMe and RAID?:

@biggen Right now, the only place we're using NVME in servers is for either cache in a hybrid storage setting (cache/capacity or cache/performance/capacity) or for servers with all NVMe.

Intel's VROC plug-in dongle enables RAID 1 in certain settings. That's driven by the CPU. Not sure Dell supports it.

For most applications, an R740xd with high performance NV-Cache and SATA SSD in RAID 6 will do. Intel SSD DC-S4610 series (D3-4610).

We have plenty of setups like that for virtualized SQL/database workloads as well as 4K/8K video storage.

EDIT: Forgot, in the Intel Server Systems we deploy we install a couple Intel NVMe drives, the VROC dongle for Intel only NVMe, and RAID 1 them for the host OS.

Thanks for that info. Yeah, I'm thinking NVMe is probably overkill for video editing over a network connection. Especially considering the fact that he would be network bound anyway. I was thinking either 12Gb SAS SSDs in RAID 1 (2TB+ variety) or 6Gb SATA SSDs in Raid 1. This at least gives the option to go back to hot/blind swap with the appropriate PERC.

@travisdh1 said in NVMe and RAID?:

@scottalanmiller said in NVMe and RAID?:

@biggen said in NVMe and RAID?:

@brianlittlejohn So I guess the days of having a Jr. Admin blind swap are over then? It takes much more care and instruction to use software RAID.

That's correct. If you want that level of performance, blind swapping is kind of over. For now.

Not so much, at least for the major server vendors. Every back plane will let you flash the activity light of a failed drive. I know iDRAC and md allow you to do this (haven't had a reason to look into HP or Supermicro, but would be surprised if it were not built in as well.)

That allows you to ID a bum drive but you still have no way to rebuild it automatically like you would in a blind swap, right?

@PhlipElder I didn't see the option playing with the Dell system builder online. Honestly, the price of NVMe is so ridiculously expensive I'm not sure it warrants much investigation at this point. Single drives are in the $3k and up range depending on the size.

@scottalanmiller said in NVMe and RAID?:

@biggen said in NVMe and RAID?:

What about 10GBe bonding with SAS SSDs? Would you remove the network bottleneck with 20Gbe links?

Bonding will help with bandwidth, but will hurt latency. When you bond, the CPU has to do some work before things are sent down the pipe, so this slows things down. And you don't get 100% efficiency. A two way bond is pretty good, you get something akin to 195% the performance of a single pipe. The third bond is much less. The fourth, way less. And the fifth, so little that no one even discusses it. By the sixth, it's assumed that you are just getting slower, not faster.

What do you suggest then for removing the network bottleneck of 12GB SAS SSDs? Bonded twin 10Gbe links or a single 25GBe connection? The cost of switches and adapter cards goes up I'm assuming moving to 25Gbe.

What about 10GBe bonding with SAS SSDs? Would you remove the network bottleneck with 20Gbe links?

At least going with SAS/SATA puts hardware RAID back on the table for easier admin.

@scottalanmiller Yeah I was thinking about bonding multiple 10Gbe together from the server to the switch. Its some stuff I'd have to learn as I don't use anything with 10Gbe currently. I'm not even sure if twin or quad 10Gbe could handle the speeds of NVMe.

@scottalanmiller Is everyone moving to NVMe solutions for servers or is that more of a niche use case? I see you can still build them out with SAS/SATA SSDs for a fraction of the price but they are of course much slower.

I have a friend that does video editing on very large files. Talking 10s to 100s of GB in size and somewhere in between. He wants to have them stored in a central location (NAS like) in his facility so that he and others can collaborate on the same files and upload different versions to the NAS as they complete work on them. He was asking about building (or buying) a system that could support the work on these files. I don't have a ton of knowledge in this area but was thinking NVMe with 10Gbe network adapters just to handle the file sizes without having to wait on transfers back and forth.

@brianlittlejohn So I guess the days of having a Jr. Admin blind swap are over then? It takes much more care and instruction to use software RAID.

I was playing around on the Dell configuration website building out an Epyc 2 socket machine with an NVMe backplane. What I noticed is there is no RAID availability for this configuration. How is this handled then if I wanted to put in two identical NVMe U.2 drives and mirror them? Is hardware RAID not an option for this configuration? Is this left to the OS you choose now?