Unsolved Is xByte still recommended for server purchases around here?
-
@jaredbusch has anyone experienced problems with corrupted backups like this
https://forums.veeam.com/veeam-backup-replication-f2/synology-nas-as-repo-t77177.html
I have never seen it myself, but it doesn't mean it is not happening. -
@beta Not into NAS for backups. If something goes wrong on the NAS side there's not a lot that can be done. They are too cookie cutter.
https://forums.veeam.com/veeam-backup-replication-f2/synology-nas-as-repo-t77177.html
Just don't do NAS.
Build a purpose built box with XFS and keep it isolated from everywhere except a PAW that's nowhere near a perp entry point.
https://www.veeam.com/blog/v11-immutable-backup-storage.html
Then the box can be set up to be immutable. -
@jaredbusch said in Is xByte still recommended for server purchases around here?:
@beta said in Is xByte still recommended for server purchases around here?:
my new server is going to be used to store Veeam backups which means
If you are only storing backups, then you don't need to pay for server hardware. Buy a quality NAS, like a Synology, and drop disks in it.
Agreed, better way to go.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
If something goes wrong on the NAS side there's not a lot that can be done. They are too cookie cutter.
Actually that's a reason that I like Synology. You can do almost anything to repair it because it's well known hardware with extremely well known enterprise software RAID that is portable to other devices both NAS and custom built.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
https://forums.veeam.com/veeam-backup-replication-f2/synology-nas-as-repo-t77177.html
That forum is so full of misinformation. If that guy really works for Veeam, I'd never trust Veeam again. He's just out selling hardware solutions based on FUD. He even mentions that Veeam has vendors pay for Veeam to support them, which he is doing.
And people saying to use SAN instead of NAS. There's tons of anecdotes in there of "this bad thing happened when I did X" but "didn't happen when I did Y" without any clear research into all of the things that they changed. No one is doing testing, no one is checking the same setup but NTFS vs ReFS or whatever.
The real problem is likely that Synology depends on checks that it needs to do on the filesystem because it does insanely risky storage operations that are expected to corrupt from time to time based on the math, and so SAN is necessary because you need storage intelligence from higher in the stack to check the data. They just don't want to say that for some reason and instead lie about it being that hardware RAID controllers are magic and don't corrupt and the same algorithms not on an ASIC are flaky and useless. That thread is all BS.
Bottom line is... Veeam is telling you their own stuff isn't stable, he's super clear about that.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
Build a purpose built box with XFS and keep it isolated from everywhere except a PAW that's nowhere near a perp entry point.
From that same Veeam thread, this will have the same problems. Appears that Veeam can only do its checksumming on a SAN via NTFS or ReFS (which I'm not sure I'd trust yet as it is known to be unstable until at least quite recently.) So, NTFS if you want to be safe. Which you can do on purpose built hardware, and on Linux, but just not with XFS (which SHOULD be the best option if Veeam knew how to handle their own storage properly) or ZFS (which should be a great option, too.)
-
@scottalanmiller said in Is xByte still recommended for server purchases around here?:
@phlipelder said in Is xByte still recommended for server purchases around here?:
If something goes wrong on the NAS side there's not a lot that can be done. They are too cookie cutter.
Actually that's a reason that I like Synology. You can do almost anything to repair it because it's well known hardware with extremely well known enterprise software RAID that is portable to other devices both NAS and custom built.
It was a fellow Microsoft MVP that put the NAS vendors under the gun to get their collective shit together because the NAS units kept corrupting ShadowProtect incremental file chains.
http://sbsfaq.com/what-have-qnap-done-about-the-data-corruption-issue/
There's zero, zippo, zilch, accountability to the end user with an econo box. None.
When the shit hits the fan, I want real support with real people. That's gonna cost more than some box with a baby motherboard, some memory, some sort of flash storage for the *NIX OS, and whoever's drives in the drive bay.
I can't count the number of times we've had SMB clients using a NAS as a file share hit issues with that NAS, its repository, or just outright resetting itself requiring GetDataBack *NIX RAID Reconstructor to hopefully pull it all in.
As far as the Veeam slamming goes, no comment. We've been working with the product for five years or more now. Before that it was StorageCraft's ShadowProtect. Both were, and are, flawless and there when we need to recover sometimes when things are extremely stressful after an all-out blowout.
Note that Veaam has a NAS Backup product. It works. Use it.
That being said, Immutable is here to stay. Veeam was one of the first on the block to utilize it built-in to the product. We're tied into BackBlaze B2 for all of our cloud tiers that are not running on our own backup (Cloud Connect) systems.
https://www.veeam.com/blog/v11-immutable-backup-storage.html
Didier's Part 1 for building one:
https://www.starwindsoftware.com/blog/veeam-hardened-linux-repository-part-1 -
@phlipelder said in Is xByte still recommended for server purchases around here?:
When the shit hits the fan, I want real support with real people. That's gonna cost more than some box with a baby motherboard, some memory, some sort of flash storage for the *NIX OS, and whoever's drives in the drive bay.
That's the thing, it's Linux, so the top enterprise support is available. That you got it from a NAS vendor doesn't really matter. Sure, it would be BETTER with better hardware and a more enterprise version of Linux. I'm not saying it doesn't get better. But you can get all that enterprise support in any low end NAS box if you want, because the components being supported are universal.
Just like if you were to put Windows on a Dell server... the quality of enterprise level support comes down to who can support Windows. That it's on Dell or a piece of crap hardware might make a tiny difference for hardware uptime, but has no bearing on the quality of support that matters. The NAS box can be replaced for a Dell, HPE, Cisco, whatever big "support" brand you want after data loss if you want. The hardware is actually interchangeable here.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
I can't count the number of times we've had SMB clients using a NAS as a file share hit issues with that NAS, its repository, or just outright resetting itself requiring GetDataBack *NIX RAID Reconstructor to hopefully pull it all in.
If you have cheap hardware failure, yeah, that's going to be a problem. But I've never seen anything like that on a Synology.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
As far as the Veeam slamming goes, no comment. We've been working with the product for five years or more now. Before that it was StorageCraft's ShadowProtect. Both were, and are, flawless and there when we need to recover sometimes when things are extremely stressful after an all-out blowout.
The Veeam slamming was by Veeam's engineer, not by me. I was just pointing out what Veeam claimed about themselves. And I'd tend to trust them.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
Note that Veaam has a NAS Backup product. It works. Use it.
I thought that they exclusively used SAN because they had problems with the NAS protocols?
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
Note that Veaam has a NAS Backup product. It works. Use it.
A product to use NAS, or a NAS themselves? Got a link, I searched but am not sure which product you are referencing.
-
@scottalanmiller said in Is xByte still recommended for server purchases around here?:
@phlipelder said in Is xByte still recommended for server purchases around here?:
Note that Veaam has a NAS Backup product. It works. Use it.
I thought that they exclusively used SAN because they had problems with the NAS protocols?
One can back up a NAS to S3 via Veeam. Sorry for not being clear.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
@scottalanmiller said in Is xByte still recommended for server purchases around here?:
@phlipelder said in Is xByte still recommended for server purchases around here?:
Note that Veaam has a NAS Backup product. It works. Use it.
I thought that they exclusively used SAN because they had problems with the NAS protocols?
One can back up a NAS to S3 via Veeam. Sorry for not being clear.
Oh yes, that I know, and that's great stuff.
Is Veeam now recommending a Veeam-built repo on top of Linux (RHEL/Ubuntu I assume) with XFS as the backup target of choice?
Edit: Now meaning V11
-
Mostly we are getting derailed. I think the bottom line items are this...
- Yes, xByte is an excellent place to buy servers. Give them a try still.
- Enterprise hardware has a reliability advantage over "business" hardware. That doesn't mean business hardware isn't usable, just be aware of the differences.
- An enterprise Linux distro that you install and maintain yourself (or via a support partner) such as Ubuntu, Fedora, CentOS, Debian, RHEL, or Suse is simply better tested and supported than a lesser known non-enterprise disto that is commonly shipped with any black box style device (like a NAS.)
- If you are using Veeam, then you want Veeam's data protection algorithms in place rather than relying on "blind" data protection algos on distance storage devices. Whether that means configuring your storage as a SAN or putting Veeam's agent on your Linux distro, one way or another you will be better served letting Veeam handle that layer of protection too. (This is because Veeam is often used in a "differential forever" style mode that incurs a huge amount of risk if not mitigated somehow.)
-
@scottalanmiller said in Is xByte still recommended for server purchases around here?:
@phlipelder said in Is xByte still recommended for server purchases around here?:
When the shit hits the fan, I want real support with real people. That's gonna cost more than some box with a baby motherboard, some memory, some sort of flash storage for the *NIX OS, and whoever's drives in the drive bay.
That's the thing, it's Linux, so the top enterprise support is available. That you got it from a NAS vendor doesn't really matter. Sure, it would be BETTER with better hardware and a more enterprise version of Linux. I'm not saying it doesn't get better. But you can get all that enterprise support in any low end NAS box if you want, because the components being supported are universal.
Just like if you were to put Windows on a Dell server... the quality of enterprise level support comes down to who can support Windows. That it's on Dell or a piece of crap hardware might make a tiny difference for hardware uptime, but has no bearing on the quality of support that matters. The NAS box can be replaced for a Dell, HPE, Cisco, whatever big "support" brand you want after data loss if you want. The hardware is actually interchangeable here.
That hasn't been my experience with any of the NAS vendors.
Even the Synology 2U NAS/SAN to NAS/SAN replication units that were supposed to be transparent to the Hyper-V cluster running in front of them. Synology refused to address our concerns with forum's posts that showed the promise was never fulfilled.
We won't ever deploy a NAS for anything critical. Backups are critical.
This is one of our chassis starting places the CS381:
https://www.silverstonetek.com/product.php?pid=861&area=enIt's an excellent platform. Their SFX power supplies are good. There are twin PSU setups out there but they are ATX so won't fit.
Micro-ATX offers a plethora of AMD EPYC, Intel Xeon Scalable, Intel Xeon, and other platforms to install in the box.
We have MLNX 25GbE RDMA running on both AMD EPYC and Intel Scalable platforms into SATA SSD for cache/journal and SATA or SAS NearLine spindles for bulk storage.
Our payback is really good with these boxes on 14TB to 16TB spindles.
We know what's in the box. We know how to fix things or recover things if thing go awry which they can. That's our comfort zone. No black boxes for critical data. Ever.
-
@beta said in Is xByte still recommended for server purchases around here?:
I was going to put 8 14TB drives (Dell drives bought with the server through xByte if I go that route) in RAID 6 for 84TB raw storage. Is that size array unwise for RAID 6?
Enterprise drives? RAID 6 is probably okay. You are on the large side a UREs will be becoming risky at that scale (but not tragic, just risky) and the larger concern is rebuild time. If one of those drives fail, you might be waiting a while to rebuild, especially if you have a hardware RAID controller which tend to have a small fraction of the computational power of software RAID. So be aware that replacing a drive might mean lots of downtime.
But as this is backup storage, a little downtime is not the risk it would be with, say, a database.
So I think RAID 6 is going to be fine. Or, you can do RAID 7 but I don't know if Veeam supports the use of ZFS instead of XFS in this scenario. If we were talking purely a SAN system to present iSCSI to the backup server, then RAID 7 might make the most sense.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
That hasn't been my experience with any of the NAS vendors.
Even the Synology 2U NAS/SAN to NAS/SAN replication units that were supposed to be transparent to the Hyper-V cluster running in front of them. Synology refused to address our concerns with forum's posts that showed the promise was never fulfilled.That's the thing, enterprise support never comes from the NAS vendors. They come from enterprise Linux support shops. This is the same even if you are using NetApp (but then a BSD support shop, obviously.) NAS, by definition, is a "black box" for people who don't want to support stuff. So the whole idea of enterprise support from a NAS vendor doesn't make too much sense. It's not an enterprise product, it's not for support technical use. Yes, some super technical people deploy them, but mostly when we want something really simple and we are going to support it completely ourselves or it is a cog that we can just replace.
If you want top end support for something like Synology, you can't get it from Synology themselves. Which makes the cost higher than it seems, and mostly defeats the purpose. So avoiding it can make sense if you need that, but it's different than not being available. You just have to treat it as a normal server and get nor system admin support resources. Then you can do effectively anything with it because it's just essentially a lower end SuperMicro chassis (not really, but super similar) and a stripped down really basic, slightly older Linux installation.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
Our payback is really good with these boxes on 14TB to 16TB spindles.
We know what's in the box. We know how to fix things or recover things if thing go awry which they can. That's our comfort zone. No black boxes for critical data. Ever.Yeah, I like this. I agree that doing your own OS install is way better, especially when you get to this size. NAS tends to make sense at the 2-4 spindle size because of the availability and price of chassis in that price range. Once you are into these bigger chassis, the value / cost savings evaporate.
In all cases, basic NAS or SAN style functionality is very basic operating system functions and there should never be a need (or value) to get that from one of these NAS vendors. It's just that they have chassis / mobo combos that are hard to compete with.
-
@phlipelder said in Is xByte still recommended for server purchases around here?:
No black boxes for critical data. Ever.
This is a good quote.
Should be, for critical workloads, though. Not only storage, but any component of it.
